vladkens.cc

My 2025 Summary

2025-12-31T00:00:00+00:00
Another year is coming to an end, and following my new tradition</a>, I would like to sum up some results in writing. Overall, the year felt rather passive for me. I don’t remember planning any big things, but still, previous years were more active for me in terms of creativity and beyond (open source, learning new skills, and so on).
Open Source</a></h2>
This year, I didn’t release any new open-source projects, although I had some work in progress. I’m still maintaining my older projects little by little, but also without much enthusiasm (for some of them, this was the main idea from the start — stability).
More details about the main projects:

twscrape</a> (my most popular project) — it constantly requires dealing with new obstacles introduced by X;</li>
apigen-ts</a> — in the near future, the codegen will need to be rewritten using templates, because TypeScript was rewritten in Go</a>, and there is a chance that AST primitives will be removed from the `typescript</code> package on npm. Also, for me personally, the project has become a bit less relevant due to using tRPC/NextJS in my main stack, but for non-JS backends it is still a must-have;</li>`
`macmon</a> — it is also quite difficult to maintain, because Apple releases new processors too often and in many different configurations, so it’s not always possible to get the required hardware for testing. I looked for online services that provide access to “bare metal” hardware via SSH for a limited time (a VPS-like model), but didn’t find anything suitable;</li>`
ecloop</a> — I added SIMD optimizations for SHA-256 and RMD-160 to the library. I think this was my most interesting task this year. I wanted to play around with CUDA and even ordered a computer for that, but ended up not using it. As for crypto itself, I got into degen stuff (more on that later).</li> </ul> Public projects on GitHub — a total of 92 commits, 79 closed issues, 11 releases created, and more than 1k stars gained. UPD: I remembered that at the end of 2024 / beginning of 2025, I worked on badges.ws</a> in Rust. It looked like a fairly simple task, but turned into several months of work and debugging various APIs. In general, I probably should have dropped the project earlier, but I didn’t and pushed it through to a release — at the cost of my motivation to start new projects. What conclusion to draw from this, I’m not sure. Blog</a></h2> Overall, this year I wanted to write more posts, and I even have many drafts, but at some point I wanted to switch to a more convenient engine than Zola</a> (which I currently use). However, I couldn’t find anything good and flexible enough. Problems with Zola: a strange templating language that lacks some features, while issues remain open for years;</li> GPT tools don’t handle this templating system well — every time I need to make a change, it’s hard even to google how to do it;</li> lack of flexibility in page navigation and file organization (I can’t arrange blog files the way I want and have to follow the engine’s structure);</li> broken redirects in content — Zola doesn’t understand relative links in text (all links must be written relative to the site navigation), so it’s not convenient to write and work with the blog locally: linking between my own posts and being sure that all links are valid. Right now, I have to solve this with a separate script.</li> </ul> As an engine, I would like something that uses JSX as a templating language, has Tailwind CSS out of the box, doesn’t have Zola’s navigation and linking issues, and is a simple and stable solution. Overall, Astro</a> fits this category, and I tried migrating the blog to it, but it also has a number of its own problems: duplicated logic and the difficulty of choosing between .astro</code> and .jsx</code> formats;</li> lack of proper MD/MDX integration (any customization is done via CSS classes instead of properly overriding the rendering of Markdown tags with JSX components);</li> no built-in Atom feed generation (only RSS is supported), which means it has to be implemented manually.</li> </ul> Overall, I did a test migration of the blog to Astro, but never released it, because the solution felt too complex. A blog is something you might not touch for months, and when you finally do, you want to immediately understand how things work, rather than having to google or ask GPT for solutions. I also considered NextJS as an option, since it can be configured to fit your needs, but there are big concerns about its stability. I really don’t want to read changelogs every few months just to understand their new vision and figure out what needs to be updated in something that already worked, just to keep it working the same way. That turned into a lot of text about the blog. In short, I’d like to write more while being less distracted by technical problems. The ideal solution for me right now would be MD/MDX with the ability to override elements. Maybe someone knows a good existing solution? (or maybe it’s an idea for a project on Bun) Skills</a></h2> In previous years, I explored and practiced DevOps, Rust, Go, LLMs, NestJS/NextJS, and other things, but this year felt neutral. I can’t quickly name anything new that I learned this year. Probably the main change in my work environment was the switch from iTerm2 to WezTerm</a>. The reason was adding LLM features to the terminal and a security vulnerability</a>. Overall, throughout the year it felt scary to update software. A lot of things are now being built with AI, and the quality of releases has become frankly weaker. For example: I’ve been using Chrome since 2009, and this year was the first time I saw rendering bugs, crashes, and similar issues;</li> Google Search was broken for several months, and they didn’t seem to care (if you type a query with a typo, Search suggests the correct version, you can click it and it appears in the search field, but if you then click into the field, it reverts back to the original typo — this might seem like a small issue to some, but it’s a very common use case for me);</li> on YouTube, for several days after releasing the new player, the volume couldn’t be increased above 60% via the UI: the slider just wouldn’t move. I don’t understand how something that wasn’t even properly tested in production could be released;</li> many similar small bugs that ruin the UX.</li> </ul> In short, something has gone wrong in the world of software development. That’s why updating feels risky — you never know how your workflow will break this time. My favorite app is Sublime Merge</a>: updates there are purely cosmetic and happen only a couple of times a year. Crypto aka Degen</a></h2> This area deserves a separate mention. In January, I worked on a small project for bridging ERC-20 / SPL tokens between the Ethereum and Solana networks. Around the same time, Trump released his Meme Coin</a>: those who were insiders or just fast made good money, while everyone else ended up at a loss. That got me interested in how crypto works in general — not the technical side, but the community itself. As a result, I subscribed to various chats and other sources with related information (though it’s a huge stream, often full of noise). Later, I found the DoubleTop community</a>, where I’m now a member and communicate regularly. Overall, during the second half of the year I was moderately active in crypto — trying different areas to understand how things work. So far, my experience is this: it is possible to make money in crypto, but it’s really difficult and requires a lot of time and attention. Losing money is much easier. There are no huge multipliers on deposits, but steady percentages are possible. Cases I participated in: Sale Fleek on CoinList</a> — rekt −70%, the team basically scammed;</li> Sale Enso on CoinList</a> — good result, +70%;</li> Sale Plasma</a> — x10 on allocation, but it was hard to get in, and funds were locked for 3 months;</li> Sale Pump.fun — x2 in 3 days, but as always, there were nuances. I was lucky to get part of the allocation through the website, while CEXs issued refunds;</li> Almanak on Legion</a> — rekt −50%. Overall, it was a decent project, but the team delayed the launch and went live in a bad market. I forgot that I participated in this sale and didn’t sell the tokens in time;</li> Farm Paradex</a> — x2 on spend. The project was expected to launch in summer, but the release was later postponed indefinitely. This was my first retrodrop-focused project, so my enthusiasm for farming other projects cooled down. In November–December, the Paradex team added an XP transfer option, which made it possible to sell points OTC, and I used that;</li> Farm Lighter</a> — a lot of multiples here. I earned most of my points from hedge positions and didn’t farm aggressively on purpose. Given how bad the market is right now, the project literally gave everyone a New Year’s gift;</li> Binance Alpha — simple daily activity that required generating trading volume on Binance. It brought about $300–400 per month per account, but it was hard to multi-account;</li> Sales on BuidlPad</a> — the platform provided stable returns for about half a year. It was easy to multi-account, but accounts were also often blocked. A lot of liquidity was required to participate, since allocations were small. On average, all sales ended up around x2 on spend;</li> Flash crash on Oct 10</a> — the whole crypto market dropped hard within a couple of hours and still hasn’t returned to previous levels. I didn’t have any perp positions, so I got lucky there, but I did have USDT/USDC loans backed by crypto assets that were liquidated;</li> Spot positions (bad market): BNB +30%, ETH +20%, SOL −20%, HYPE −40%.</li> </ul> Of course, there were more cases — I listed the main closed ones. In general, it is possible to make money in crypto, but it’s really hard and requires a systematic approach. The crypto market is alive only for a few months a year, when so many events happen that it’s impossible to participate in everything (not enough capital, energy, or time). For most of the year, nothing really happens — that’s when people go work at McDonald’s. That quiet period is usually the time to farm retrodrops, but it’s genuinely difficult work: the results of your actions and money spent come only after 6–12 months, and psychologically that’s hard for me. The current meta is Perp DEXes (Paradex / Lighter / Aster / Extended / Pacifica.fi / Ethereal / Trade.xyz / etc.) and opinion markets (Polymarket / Opinion / Limitless / etc.). The hardest part of crypto for me is tracking balances and understanding whether I’m up or down overall (it’s basically real accounting work). Roughly speaking, I finished the year at about the same level I started, although at some points my deposit was up to +50%. It’s important to note that everything related to crypto should be done with proper risk control — I have a separate budget that I use only for experiments. Anyway, this is a big topic, and I’ll probably need to talk about it separately. Health</a></h2> Still alive — and that’s good. This year I quit smoking</a> — I’m very happy about it, I feel better, and I’ve reduced the risk of various cancers. Overall, quitting wasn’t that hard: I watched a few YouTube videos to understand what to expect and what kind of “tricks” my brain would use to convince me to smoke again. For the first three weeks or so, you constantly think about smoking, and then it fades. Most of this “thinking” isn’t about smoking itself, but about habits and routines — like taking a break, looking out the window, having a coffee, and going for a smoke. Now, from time to time, I get short five-minute urges in the form of flashbacks. I also started doing regular stretching/exercises — at least a couple of times a week (I track it in Sheets). Without this, I genuinely feel worse: it’s hard to work when your back hurts, for example. I periodically take body measurements, and probably the main change is my biceps: +2 cm. Leisure</a></h2> I bought an Xbox to relax by playing games (I had a PlayStation 4 in the past, so I wanted to try a different platform). I tried quite a lot of games via Game Pass — before that, I hadn’t played much for several years. From what I liked: Clair Obscur: Expedition 33 — an obvious game of the year, 100% worth playing: the story, art, music, and combat are all excellent.</li> Mafia: Definitive Edition — a remake of a classic, slightly simplified. The racing mission still takes dozens of attempts — and that’s a good thing.</li> Indiana Jones and the Great Circle — a solid game. It was enjoyable to explore Rome and the Egyptian pyramids and solve local puzzles. The fistfights are also quite fun, and you don’t really want to use firearms.</li> Doom: The Dark Ages — the legendary demon-slaying series. I only played Doom 1 and Doom 3 before and skipped the entire modern lineup. I decided to start with The Dark Ages. Overall, the first half of the game is interesting: great locations, new weapons constantly unlocking, and so on. But the second half feels tedious — all locations turn into the same gray corridors. The robot titan missions were cool, though.</li> Assassin's Creed Mirage — at first it felt like just another Assassin’s Creed, but the more I played, the better it got. I have warm memories of Baghdad. In my opinion, it’s one of the strongest games in the series. What’s nice is that it’s short. I played it before the recent DLC</a>; people say it’s even better now.</li> Escape from Duckov — a meme game in the extraction genre. I didn’t finish it because it’s quite long, but it’s worth trying to understand the genre. It’s addictive.</li> Lies of P — my first proper soulslike (before that, I only tried Sekiro, which was hell for me). I liked it: the bosses are manageable for a regular player, and there are many ways to beat them (shadow pulls, throwable jars with liquids). The story is simple and interesting, although the playtime feels a bit stretched toward the end.</li> </ul> Other media that stood out. Movies / TV shows: Fly Me to the Moon (2024)</a>, The Fighting Sullivans (1944)</a>, American Beauty (1999)</a>, The Naked Gun (2025)</a>, How to Steal a Million (1966)</a>, Once Upon a Time... in Hollywood (2019)</a>, Bugonia (2025)</a>, Аватар: Путь воды (2022)</a>. Music: Sabrina Carpenter, The Velvet Sundown (yes, AI slop), Alice in Chains. Books: I didn’t read much; the most recent one was Ionesco’s Rhinoceros. Happy New Year to everyone! 🎉



Optimizing RIPEMD-160 with SIMD
2025-05-08T00:00:00+00:00

</p>
I have a hobby project – ecloop</a> – a Bitcoin key "calculator" designed to find Bitcoin puzzles, verify brain wallets, and so on. The mathematical chances of finding a private key from a used address tend toward zero. I'm interested in this program as a collection of tricks involving the elliptic curve (secp256k1) and as a way to practice low-level programming (fast 256-bit arithmetic), so I continue to develop it from time to time.</p>
To calculate a Bitcoin address from a private key, several operations need to be performed:</p>

Calculate the point on the elliptic curve (public key): P = G * PrivKey</code>.</li>
Compute SHA-256 from the compressed public key: (P.y % 2 == 0 ? 0x02 : 0x03) + P.x</code>.</li>
Compute RIPEMD-160 from the resulting SHA-256 hash.</li>
</ol>
The result is the so-called hash160</code>, which is then encoded into a Bitcoin address using either base58</code> or bech32</code>.</p>
As slow as elliptic curve operations are, the slowest part of address generation is the RIPEMD-160 (RMD160) calculation. In fact, it takes about half of the total execution time (SHA-256 is hardware-accelerated on modern processors).</p>
Almost all modern processors have SIMD support: AVX2 on amd64 and Neon on arm64. That's why I decided it would be worthwhile to speed up RMD160 by implementing it using parallel calculations. Moreover, I had never written SIMD code before, so I was interested in trying it out.</p>
My main computer is a MacBook with Apple Silicon (M-series chips). Initially, I wanted to implement RMD160 SIMD using SVE (256-bit / 8-lane), but it turned out that Apple chips do not support SVE 🤦. (To clarify: the M2 chip implements the ARMv8.6 standard, while SVE was introduced in ARMv8.2 as an optional feature). So I had to use Neon (128-bit / 4-lane). If that's not the case and there is a way to run SVE instructions on M-series chips, I'd be happy to hear about it.</p>
What is RMD160?</a></h2>
RIPEMD-160 (RMD160) is a cryptographic hash function that produces a 160-bit hash from arbitrary data. It was developed as a secure alternative to earlier algorithms such as MD5 and SHA-1. RMD160 is widely used in blockchain technology, particularly in Bitcoin, where it is used to generate wallet addresses: the public key is first hashed using SHA-256, and then hashed again with RMD160 to improve security and reduce length.</p>
The RMD160 algorithm consists of five rounds. Each round includes basic logical functions, cyclic shifts (ROTL), and addition. A distinctive feature of RMD160 is that each round is executed in two parallel branches: the main (left) branch and the parallel (right) branch. These two branches use different constants, word orders, and logic functions, and their results are later combined.</p>
There is a classic</a> C implementation</a> of RMD160 that uses a bunch of macros to define rounds, logical functions, and so on—but such code is hard to read. So I prefer the Golang implementation</a>, which I previously ported to ecloop</code>. I plan to continue the porting to Neon based on this codebase.</p>
Stepping back a bit, SIMD instructions aren't extremely complex, but they lack "syntactic sugar" — so instead of writing a + b</code>, you need to write something like vaddq_u32(a, b)</code>. There are special functions like this for every standard operation multiplied by the number of numeric types (u/i 8/16/32/64, f16/32/64).</p>
RMD160 (like other hash functions) should not be too difficult to port to SIMD because their algorithms contain no branching. Essentially, the logic stays the same; it's just a matter of replacing all operations with SIMD-specific instructions.</p>
Simple Neon Program</a></h2>
To understand how to write using Neon, we should start with the simplest possible program, such as multiplying 42 × 2. Since SIMD involves parallel computations, its operations are applied to the entire vector at once, and the result should be the same across all parts of the vector. To verify this, the result can be printed to the console.</p>
#include </span><</span>arm_neon.h</span>>
</span>#include </span><</span>stdint.h</span>>
</span>#include </span><</span>stdio.h</span>>
</span>
</span>void </span>print_check</span>(uint32x4_t *</span>a</span>) {
</span>  uint32_t arr[</span>4</span>];
</span>  </span>vst1q_u32</span>(arr, *a); </span>// store 4x32-bit vector into a regular array
</span>  </span>for </span>(</span>int</span> i = </span>0</span>; i < </span>4</span>; i++) {
</span>    </span>printf</span>("</span>%x%c</span>", arr[i], i == </span>3 </span>? '</span>\n</span>' : ' ');
</span>  }
</span>}
</span>
</span>int </span>main</span>() {
</span>  uint32_t a = </span>42</span>;
</span>  uint32x4_t b = </span>vdupq_n_u32</span>(</span>42</span>); </span>// load u32 to all 4 lanes (42, 42, 42, 42)
</span>  </span>printf</span>("</span>%x</span> = </span>", a);
</span>  </span>print_check</span>(&b); </span>// out: 2a = 2a 2a 2a 2a
</span>
</span>  a = a * </span>2</span>;
</span>  b = </span>vmulq_n_u32</span>(b, </span>2</span>); </span>// multiply each lane by 2
</span>  </span>printf</span>("</span>%x</span> = </span>", a);
</span>  </span>print_check</span>(&b); </span>// out: 54 = 54 54 54 54
</span>
</span>  </span>return </span>0</span>;
</span>}
</span></code></pre>
In general, I think the concept of how SIMD calculations work is clear. From here on, print_check</code> will be used frequently to check the correctness of the hashing algorithm.</p>
Basic Functions and ROTL</a></h2>
The RMD160 calculation uses 5 basic functions and ROTL; everything else is a shuffling of data in a specific order. GPT has rewritten these macros, and I have verified that they are correct:</p>
// original functions
</span>#define </span>OLD_F1</span>(</span>x</span>, </span>y</span>, </span>z</span>) ((x) ^ (y) ^ (z))
</span>#define </span>OLD_F2</span>(</span>x</span>, </span>y</span>, </span>z</span>) (((x) & (y)) | (~(x) & (z)))
</span>#define </span>OLD_F3</span>(</span>x</span>, </span>y</span>, </span>z</span>) (((x) | ~(y)) ^ (z))
</span>#define </span>OLD_F4</span>(</span>x</span>, </span>y</span>, </span>z</span>) (((x) & (z)) | ((y) & ~(z)))
</span>#define </span>OLD_F5</span>(</span>x</span>, </span>y</span>, </span>z</span>) ((x) ^ ((y) | ~(z)))
</span>#define </span>OLD_ROTL</span>(</span>x</span>, </span>n</span>) (((x) << (n)) | ((x) >> (</span>32 </span>- (n))))
</span>
</span>// simd functions
</span>#define </span>F1</span>(</span>x</span>, </span>y</span>, </span>z</span>) </span>veorq_u32</span>(</span>veorq_u32</span>(x, y), z)
</span>#define </span>F2</span>(</span>x</span>, </span>y</span>, </span>z</span>) </span>vorrq_u32</span>(</span>vandq_u32</span>(x, y), </span>vandq_u32</span>(</span>vmvnq_u32</span>(x), z))
</span>#define </span>F3</span>(</span>x</span>, </span>y</span>, </span>z</span>) </span>veorq_u32</span>(</span>vorrq_u32</span>(x, </span>vmvnq_u32</span>(y)), z)
</span>#define </span>F4</span>(</span>x</span>, </span>y</span>, </span>z</span>) </span>vorrq_u32</span>(</span>vandq_u32</span>(x, z), </span>vandq_u32</span>(y, </span>vmvnq_u32</span>(z)))
</span>#define </span>F5</span>(</span>x</span>, </span>y</span>, </span>z</span>) </span>veorq_u32</span>(x, </span>vorrq_u32</span>(y, </span>vmvnq_u32</span>(z)))
</span>#define </span>ROTL</span>(</span>x</span>, </span>n</span>) </span>vorrq_u32</span>(</span>vshlq_n_u32</span>(x, n), </span>vshrq_n_u32</span>(x, </span>32 </span>- (n)))
</span>
</span>void </span>print_check</span>(</span>char </span>*</span>l</span>, uint32_t </span>c</span>, uint32x4_t </span>a</span>) {
</span>  </span>printf</span>("</span>%s</span>: </span>%08x</span> = </span>", l, c);
</span>  uint32_t arr[</span>4</span>];
</span>  </span>vst1q_u32</span>(arr, a); </span>// store 4x32-bit vector into a regular array
</span>  </span>for </span>(</span>int</span> i = </span>0</span>; i < </span>4</span>; i++) {
</span>    </span>printf</span>("</span>%08x%c</span>", arr[i], i == </span>3 </span>? '</span>\n</span>' : ' ');
</span>    </span>// assert(arr[i] == c);
</span>  }
</span>}
</span>
</span>uint32_t a1, b1, c1;
</span>uint32x4_t a2, b2, c2;
</span>
</span>a1 = </span>0x67452301</span>, b1 = </span>0xefcdab89</span>, c1 = </span>0x98badcfe</span>;
</span>a2 = </span>vdupq_n_u32</span>(a1), b2 = </span>vdupq_n_u32</span>(b1), c2 = </span>vdupq_n_u32</span>(c1); </span>// loading vectors
</span>
</span>// compare original and simd functions
</span>print_check</span>("</span>F1</span>", </span>OLD_F1</span>(a1, b1, c1), </span>F1</span>(a2, b2, c2));
</span>print_check</span>("</span>F2</span>", </span>OLD_F2</span>(a1, b1, c1), </span>F2</span>(a2, b2, c2));
</span>print_check</span>("</span>F3</span>", </span>OLD_F3</span>(a1, b1, c1), </span>F3</span>(a2, b2, c2));
</span>print_check</span>("</span>F4</span>", </span>OLD_F4</span>(a1, b1, c1), </span>F4</span>(a2, b2, c2));
</span>print_check</span>("</span>F5</span>", </span>OLD_F5</span>(a1, b1, c1), </span>F5</span>(a2, b2, c2));
</span>print_check</span>("</span>RL</span>", </span>OLD_ROTL</span>(a1, </span>12</span>), </span>ROTL</span>(a2, </span>12</span>));
</span>
</span>// output:
</span>// F1: 10325476 = 10325476 10325476 10325476 10325476
</span>// F2: ffffffff = ffffffff ffffffff ffffffff ffffffff
</span>// F3: efcdab89 = efcdab89 efcdab89 efcdab89 efcdab89
</span>// F4: 67452301 = 67452301 67452301 67452301 67452301
</span>// F5: 88888888 = 88888888 88888888 88888888 88888888
</span>// RL: 52301674 = 52301674 52301674 52301674 52301674
</span></code></pre>
Problem with Porting Golang Implementation</a></h2>
The Golang implementation consists of 5 large loops, each performing 16 left and right rounds. Inside each operation, the input data is mixed at index _n[i]</code>, and ROTL is performed at index _r[i]</code>. The reference C implementation uses a series of consecutive macros, which makes it harder to read (in my opinion).</p>
// Golang implementation
</span>static const</span> u8 _n[</span>80</span>] = { </span>/* ... */ </span>}; </span>// Left DATA indexes
</span>static const</span> u8 _r[</span>80</span>] = { </span>/* ... */ </span>}; </span>// Left ROTL indexes
</span>
</span>// round 1
</span>for </span>(; i < </span>16</span>; ++i) {
</span>  </span>// left branch
</span>  alpha = a1 + </span>F1</span>(b1, c1, d1) + x[_n[i]];
</span>  alpha = </span>rotl32</span>(alpha, _r[i]) + e1;
</span>  beta = </span>rotl32</span>(c1, </span>10</span>);
</span>  a1 = e1, c1 = b1, e1 = d1, b1 = alpha, d1 = beta;
</span>  </span>// right branch
</span>  </span>// ...
</span>}
</span>
</span>// Reference C-implementation
</span>#define </span>F</span>(</span>x</span>, </span>y</span>, </span>z</span>)        ((x) ^ (y) ^ (z))
</span>#define </span>FF</span>(</span>a</span>, </span>b</span>, </span>c</span>, </span>d</span>, </span>e</span>, </span>x</span>, </span>s</span>)        {\
</span>      (a) += </span>F</span>((b), (c), (d)) + (x);\
</span>      (a) = </span>ROL</span>((a), (s)) + (e);\
</span>      (c) = </span>ROL</span>((c), </span>10</span>);\
</span>   }
</span>
</span>// round 1 - left branch
</span>FF</span>(aa, bb, cc, dd, ee, X[ </span>0</span>], </span>11</span>);
</span>FF</span>(ee, aa, bb, cc, dd, X[ </span>1</span>], </span>14</span>);
</span>// ...
</span>FF</span>(bb, cc, dd, ee, aa, X[</span>14</span>],  </span>9</span>);
</span>FF</span>(aa, bb, cc, dd, ee, X[</span>15</span>],  </span>8</span>);
</span></code></pre>
In general, if we rewrite the code directly using Neon instructions, the compilation crashes with an error stating that vshlq_n_u32</code> and vshrq_n_u32</code> require a known rotation value (the second argument) at compile time. See:</p>
// Golang implementation (original)
</span>#define </span>F1</span>(</span>x</span>, </span>y</span>, </span>z</span>) ((x) ^ (y) ^ (z))
</span>#define </span>rotl32</span>(</span>x</span>, </span>n</span>) (((x) << (n)) | ((x) >> (</span>32 </span>- (n))))
</span>alpha = a1 + </span>F1</span>(b1, c1, d1) + x[_n[i]];
</span>alpha = </span>rotl32</span>(alpha, _r[i]) + e1;
</span>
</span>// Golang implementation (SIMD)
</span>#define </span>F1</span>(</span>x</span>, </span>y</span>, </span>z</span>) </span>veorq_u32</span>(</span>veorq_u32</span>(x, y), z)
</span>#define </span>ROTL</span>(</span>x</span>, </span>n</span>) </span>vorrq_u32</span>(</span>vshlq_n_u32</span>(x, n), </span>vshrq_n_u32</span>(x, </span>32 </span>- (n)))
</span>alpha = </span>vaddq_u32</span>(a1, </span>F1</span>(b1, c1, d1));
</span>alpha = </span>vaddq_u32</span>(alpha, X[_n[i]]);
</span>alpha = </span>vaddq_u32</span>(</span>ROTL</span>(alpha, _r[i]), e1);
</span>// err: argument to '__builtin_neon_vshlq_n_v' must be a constant integer
</span>// err: argument to '__builtin_neon_vshrq_n_v' must be a constant integer
</span></code></pre>
So, I have to use the macro version (C reference) because, in that case, the indexes are passed directly (the last argument in the FF macro) and expanded into constant values at compile time. Maybe this change is for the better (we will see why later).</p>
Generic Round Macro</a></h2>
If we look at RMD160 rounds, the same actions are performed, but with the following changes: base function, constant, data index, and rotation. In general, the round macro was shown above (I took a slightly different version from GitHub). My goal is to port the round macro to SIMD.</p>
In the round, we add up 4 variables, perform a ROTL + another addition, and a separate ROTL for another variable. Since the "+" operation is not available in SIMD, we need to use special instructions.</p>
I added some macros for vector addition and described the round itself:</p>
#define </span>ADD2</span>(</span>a</span>, </span>b</span>) </span>vaddq_u32</span>(a, b)
</span>#define </span>ADD3</span>(</span>a</span>, </span>b</span>, </span>c</span>) </span>vaddq_u32</span>(</span>vaddq_u32</span>(a, b), c)
</span>#define </span>ADD4</span>(</span>a</span>, </span>b</span>, </span>c</span>, </span>d</span>) </span>vaddq_u32</span>(</span>vaddq_u32</span>(</span>vaddq_u32</span>(a, b), c), d)
</span>
</span>#define </span>RN</span>(</span>a</span>, </span>b</span>, </span>c</span>, </span>d</span>, </span>e</span>, </span>f</span>, </span>x</span>, </span>k</span>, </span>r</span>) \
</span>  u = </span>ADD4</span>(a, f, x, </span>vdupq_n_u32</span>(k));  \
</span>  a = </span>ADD2</span>(</span>ROTL</span>(u, r), e);            \
</span>  c = </span>ROTL</span>(c, </span>10</span>);
</span></code></pre>
In the macro, a</code>, b</code>, c</code>, d</code>, e</code> are the state variables, f</code> is the value after the base function computation, x</code> is the uint32 index data for the current iteration, k</code> is a constant, and r</code> is the rotation value for ROTL.</p>
vdupq_n_u32(k)</code> loads a constant into a vector (the same value in all 4 lanes). Earlier, we wrote code to multiply a vector by a number, and for this purpose, vmulq_n_u32</code></a> is used. Logically, the instruction to add a number to a vector should be vaddq_n_u32</code>, but it does not exist</a>. Instead, it should be written in vaddq_u32(vec1, vdupq_n_u32(2))</code> style (if anyone knows why this is the case – please leave a comment).</p>
Then, based on this round macro, we can define left and right rounds. The code here is similar to any other macro implementation (except that I named the rounds as Li</code>/Ri</code>).</p>
#define </span>L1</span>(</span>a</span>, </span>b</span>, </span>c</span>, </span>d</span>, </span>e</span>, </span>x</span>, </span>r</span>) </span>RN</span>(a, b, c, d, e, </span>F1</span>(b, c, d), x, </span>0</span>, r)
</span>#define </span>L2</span>(</span>a</span>, </span>b</span>, </span>c</span>, </span>d</span>, </span>e</span>, </span>x</span>, </span>r</span>) </span>RN</span>(a, b, c, d, e, </span>F2</span>(b, c, d), x, </span>0x5A827999</span>ul</span>, r)
</span>#define </span>L3</span>(</span>a</span>, </span>b</span>, </span>c</span>, </span>d</span>, </span>e</span>, </span>x</span>, </span>r</span>) </span>RN</span>(a, b, c, d, e, </span>F3</span>(b, c, d), x, </span>0x6ED9EBA1</span>ul</span>, r)
</span>#define </span>L4</span>(</span>a</span>, </span>b</span>, </span>c</span>, </span>d</span>, </span>e</span>, </span>x</span>, </span>r</span>) </span>RN</span>(a, b, c, d, e, </span>F4</span>(b, c, d), x, </span>0x8F1BBCDC</span>ul</span>, r)
</span>#define </span>L5</span>(</span>a</span>, </span>b</span>, </span>c</span>, </span>d</span>, </span>e</span>, </span>x</span>, </span>r</span>) </span>RN</span>(a, b, c, d, e, </span>F5</span>(b, c, d), x, </span>0xA953FD4E</span>ul</span>, r)
</span>#define </span>R1</span>(</span>a</span>, </span>b</span>, </span>c</span>, </span>d</span>, </span>e</span>, </span>x</span>, </span>r</span>) </span>RN</span>(a, b, c, d, e, </span>F5</span>(b, c, d), x, </span>0x50A28BE6</span>ul</span>, r)
</span>#define </span>R2</span>(</span>a</span>, </span>b</span>, </span>c</span>, </span>d</span>, </span>e</span>, </span>x</span>, </span>r</span>) </span>RN</span>(a, b, c, d, e, </span>F4</span>(b, c, d), x, </span>0x5C4DD124</span>ul</span>, r)
</span>#define </span>R3</span>(</span>a</span>, </span>b</span>, </span>c</span>, </span>d</span>, </span>e</span>, </span>x</span>, </span>r</span>) </span>RN</span>(a, b, c, d, e, </span>F3</span>(b, c, d), x, </span>0x6D703EF3</span>ul</span>, r)
</span>#define </span>R4</span>(</span>a</span>, </span>b</span>, </span>c</span>, </span>d</span>, </span>e</span>, </span>x</span>, </span>r</span>) </span>RN</span>(a, b, c, d, e, </span>F2</span>(b, c, d), x, </span>0x7A6D76E9</span>ul</span>, r)
</span>#define </span>R5</span>(</span>a</span>, </span>b</span>, </span>c</span>, </span>d</span>, </span>e</span>, </span>x</span>, </span>r</span>) </span>RN</span>(a, b, c, d, e, </span>F1</span>(b, c, d), x, </span>0</span>, r)
</span></code></pre>
Now, using these macros, we can write the first iteration of the first round and compare it with the working implementation. If everything is OK, we can copy the entire round, check it, and then move on to the remaining rounds. I compared the result with the print_check</code> function I created earlier.</p>
First left round, first iteration:</p>
// RMD160 initial constants
</span>#define </span>K1 </span>0x67452301
</span>#define </span>K2 </span>0xEFCDAB89
</span>#define </span>K3 </span>0x98BADCFE
</span>#define </span>K4 </span>0x10325476
</span>#define </span>K5 </span>0xC3D2E1F0
</span>
</span>void </span>rmd160_block</span>(uint32x4_t *</span>s</span>, </span>const </span>uint32_t </span>x</span>[</span>4</span>][</span>16</span>]) {
</span>  </span>// a1-e1 left rounds state, a2-e2 right rounds state, u - temp varible used in RD macro
</span>  uint32x4_t a1, b1, c1, d1, e1, a2, b2, c2, d2, e2, u;
</span>  </span>// Load initial constants
</span>  a1 = a2 = </span>vdupq_n_u32</span>(K1);
</span>  b1 = b2 = </span>vdupq_n_u32</span>(K2);
</span>  c1 = c2 = </span>vdupq_n_u32</span>(K3);
</span>  d1 = d2 = </span>vdupq_n_u32</span>(K4);
</span>  e1 = e2 = </span>vdupq_n_u32</span>(K5);
</span>
</span>  uint32x4_t w[</span>16</span>]; </span>// Load data to vector
</span>  </span>for </span>(</span>int</span> i = </span>0</span>; i < </span>16</span>; i++) {
</span>    </span>// Load 4x32-bit integers from x[0][i], x[1][i], x[2][i], x[3][i]
</span>    </span>// w[i] = vsetq_lane_u32(x[0][i], w[i], 0);
</span>    </span>// w[i] = vsetq_lane_u32(x[1][i], w[i], 1);
</span>    </span>// w[i] = vsetq_lane_u32(x[2][i], w[i], 2);
</span>    </span>// w[i] = vsetq_lane_u32(x[3][i], w[i], 3);
</span>    w[i] = </span>vld1q_u32</span>(((uint32_t[</span>4</span>]){x[</span>0</span>][i], x[</span>1</span>][i], x[</span>2</span>][i], x[</span>3</span>][i]})); </span>// A bit faster
</span>  }
</span>
</span>  </span>L1</span>(a1, b1, c1, d1, e1, w[</span>0</span>], </span>11</span>);
</span>  </span>print_check</span>("</span>a1</span>", </span>0</span>, a1);
</span>  </span>print_check</span>("</span>b1</span>", </span>0</span>, b1);
</span>  </span>print_check</span>("</span>c1</span>", </span>0</span>, c1);
</span>  </span>print_check</span>("</span>d1</span>", </span>0</span>, d1);
</span>  </span>print_check</span>("</span>e1</span>", </span>0</span>, e1);
</span>}
</span>
</span>uint32x4_t s[</span>5</span>] = {</span>0</span>}; </span>// initial state
</span>s[</span>0</span>] = </span>vdupq_n_u32</span>(K1);
</span>s[</span>1</span>] = </span>vdupq_n_u32</span>(K2);
</span>s[</span>2</span>] = </span>vdupq_n_u32</span>(K3);
</span>s[</span>3</span>] = </span>vdupq_n_u32</span>(K4);
</span>s[</span>4</span>] = </span>vdupq_n_u32</span>(K5);
</span>
</span>uint32_t x[</span>4</span>][</span>16</span>] = {</span>0</span>}; </span>// data block, filled with zeros
</span>rmd160_block</span>((uint32x4_t *)s, x);
</span></code></pre>
It should be noted that hash functions are usually tested on zero data (for simplicity). Data in hash functions are processed in blocks. A block in RMD160 is 32×16 = 512 bits. rmd160_block</code> can be called several times with the same state (which changes) and new data, for cases where you need to calculate the hash of a message larger than one round. In my task (address generation), all messages are placed in one block. The result of the first round compare to current implementation:</p>
// a1: 1602f864 1602f864 1602f864 1602f864 vs c3d2e1f0
</span>// a1: efcdab89 efcdab89 efcdab89 efcdab89 vs 1602f864
</span>// a1: eb73fa62 eb73fa62 eb73fa62 eb73fa62 vs efcdab89
</span>// a1: 10325476 10325476 10325476 10325476 vs eb73fa62
</span>// a1: c3d2e1f0 c3d2e1f0 c3d2e1f0 c3d2e1f0 vs 10325476
</span></code></pre>
In general, these values ± are similar to the values from the current version, differing by one offset. This is not a problem, since there are 5 variables, and the offsets will be aligned by the end. It's just a difference in the implementations.</p>
I won't write all the rounds, as there are 80 on each side (160 total) – it would result in an overly large code block. Left and right rounds are independent of each other and can be calculated in any order: either first all left / all right, or alternating left / right, or alternating iterations within a round. This will not affect the final result.</p>
Finalizing RMD160</a></h2>
At the end of RMD160 block, we need to combine the old state with the local state – this also involves three additions with index offsets.</p>
void </span>rmd160_block</span>(uint32x4_t *</span>s</span>, </span>const </span>uint32_t </span>x</span>[</span>4</span>][</span>16</span>]) {
</span>  </span>// ... 160 rounds
</span>
</span>  uint32x4_t t = s[</span>0</span>];
</span>  s[</span>0</span>] = </span>ADD3</span>(s[</span>1</span>], c1, d2);
</span>  s[</span>1</span>] = </span>ADD3</span>(s[</span>2</span>], d1, e2);
</span>  s[</span>2</span>] = </span>ADD3</span>(s[</span>3</span>], e1, a2);
</span>  s[</span>3</span>] = </span>ADD3</span>(s[</span>4</span>], a1, b2);
</span>  s[</span>4</span>] = </span>ADD3</span>(t, b1, c2);
</span>}
</span></code></pre>
The final step is to change the endianness of the values (RMD160 uses a different endianness) and unload the values from the vector into the resulting array.</p>
// ... init & rmd160_block
</span>
</span>for </span>(</span>int</span> i = </span>0</span>; i < </span>5</span>; ++i) {
</span>  </span>// swap32 for uint32x4_t (can it be shorter?)
</span>  s[i] = </span>vreinterpretq_u32_u8</span>(</span>vrev32q_u8</span>(</span>vreinterpretq_u8_u32</span>(s[i])));
</span>}
</span>
</span>uint32_t r[</span>4</span>][</span>5</span>] = {</span>0</span>}; </span>// result stored as 4x5 uint32_t
</span>for </span>(</span>int</span> i = </span>0</span>; i < </span>5</span>; i++) { </span>// load it from uint32x4_t
</span>  r[</span>0</span>][i] = </span>vgetq_lane_u32</span>(s[i], </span>0</span>);
</span>  r[</span>1</span>][i] = </span>vgetq_lane_u32</span>(s[i], </span>1</span>);
</span>  r[</span>2</span>][i] = </span>vgetq_lane_u32</span>(s[i], </span>2</span>);
</span>  r[</span>3</span>][i] = </span>vgetq_lane_u32</span>(s[i], </span>3</span>);
</span>}
</span></code></pre>
That is all – r</code> can be used further where needed (r[0]</code>, r[1]</code>, r[2]</code>, r[3]</code> are the computed hashes — four at once).</p>
To summarize this section, the complete parallel RMD160 algorithm looks like this:</p>

Initialize a state of length 160 bits × 4 lanes (uint32x4_t s[5]</code>).</li>
Split the message (data) into blocks of 512 bits × 4 lanes (uint32_t x[4][16]</code>).</li>
Iterate RMD160 rounds until the data runs out (rmd160_block</code> reads the data into the vector itself).</li>
Change the endianness in the final state.</li>
Unload the vector of the final state into a hash array (int32_t r[4][5]</code>).</li>
</ol>
Performance of RMD160 SIMD</a></h2>
Now, it's time to measure the performance of this code – to compare the performance of the original and SIMD implementations. For this purpose, I created a small benchmark:</p>
size_t </span>tsnow</span>() {
</span>  </span>struct</span> timespec ts;
</span>  </span>clock_gettime</span>(CLOCK_REALTIME, &ts);
</span>  </span>return</span> ts.</span>tv_sec </span>* </span>1000 </span>+ ts.</span>tv_nsec </span>/ </span>1e6</span>;
</span>}
</span>
</span>void </span>rmd160_simd</span>() {
</span>  uint32_t r[</span>4</span>][</span>5</span>] = {</span>0</span>};
</span>  uint32_t x[</span>4</span>][</span>16</span>] = {</span>0</span>};
</span>
</span>  size_t stime = </span>tsnow</span>();
</span>  size_t iters = </span>1000 </span>* </span>1000 </span>* </span>32</span>;
</span>  </span>for </span>(size_t i = </span>0</span>; i < iters; ++i) </span>rmd160_4w</span>(r, x);
</span>  </span>double</span> dt = (</span>tsnow</span>() - stime) / </span>1000.0</span>;
</span>  </span>double</span> ir = iters / dt / </span>1000000</span>;
</span>  </span>double</span> hr = ir * </span>4</span>; </span>// 4 hash per iter
</span>  </span>printf</span>("</span>%.2f</span>M it/s ~ </span>%.2f</span>M h/s ~ </span>%.2f</span>s</span>\n</span>", ir, hr, dt);
</span>
</span>  </span>printf</span>("</span>s[0]: </span>%08x</span>\n</span>", r[</span>0</span>][</span>0</span>]);
</span>  </span>printf</span>("</span>s[1]: </span>%08x</span>\n</span>", r[</span>0</span>][</span>1</span>]);
</span>  </span>printf</span>("</span>s[2]: </span>%08x</span>\n</span>", r[</span>0</span>][</span>2</span>]);
</span>  </span>printf</span>("</span>s[3]: </span>%08x</span>\n</span>", r[</span>0</span>][</span>3</span>]);
</span>  </span>printf</span>("</span>s[4]: </span>%08x</span>\n</span>", r[</span>0</span>][</span>4</span>]);
</span>}
</span>
</span>void </span>rmd160_naive</span>() {
</span>  uint32_t s[</span>5</span>] = {</span>0</span>};
</span>  uint32_t x[</span>16</span>] = {</span>0</span>};
</span>
</span>  size_t stime = </span>tsnow</span>();
</span>  size_t iters = </span>1000 </span>* </span>1000 </span>* </span>32</span>;
</span>  </span>for </span>(size_t i = </span>0</span>; i < iters; ++i) </span>rmd160_1w</span>(s, x);
</span>  </span>double</span> dt = (</span>tsnow</span>() - stime) / </span>1000.0</span>;
</span>  </span>double</span> ir = iters / dt / </span>1000000</span>;
</span>  </span>double</span> hr = ir * </span>1</span>; </span>// 1 hash per iter
</span>  </span>printf</span>("</span>%.2f</span>M it/s ~ </span>%.2f</span>M h/s ~ </span>%.2f</span>s</span>\n</span>", ir, hr, dt);
</span>
</span>  </span>printf</span>("</span>s[0]: </span>%08x</span>\n</span>", s[</span>0</span>]);
</span>  </span>printf</span>("</span>s[1]: </span>%08x</span>\n</span>", s[</span>1</span>]);
</span>  </span>printf</span>("</span>s[2]: </span>%08x</span>\n</span>", s[</span>2</span>]);
</span>  </span>printf</span>("</span>s[3]: </span>%08x</span>\n</span>", s[</span>3</span>]);
</span>  </span>printf</span>("</span>s[4]: </span>%08x</span>\n</span>", s[</span>4</span>]);
</span>}
</span></code></pre>
I compiled both programs with -O3</code> and ran them (on a basic Apple M2):</p>
❯</span> clang</span> -O3 -march</span>=native ./lib/rmd160.c && </span>./a.out </span># original
</span>5.50M</span> it/s </span>~</span> 5.50M h/s </span>~</span> 5.81s
</span>
</span>❯</span> clang</span> -O3 -march</span>=native ./lib/rmd160s.c && </span>./a.out </span># neon
</span>2.14M</span> it/s </span>~</span> 8.55M h/s </span>~</span> 14.98s
</span></code></pre>
The Neon version (128-bit / 4 lanes) is 55% faster. This is a great result, but it's unfortunate that M-chips don't have SVE for 256/512 bits (8/16 lanes), as that would make it even better!</p>
One More Thing</a></h2>
While I was writing about the results above, I became curious</strong> to see what would happen if I changed the order of the rounds in RMD160. The original order of rounds was as follows: first all left rounds, then all right rounds. I thought this was good for the processor, because it seemed to require less "context switching".</p>
I changed the order of rounds to alternating left and right rounds (L1 / R1, L2 / R2), and the performance increased significantly. Initially, I thought there was a data error, but print_check</code> (as tests) confirms everything is fine.</p>
I decided to try alternating iterations (L1_1 R1_1 L1_2 R1_2 R1_2, etc.). To be honest, rearranging 160 lines is not the most fun, but the result surprised me even more.</p>
Comparison of different round/iteration placements (3 tests of RMD160 function itself and a full cycle of ecloop</code> logic):</p>
# L1_1 L2_2 .. L5_16 R1_1 R2_2 .. R5_16 (+56%)
</span>2.25M</span> it/s </span>~</span> 9.02M h/s </span>~</span> 14.19s
</span>2.23M</span> it/s </span>~</span> 8.93M h/s </span>~</span> 14.33s
</span>2.23M</span> it/s </span>~</span> 8.94M h/s </span>~</span> 14.32s
</span>ecloop</span> (addr33 x 8 core) </span>~</span> 19.53M it/s (+22%)
</span>
</span># L1_1-L1_16 R1_1-R1_16 L2_1-L2_16 .. (+165%)
</span>3.70M</span> it/s </span>~</span> 14.80M h/s </span>~</span> 8.65s
</span>3.85M</span> it/s </span>~</span> 15.42M h/s </span>~</span> 8.30s
</span>3.87M</span> it/s </span>~</span> 15.46M h/s </span>~</span> 8.28s
</span>ecloop</span> (addr33 x 8 core) </span>~</span> 22.46M it/s (+40%)
</span>
</span># L1_1 R1_1 L1_2 R1_2 .. L5_16 R5_16 (+175%)
</span>3.96M</span> it/s </span>~</span> 15.82M h/s </span>~</span> 8.09s
</span>3.94M</span> it/s </span>~</span> 15.78M h/s </span>~</span> 8.11s
</span>3.94M</span> it/s </span>~</span> 15.76M h/s </span>~</span> 8.12s
</span>ecloop</span> (addr33 x 8 core) </span>~</span> 24.83M it/s (+55%)
</span></code></pre>
It's a mystery to me why it works this way, and maybe there is an even more effective arrangement. Who knows? Please, write in the comments.</p>
Support for AVX2 (AMD64)</a></h2>
Initially, I didn't plan to implement this, but RMD160 algorithm using macros was quite abstract, and further porting to AVX2 seemed fairly simple. The main difference between AVX2 and Neon (aside from the different instruction sets) is the vector size – 256 bits vs 128 bits – which allows us to process 8 hashes in parallel (vs 4 in Neon).</p>
Currently, the following Neon instructions are used directly in the algorithm code: vector type (uint32x4_t</code>), state initialization via vdupq_n_u32</code>, endian-swap, and load / dump data into the vector.</p>
I'm moving these things to macros (just in case, I added RMD_</code> prefix to avoid conflicts with other files):</p>
#define </span>RMD_LEN </span>4                    </span>// vector length
</span>#define </span>RMD_VEC uint32x4_t           </span>// vector type
</span>#define </span>RMD_LD_NUM</span>(</span>x</span>) </span>vdupq_n_u32</span>(x) </span>// load same number into all lanes
</span>
</span>#define </span>RMD_SWAP</span>(</span>x</span>) </span>vreinterpretq_u32_u8</span>(</span>vrev32q_u8</span>(</span>vreinterpretq_u8_u32</span>(x)))
</span>#define </span>RMD_LOAD</span>(</span>x</span>, </span>i</span>) </span>vld1q_u32</span>(((uint32_t[</span>4</span>]){x[</span>0</span>][i], x[</span>1</span>][i], x[</span>2</span>][i], x[</span>3</span>][i]}))
</span>#define </span>RMD_DUMP</span>(</span>r</span>, </span>s</span>, </span>i</span>)                                                                        \
</span>  </span>do </span>{                                                                                           \
</span>    r[</span>0</span>][i] = </span>vgetq_lane_u32</span>(s[i], </span>0</span>);                                                           \
</span>    r[</span>1</span>][i] = </span>vgetq_lane_u32</span>(s[i], </span>1</span>);                                                           \
</span>    r[</span>2</span>][i] = </span>vgetq_lane_u32</span>(s[i], </span>2</span>);                                                           \
</span>    r[</span>3</span>][i] = </span>vgetq_lane_u32</span>(s[i], </span>3</span>);                                                           \
</span>  } </span>while </span>(</span>0</span>);
</span></code></pre>
And update the current code to something like this:</p>
void </span>rmd160_block</span>(RMD_VEC *</span>s</span>, </span>const </span>uint32_t </span>x</span>[RMD_LEN][</span>16</span>]) {
</span>  RMD_VEC a1, b1, c1, d1, e1, a2, b2, c2, d2, e2, u;
</span>  a1 = a2 = </span>RMD_LD_NUM</span>(RMD_K1);
</span>  b1 = b2 = </span>RMD_LD_NUM</span>(RMD_K2);
</span>  c1 = c2 = </span>RMD_LD_NUM</span>(RMD_K3);
</span>  d1 = d2 = </span>RMD_LD_NUM</span>(RMD_K4);
</span>  e1 = e2 = </span>RMD_LD_NUM</span>(RMD_K5);
</span>
</span>  RMD_VEC w[</span>16</span>];
</span>  </span>for </span>(</span>int</span> i = </span>0</span>; i < </span>16</span>; i++) w[i] = </span>RMD_LOAD</span>(x, i);
</span>  </span>// ... rounds and iterations
</span>}
</span>
</span>// new function to process full single block
</span>void </span>rmd160_batch</span>(uint32_t </span>r</span>[RMD_LEN][</span>5</span>], </span>const </span>uint32_t </span>x</span>[RMD_LEN][</span>16</span>]) {
</span>  RMD_VEC s[</span>5</span>] = {</span>0</span>}; </span>// load initial state
</span>  s[</span>0</span>] = </span>RMD_LD_NUM</span>(RMD_K1);
</span>  s[</span>1</span>] = </span>RMD_LD_NUM</span>(RMD_K2);
</span>  s[</span>2</span>] = </span>RMD_LD_NUM</span>(RMD_K3);
</span>  s[</span>3</span>] = </span>RMD_LD_NUM</span>(RMD_K4);
</span>  s[</span>4</span>] = </span>RMD_LD_NUM</span>(RMD_K5);
</span>
</span>  </span>rmd160_block</span>((RMD_VEC *)s, x);                     </span>// round
</span>  </span>for </span>(</span>int</span> i = </span>0</span>; i < </span>5</span>; ++i) s[i] = </span>RMD_SWAP</span>(s[i]); </span>// change endian
</span>  </span>for </span>(</span>int</span> i = </span>0</span>; i < </span>5</span>; ++i) </span>RMD_DUMP</span>(r, s, i);     </span>// dump data to array
</span>}
</span></code></pre>
Of course, it's already quite magical, but there aren't really many changes. Now, I should add redefined macros for AVX2. In addition, I wrapped a series of architecture-specific macros in #ifdef</code>. In fact, I have a single codebase for RMD160 algorithm, and the necessary macros are included depending on which processor the program is being compiled for.</p>
#if defined</span>(__aarch64__) && </span>defined</span>(__ARM_NEON)
</span>  </span>#include </span><</span>arm_neon.h</span>>
</span>
</span>  </span>#define </span>RMD_LEN </span>4                    </span>// vector length
</span>  </span>#define </span>RMD_VEC uint32x4_t           </span>// vector type
</span>  </span>// ... move all current Neon related macros here
</span>#elif defined</span>(__x86_64__) && </span>defined</span>(__AVX2__)
</span>  </span>#include </span><</span>immintrin.h</span>>
</span>
</span>  </span>#define </span>RMD_LEN </span>8
</span>  </span>#define </span>RMD_VEC __m256i
</span>  </span>#define </span>RMD_LD_NUM</span>(</span>x</span>) </span>_mm256_set1_epi32</span>(x)
</span>
</span>  </span>#define </span>RMD_SWAP</span>(</span>x</span>)                                                                              \
</span>    </span>_mm256_shuffle_epi8</span>((x), </span>_mm256_setr_epi8</span>(</span>3</span>, </span>2</span>, </span>1</span>, </span>0</span>, </span>7</span>, </span>6</span>, </span>5</span>, </span>4</span>, </span>11</span>, </span>10</span>, </span>9</span>, </span>8</span>, </span>15</span>, </span>14</span>, </span>13</span>,    \
</span>                                              </span>12</span>, </span>19</span>, </span>18</span>, </span>17</span>, </span>16</span>, </span>23</span>, </span>22</span>, </span>21</span>, </span>20</span>, </span>27</span>, </span>26</span>, </span>25</span>, </span>24</span>,  \
</span>                                              </span>31</span>, </span>30</span>, </span>29</span>, </span>28</span>))
</span>
</span>  </span>#define </span>RMD_LOAD</span>(</span>x</span>, </span>i</span>)                                                                           \
</span>    </span>_mm256_set_epi32</span>(x[</span>0</span>][i], x[</span>1</span>][i], x[</span>2</span>][i], x[</span>3</span>][i], x[</span>4</span>][i], x[</span>5</span>][i], x[</span>6</span>][i], x[</span>7</span>][i])
</span>
</span>  </span>#define </span>RMD_DUMP</span>(</span>r</span>, </span>s</span>, </span>i</span>)                                                                        \
</span>    </span>do </span>{                                                                                           \
</span>      r[</span>0</span>][i] = </span>_mm256_extract_epi32</span>(s[i], </span>0</span>);                                                     \
</span>      r[</span>1</span>][i] = </span>_mm256_extract_epi32</span>(s[i], </span>1</span>);                                                     \
</span>      r[</span>2</span>][i] = </span>_mm256_extract_epi32</span>(s[i], </span>2</span>);                                                     \
</span>      r[</span>3</span>][i] = </span>_mm256_extract_epi32</span>(s[i], </span>3</span>);                                                     \
</span>      r[</span>4</span>][i] = </span>_mm256_extract_epi32</span>(s[i], </span>4</span>);                                                     \
</span>      r[</span>5</span>][i] = </span>_mm256_extract_epi32</span>(s[i], </span>5</span>);                                                     \
</span>      r[</span>6</span>][i] = </span>_mm256_extract_epi32</span>(s[i], </span>6</span>);                                                     \
</span>      r[</span>7</span>][i] = </span>_mm256_extract_epi32</span>(s[i], </span>7</span>);                                                     \
</span>    } </span>while </span>(</span>0</span>);
</span>
</span>  </span>#define </span>_mm256_not_si256</span>(</span>x</span>) </span>_mm256_xor_si256</span>((x), </span>_mm256_set1_epi32</span>(</span>0xffffffff</span>))
</span>  </span>#define </span>RMD_F1</span>(</span>x</span>, </span>y</span>, </span>z</span>) </span>_mm256_xor_si256</span>(x, </span>_mm256_xor_si256</span>(y, z))
</span>  </span>#define </span>RMD_F2</span>(</span>x</span>, </span>y</span>, </span>z</span>) </span>_mm256_or_si256</span>(</span>_mm256_and_si256</span>(x, y), </span>_mm256_andnot_si256</span>(x, z))
</span>  </span>#define </span>RMD_F3</span>(</span>x</span>, </span>y</span>, </span>z</span>) </span>_mm256_xor_si256</span>(</span>_mm256_or_si256</span>(x, </span>_mm256_not_si256</span>(y)), z)
</span>  </span>#define </span>RMD_F4</span>(</span>x</span>, </span>y</span>, </span>z</span>) </span>_mm256_or_si256</span>(</span>_mm256_and_si256</span>(x, z), </span>_mm256_andnot_si256</span>(z, y))
</span>  </span>#define </span>RMD_F5</span>(</span>x</span>, </span>y</span>, </span>z</span>) </span>_mm256_xor_si256</span>(x, </span>_mm256_or_si256</span>(y, </span>_mm256_not_si256</span>(z)))
</span>
</span>  </span>#define </span>RMD_ROTL</span>(</span>x</span>, </span>n</span>) </span>_mm256_or_si256</span>(</span>_mm256_slli_epi32</span>(x, n), </span>_mm256_srli_epi32</span>(x, </span>32 </span>- (n)))
</span>  </span>#define </span>RMD_ADD2</span>(</span>a</span>, </span>b</span>) </span>_mm256_add_epi32</span>(a, b)
</span>  </span>#define </span>RMD_ADD3</span>(</span>a</span>, </span>b</span>, </span>c</span>) </span>_mm256_add_epi32</span>(</span>_mm256_add_epi32</span>(a, b), c)
</span>  </span>#define </span>RMD_ADD4</span>(</span>a</span>, </span>b</span>, </span>c</span>, </span>d</span>) </span>_mm256_add_epi32</span>(</span>_mm256_add_epi32</span>(a, b), </span>_mm256_add_epi32</span>(c, d))
</span>#else
</span>  </span>#error </span>"</span>Unsupported arch for RIPEMD-160 (AVX2 or NEON required)</span>"
</span>#endif
</span></code></pre>
The main differences are:</p>

A different header file.</li>
A different vector type (8 lanes instead of 4 in Neon) and different names for intrinsics.</li>
AVX2 has no Bitwise NOT, so I had to add it separately as _mm256_not_si256</code>.</li>
No separate function for endian-swap, but there is a more generalized function to rearrange bits in a given order: _mm256_shuffle_epi8</code> (the first argument specifies where to rearrange the bits, and the second specifies how to rearrange them).</li>
_mm256_set_epi32</code> makes it more convenient to load data into different lanes, while in Neon we had to use a temporary array (the variant with setting each lane separately (vsetq_lane_u32</code>) is slower).</li>
</ul>
The rewriting of the macros was mostly handled by GPT; I just checked the correctness once again.</p>
Performance of AVX2 Version</a></h2>
I have a small fanless PC running Linux on an Intel N100, which I use for native application testing. I ran the benchmark written earlier on it and got the following results:</p>
❯</span> clang</span> -O3 -march</span>=native ./lib/rmd160.c && </span>./a.out </span># original (on Intel N100)
</span>4.26M</span> it/s </span>~</span> 4.26M h/s </span>~</span> 7.51s
</span>
</span>❯</span> clang</span> -O3 -march</span>=native ./lib/rmd160s.c && </span>./a.out </span># avx2 (on Intel N100)
</span>2.25M</span> it/s </span>~</span> 17.96M h/s </span>~</span> 14.25s
</span></code></pre>
8 lanes of AVX2 and the correct arrangement of rounds in the algorithm (zebra iterations) result in a 320% increase in the number of hashes per second compared to the original code. What's interesting is that AVX2 on the Intel N100 runs 20% faster than Neon on the Apple M2 (mainly because of the vector size). Overall ecloop</code> speedup: 5.45M it/s vs 7.73M it/s (+42%)</code>.</p>
Fallback Implementation for Older Processors / VMs</a></h2>
In the #ifdef</code> above, I left the #else</code> section with #error</code> to prevent compilation on unsupported systems. In general, this is not ideal, and I would like the program to work everywhere (mainly for potential runs in VMs). Since the whole algorithm is already written in macros, adding a new implementation is not difficult. I simply redefine all macros to use a vector size of 1 and uint32_t</code> as the "vector" type. In reality, the program will work with a single array, which, from a memory perspective in C, is essentially the same as just using uint32_t</code>.</p>
// ... #ifdef for neon / avx2
</span>#else
</span>  </span>#warning </span>"</span>Fallback RIPEMD-160 implementation used. AVX2 or NEON required for SIMD</span>"
</span>
</span>  </span>#define </span>RMD_LEN </span>1
</span>  </span>#define </span>RMD_VEC uint32_t
</span>  </span>#define </span>RMD_LD_NUM</span>(</span>x</span>) x
</span>
</span>  </span>#define </span>RMD_SWAP</span>(</span>x</span>) </span>__builtin_bswap32</span>(x)
</span>  </span>#define </span>RMD_LOAD</span>(</span>x</span>, </span>i</span>) x[</span>0</span>][i]
</span>  </span>#define </span>RMD_DUMP</span>(</span>r</span>, </span>s</span>, </span>i</span>) r[</span>0</span>][i] = s[i]
</span>
</span>  </span>#define </span>RMD_F1</span>(</span>x</span>, </span>y</span>, </span>z</span>) ((x) ^ (y) ^ (z))
</span>  </span>#define </span>RMD_F2</span>(</span>x</span>, </span>y</span>, </span>z</span>) (((x) & (y)) | (~(x) & (z)))
</span>  </span>#define </span>RMD_F3</span>(</span>x</span>, </span>y</span>, </span>z</span>) (((x) | ~(y)) ^ (z))
</span>  </span>#define </span>RMD_F4</span>(</span>x</span>, </span>y</span>, </span>z</span>) (((x) & (z)) | ((y) & ~(z)))
</span>  </span>#define </span>RMD_F5</span>(</span>x</span>, </span>y</span>, </span>z</span>) ((x) ^ ((y) | ~(z)))
</span>
</span>  </span>#define </span>RMD_ROTL</span>(</span>x</span>, </span>n</span>) (((x) << (n)) | ((x) >> (</span>32 </span>- (n))))
</span>  </span>#define </span>RMD_ADD2</span>(</span>a</span>, </span>b</span>) (a + b)
</span>  </span>#define </span>RMD_ADD3</span>(</span>a</span>, </span>b</span>, </span>c</span>) (a + b + c)
</span>  </span>#define </span>RMD_ADD4</span>(</span>a</span>, </span>b</span>, </span>c</span>, </span>d</span>) (a + b + c + d)
</span>#endif
</span></code></pre>
I ran it and checked it – it works correctly, except that the speed became a bit faster due to the new order of rounds. It is not difficult to add an implementation for AVX512 (16 lanes) in the same way, but I don't have a processor to test it, so I didn't do it. Also, the article was already getting quite long, so I’m wrapping it up here.</p>
Fin</a></h1>
SIMD programming turned out to be easier than it seemed. The necessary intrinsics can be "Googled" in GPT; it is difficult to search for them yourself because there are thousands of combinations. RMD160 itself is mostly used in cryptocurrencies (at least I don't know of other popular use cases), so the usefulness of the obtained code outside of the learning factor is questionable.</p>
SIMD calculations give a good boost to execution speed, but, of course, you should take into account the specifics of the task: you need to have a lot of data to process, and these data should have the same size. It makes no sense to use SIMD if some data to be hashed fits into one block and others into 100 (for example, when processing files). Also, the main program should be able to process data in batches.</p>
The final code as a single file is available on Github</a>.</p>


From iTerm To WezTerm
2025-01-09T00:00:00+00:00
</p>
For many years, I used iTerm2</a> as my main terminal emulator and probably spent hundreds of hours in it. Overall, I was satisfied with it, despite some strange recent updates like adding AI features, KeyChain integration, and security vulnerabilities</a>.</p>
In recent years, new terminal emulators have appeared. I tried using them mainly for testing macmon</a>. A couple of years ago, I tried switching to kitty</a>, which was faster due to GPU acceleration. However, it required too much customization and still looked very non-native for macOS. GPU acceleration was added to iTerm as well, so I stayed with it.</p>
I also tried Alacritty</a>, but it is very basic and lacks the features I need. Recently, I tested Ghostty</a>, which has gained huge attention – it has nice defaults, but its RAM usage is concerning (around 250MB per empty tab). Currently, it also lacks buffer search</a>, which makes the terminal useless for me.</p>
I heard a lot of positive feedback about WezTerm</a> from bobuk</a>. So I decided to try it out and use it for some time – and I am very satisfied with it.</p>
The next part of the article will be about how to make WezTerm understandable for iTerm users to try it or switch completely.</p>
Initial</a></h2>
WezTerm can be installed from Homebrew:</p>
brew</span> install</span> --cask</span> wezterm
</span></code></pre>
When you first open WezTerm, it looks plain, non-native, and not very functional. But actually, it has everything you need. This is because WezTerm is cross-platform, unlike iTerm2, so the author left minimal default settings, leaving the rest of the configuration to the user. My goal is to achieve a similar experience in WezTerm as I had in iTerm.</p>
</p>
WezTerm does not have a graphical interface for configuration, and the setup is done through a Lua file (this is unusual for me, but Vim users are familiar with it). The configuration file can be located at ~/.wezterm.lua</code> or ~/.config/wezterm/wezterm.lua</code>. I prefer the second path because it is where Fish</a> and starship</a> store configs too.</p>
Create a configuration file:</p>
mkdir -p ~</span>/.config/wezterm/ && </span>touch ~</span>/.config/wezterm/wezterm.lua
</span></code></pre>
And then add the boilerplate code for configuration:</p>
local </span>wezterm </span>= </span>require </span>"</span>wezterm</span>"
</span>local </span>config </span>= </span>wezterm</span>.</span>config_builder</span>()
</span>local </span>action </span>= </span>wezterm</span>.action
</span>
</span>-- (here will be added actual configuration)
</span>
</span>return </span>config
</span></code></pre>
Appearance</a></h2>
First, I want to change the default font, its size, and disable ligatures (because it's important to see the actual characters in case of some logs).</p>
config</span>.font = </span>wezterm</span>.</span>font </span>{
</span>  </span>family </span>= '</span>JetBrains Mono</span>',
</span>  </span>weight </span>= '</span>Medium</span>',
</span>  </span>harfbuzz_features </span>= { '</span>calt=0</span>', '</span>clig=0</span>', '</span>liga=0</span>' }, </span>-- disable ligatures
</span>}
</span>config</span>.font_size = </span>14.0
</span>config</span>.line_height = </span>1.0
</span></code></pre>
Save the file, and the changes should be applied immediately (yes, no reload required!).</p>
Next, I want to change the theme and set it to follow the current system theme automatically. For the past year, I have been using Catppuccin</a> themes, and I am very happy with them. They rarely have issues where some text colors blend with the background. To make the theme switch automatically based on the current system theme, we need to write a small Lua function (I copied it from the documentation):</p>
local function </span>scheme_for_appearance</span>(appearance)
</span>  </span>if </span>appearance</span>:</span>find </span>"</span>Dark</span>" </span>then
</span>    </span>return </span>"</span>Catppuccin Macchiato</span>"
</span>  </span>else
</span>    </span>return </span>"</span>Catppuccin Latte</span>"
</span>  </span>end
</span>end
</span>
</span>config</span>.color_scheme = </span>scheme_for_appearance</span>(</span>wezterm</span>.gui.</span>get_appearance</span>())
</span></code></pre>
I forgot to mention that the Catppuccin theme, like many other popular themes, is pre-installed in WezTerm, so you don't need to install it separately like in iTerm, which is very cool. Of course, you can also set up your own colors</a>.</p>
The default padding also seems a bit large, so let's make it smaller. I also had a Blinking Line Cursor (I don't remember if this is the default or not in iTerm). Let's set both configurations:</p>
config</span>.window_padding = { </span>left </span>= '</span>0.5cell</span>', </span>right </span>= '</span>0.5cell</span>', </span>top </span>= '</span>0.5cell</span>', </span>bottom </span>= '</span>0.5cell</span>' }
</span>config</span>.default_cursor_style = '</span>BlinkingBar</span>'
</span></code></pre>
Window style</a></h2>
In iTerm, I used the Minimal theme to save vertical space for tabs. We can do the same in WezTerm (the default tabs don't look very good). Also possible to add some transparency and blur to the window. I didn't use this in iTerm, but let's try it now:</p>
config</span>.window_decorations = '</span>RESIZE|INTEGRATED_BUTTONS</span>'
</span>config</span>.window_background_opacity = </span>0.96
</span>config</span>.macos_window_background_blur = </span>20
</span></code></pre>
</p>
Maximize on start-up</a></h2>
In iTerm, I usually set large values for Cols/Rows to make the start window bigger. This approach is a bit outdated, so let's make the window open in full screen when the terminal starts.</p>
-- https://github.com/wez/wezterm/issues/3299#issuecomment-2145712082
</span>wezterm</span>.</span>on</span>("</span>gui-startup</span>", </span>function</span>(cmd)
</span>  </span>local </span>active </span>= </span>wezterm</span>.gui.</span>screens</span>().active
</span>	</span>local </span>tab</span>, </span>pane</span>, </span>window </span>= </span>wezterm</span>.mux.</span>spawn_window</span>(</span>cmd </span>or {})
</span>  </span>window</span>:</span>gui_window</span>():</span>set_position</span>(</span>active</span>.x, </span>active</span>.y)
</span>  </span>window</span>:</span>gui_window</span>():</span>set_inner_size</span>(</span>active</span>.width, </span>active</span>.height)
</span>end</span>)
</span></code></pre>
Key bindings</a></h2>
Finally, the most important part is setting up hotkeys – it's hard to give up what you've been used to for years. An important thing for me is Split Panels with Cmd + D</code> / Cmd + Shift + D</code>. Cmd + W</code> should close the current active panel, not the whole tab. Cmd + K</code> – clear the current screen. Cmd + ←</code> / Cmd + →</code> for start/end of the line. I think these are all defaults from iTerm, so let's make them the same in WezTerm:</p>
config</span>.keys = {
</span>  { </span>key </span>= '</span>d</span>', </span>mods </span>= '</span>CMD|SHIFT</span>', </span>action </span>= </span>action</span>.</span>SplitVertical </span>{ </span>domain </span>= '</span>CurrentPaneDomain</span>' } },
</span>  { </span>key </span>= '</span>d</span>', </span>mods </span>= '</span>CMD</span>', </span>action </span>= </span>action</span>.</span>SplitHorizontal </span>{ </span>domain </span>= '</span>CurrentPaneDomain</span>' } },
</span>  { </span>key </span>= '</span>k</span>', </span>mods </span>= '</span>CMD</span>', </span>action </span>= </span>action</span>.</span>ClearScrollback </span>'</span>ScrollbackAndViewport</span>' },
</span>  { </span>key </span>= '</span>w</span>', </span>mods </span>= '</span>CMD</span>', </span>action </span>= </span>action</span>.</span>CloseCurrentPane </span>{ </span>confirm </span>= </span>false </span>} },
</span>  { </span>key </span>= '</span>w</span>', </span>mods </span>= '</span>CMD|SHIFT</span>', </span>action </span>= </span>action</span>.</span>CloseCurrentTab </span>{ </span>confirm </span>= </span>false </span>} },
</span>  { </span>key </span>= '</span>LeftArrow</span>', </span>mods </span>= '</span>CMD</span>', </span>action </span>= </span>action</span>.</span>SendKey </span>{ </span>key </span>= '</span>Home</span>' } },
</span>  { </span>key </span>= '</span>RightArrow</span>', </span>mods </span>= '</span>CMD</span>', </span>action </span>= </span>action</span>.</span>SendKey </span>{ </span>key </span>= '</span>End</span>' } },
</span>  { </span>key </span>= '</span>p</span>', </span>mods </span>= '</span>CMD|SHIFT</span>', </span>action </span>= </span>action</span>.ActivateCommandPalette },
</span>}
</span></code></pre>
WezTerm also has a Command Execution menu. I haven't used it much yet, but just in case, I bind it to the same keys as in VSCode / Zed – Cmd + Shift + P</code>.</p>
Note: In the configuration, the key</code> value is case-sensitive, so { key = 'd', ... }</code> and { key = 'D', ... }</code> are different things (basically key='D'</code> is the same as key='d', mods='SHIFT'</code>). I initially got confused with this.</em></p>
Fin</a></h2>
Overall, I am very satisfied with WezTerm and have fully switched to using it daily. One cool feature I want to mention is the quick connect to servers in Known hosts via SSH (Command Palette → type server name → Enter OR Nav Bar → Shell → Click on server name). WezTerm also has Workspaces</a> – an alternative to tmux, but I haven't used it actively yet.</p>
Here a gist</a> with the final config from this article.</p>


https://wezfurlong.org/wezterm/config/files.html</a></li>
https://alexplescan.com/posts/2024/08/10/wezterm/</a></li>
</ol>


2024 In Review
2025-01-01T00:00:00+00:00
</p>
I have never written an annual report on my work before, but since I also spend time on open source projects, I decided to write one (maybe it will become a tradition).</p>
Programming Languages and Tools</a></h2>
Rust</a></h3>
The main change in my work was the active use of Rust in the second half of the past year. I mainly write in Rust where I used to write in Go. Overall, the language is more complex than Go and sometimes confusing with the borrow checker, but in general, it is quite convenient to write – I can't say that the speed of writing code significantly decreases compared to other languages – in general, the approaches that need to be used are roughly the same for all languages.</p>
What I like about Rust:</p>

crates ecosystem – many libraries available and can be installed in one command</li>
single binary file and low memory consumption</li>
ability to write applications of various directions (web-app, tui, gui, parsers, etc.)</li>
</ul>
What I don't like about Rust:</p>

slow Docker image builds (my solution</a>)</li>
a large number of sub-dependencies for web projects (any project on axum will contain 200+ dependencies)</li>
use of C-binding in "core" libraries like rust-tls (ring) – complicates cross-compilation because you have to maintain a C toolchain as well (fixed through Zig build, but it looks monstrous)</li>
lack of finished solutions for serde (default values</a>, partial initialization</a>)</li>
</ul>
It's nice to write various parsers in Rust, and then they can be coolly parallelized (with rayon</a> or tokio</a>), in Go I had a problems with this due to immutable strings and lack of GC control.</p>
Main languages</a></h3>
The main languages remained the same: Python, NodeJS/TypeScript. I also actively used NextJS on several projects, overall I was satisfied – it's faster and easier to write frontends, but it's quite difficult to find problem's solutions sometimes (many Github issues, often they are open, and the comments contain answers that no longer work).</p>
The main fail of the year in the NodeJS ecosystem is the update of ESLint from 8 -> 9 – completely incompatible API with all existing plugins, the configuration file became larger and more complex – it's not clear why the authors made this diversion. I also tried Binome</a> as an alternative to ESLint / Prettier, but it's completely different and you need to redo a lot, and not everything you need is available at the moment, I continue to monitor this project.</p>
I also tried Zig, but didn't write anything serious. Personally, I didn't understand why this language is needed. At the time I tried it, there was no even package manager yet. Obviously Zig is a competitor to C (not Rust), but C is now much richer in its toolkit and support in other languages. The only thing I can point out is zig build</code> which can build C projects too and does it with less problems.</p>
About LLM</a></h3>
I want to separately note about use of GPTs in work. I have an opportunity to do business tasks in the format of parsing and analyzing texts. What a couple of years ago was done with a bunch of regex, etc., can now be done through GPT and it works great! I tried using local Llama2, but it would have taken weeks to process my data on M2 chip. I redo the solution to use ChatGPT API and processed the data in a couple of days. It's good that all LLMs support openai-client API in Python, so the code almost didn't need to be rewritten. Among the disadvantages of LLama-like aka local running, I also want to note the lack of support for Function calling</a> (to guarantee to bring the LLM response to the desired structure, in my case). The last time I did something serious was more than six months ago, maybe a lot has changed. I also tested gemma2, the inference speed was impressive, and the quality of the answers was not much worse.</p>
As local runners, I used llama.cpp</a> and https://github.com/ml-explore/mlx</a> – the second one uses Mac resources better (checked through macmon</a>), but new models come out a bit slower on it. Some people use ollama</a>, but I didn't understand why, considering that it has an incompatible client with openai-client API and slower performance.</p>
Blog</a></h2>
Initially, I ran my blog only on Medium, but it's not comfortable to write text there – it's much easier to write and edit in the same editor where I write code (plus my editor has Copilot). Therefore, I decided to move my blog to a static page generator. As an engine, I chose Zola</a> (Rust-based, simple, and fast); I also tried Astro</a>, but it's too big and complex for such a simple task. Overall, Zola is good, but it lacks some filters in templating</a>, not enough flexibility in file structure, and not all popular languages are supported in code highlighting.</p>
I left Medium as a fallback for old readers. I also made a cross-post on dev.to</a>, but there are very few views. Medium is disappointed with the lack of the ability to automate cross-posting (they have a deprecated API, but it does not support updating articles and code highlights). Most likely, it's possible to reverse engineer how they save articles through their editor (changes are sent as diffs). Does anyone else need this besides me?</p>
New projects</a></h2>
Last year I launched several new projects (in chronological order):</p>


timewiz.cc</a> (react</code>, webapp</code>) – a tool to find common working hours for distributed teams (an alternative to worldtimebuddy.com, I don't remember exactly why I made my own version). Score: fail.</p>
</li>

ecloop</a> (c</code>, cli</code>) – secp256k1 elliptic curve implementation for Apple chips. Mainly an educational project to understand cryptography, SIMD optimizations, etc. Added several standard use cases for finding Bitcoin keys by brute force, but mathematically this is pointless, so as I said earlier, this is more of an interesting educational project (and I wanted to publish a series of articles on cryptography, but didn't have time last year, I hope to do it this year). Some people use this project in hopes of finding something. I also had ideas to add Metal API support – but that takes a lot of time, so I switched to other projects. Score: okay.</p>
</li>

macmon</a> (rust</code>, tui</code>) – TUI program for monitoring MacOS performance (CPU/GPU/ANE usage, RAM, temps). My first public Rust project, a lot of reverse engineering of how powermetrics in MacOS</a> works, C-bindings. The project is used both by users in TUI mode and by other projects in headless mode (JSON metrics). Score: success.</p>
</li>

ghstats</a> (rust</code>, axum</code>, htmx</code>, webapp</code>) – a small self-hosted service that collects and saves GitHub repo statistics, has several built-in views and an API to retrieve all statistics. The project started well, but then kind of fizzled out in terms of issues and feature requests, most likely users are satisfied with the current functionality. I would like to add more visualizations, but I haven't had time for that yet. The main thing for now is that the statistics are collected and saved, so something can be done with this in the future. Technically, I like that the service consumes only 10 MB of RAM. Score: okay.</p>
</li>

ogp</a> (rust</code>, axum</code>, api-service</code>) – a small project for generating OGP articles preview</a> as a service. Used in this blog (you can check by sending this article to your friends in a messenger or social network 😀). Worked with the SVG standard (I thought it was more adequate) and generating PNG from SVG. Score: more of fail.</p>
</li>

compose-updater</a> (python</code>, api-service</code>) – a project for updating the version of a docker image from CI in a closed infrastructure. I didn't find anything ready to solve this task. There is watchtower</a>, but it's more about keeping up-to-date docker images, rather than updating a specific one, and also I couldn't run it with ECR repos – the author made a super minimalist image in which I couldn't enter through sh</code> to debug. The second alternative is umputun/updater</a>, but it's more about running different shell commands through a webhook. I chose Python because I didn't find a Rust crate for Docker API (most likely it exists, I just didn't understand which one is really good). Score: – (I didn't promote the project anywhere, I use it for my own purposes for now).</p>
</li>
</ul>
Existing projects</a></h2>


twscrape</a> (python</code>, lib</code>) – a library for parsing</a> data from X/Twitter. Mainly the project is in maintenance mode, I check its functionality every few months, and there have been no requests for new features.</p>
</li>

apigen-ts</a> (typescript</code>, lib</code>) – a library for generating</a> TypeScript client OpenAPI spec. I continue to use it in my projects; I also see that there are a small number of other users. My relevance has slightly decreased due to the NextJS/TRPC stack, but in other configurations (e.g. FastApi + React), it is still relevant, so use it. Periodically there are requests to fix something or add a feature.</p>
</li>
</ul>
Abandoned projects</a></h2>
I also have several projects that I didn't finish and abandoned. I'll write about them here, maybe someone needs something from this, then I can return to on it.</p>


anypost</code> – cross-posting between static blog, Medium, Dev.to, Substack, and collecting joined article statistics. I abandoned the project due to the lack of an API at Medium – the only solution I found was through protocol reverse and session cookie.</p>
</li>

bcloop</code> – a program to walk Bitcoin-like blockchain and compile the final UTXO list. I abandoned it because I couldn't figure out how to effectively create a HashMap-like structure with external memory support to run on computers with small RAM (less than 32GB, LOL).</p>
</li>

tg-watcher</code> – a program that listens to all incoming messages in Telegram (chats & channels) and saves their history in a separate database with the ability to search through these messages later. I have the first version that just saves messages in json files, but I didn't develop the idea further.</p>
</li>
</ul>
Devices</a></h2>
I'm not a big tech geek, but I want to say about a few devices which made my life easier:</p>

MeLE Fanless Quieter 4C</a> – a small fanless PC for homelab</em>, I use it as a native Linux via SSH for testing various programs and keep some cron jobs on it for data collection. It's cool that it's silent because it has no fan, but it heats up a lot under load. The performance of the N100 chip surprised me – only 2x slower than M2 chip (tested with ecloop bench</code>, single core).</li>
Promate HDMI-PD100</a> – nice cable format to connect a monitor and my Mac. One cable is needed to charge and transmit the video stream. Before that, I used hubs, but it's a big block. I'm glad I see fewer wires.</li>
</ul>
At Glance</a></h2>
Over the year, 8 articles were published, totaling 9039 words (~45 minutes to read), and gained 5203 views. Top 3 articles:</p>

Table sorting and pagination with HTMX</a> – 1714 views</li>
macmon – Mac Usage Monitor in Terminal</a> – 953 views</li>
Setting up WireGuard VPN at AWS with Terraform</a> – 642 views</li>
</ul>
Data aggregated from this site (with umami</a>), Medium, and Dev.to. Script</a>.</em></p>
Public GitHub projects – totaling 312 commits, 49 issues closed, 43 releases created, and 1394 stars gained. Top-3 repos:</p>

twscrape</a> – +802 stars / +52515 views</li>
macmon</a> – +420 stars / +6899 views</li>
ghstats</a> – +128 stars / +2757 views</li>
</ul>
Data aggregated with ghstats</code>.</em></p>
I hope I will see a different top with new projects this year 😀</p>
Donations</a></h2>
In twscrape</code> repository I was asked to make it possible to send coffee as donations. Over the year I received 38 cups of coffee from 10 people. I spend these coffees on an annual Parallels</a> subscription, which allows me to test some apps on Windows. Also, I partially paid from this fund for MeLE and VPS, which I used to test apps as well. Many thanks for supporting my work!</p>


Creating OG Images for Static Pages with Rust
2024-11-30T00:00:00+00:00
OGP (Open Graph Protocol</a>) is an HTML meta-tags specification for websites to make links look nice in third-party services such as Facebook, X, LinkedIn or chat apps. OGP also allows to specify a link to page thumbnail that will be shown as preview when link shared, which can increase clickability.</p>
Some blog engines have built-in functions for this. I use Zola</a> (static page generator) to create pages for this blog. Unfortunately, Zola does not have an ability to generate OG images. This can be achieved with external tools or scripts during the build process. Another approach to use edge functions (such as vercel/og</a>) to generate thumbnails on the fly. However, this approach relies on a specific service provider, which I want to avoid.</p>
I thought, why not use OG image generation as a service? I searched around and found og-image-generator</a> by sagarhani.</p>
The functionality of og-image-generator</code> worked well for me, and I used it for this blog for some time. However, the project seems abandoned, and I'm unsure how the author feels about additional load on their service. I decided it shouldn't be too hard to create a similar service myself — a single endpoint for generating images with search params. Moreover, when I looked into the source code of og-image-generator</code>, I noticed that it launches Chrome to render an image from HTML, which felt like significant overhead to me.</p>
Coding</a></h2>
I need an endpoint that accepts: article title, URL/domain, author name and avatar via search parameters. Lately, I've been using Rust for most of my pet projects, so I decided to build this service with Rust as well. I added a route in axum</code> and implemented a handler that generates SVG images, using some basic math to calculate the positioning of elements. The size of the OG image is predefined as 1200x630</code>, following the specefication. SVG allows the use of a <style></code> tag, making it easy to customise default font-family.</p>
The only challenge I encountered was creating a mask for the image. I initially thought SVG might have some syntactic sugar for this, but it essentially works the same way as masks in Photoshop.</p>
Example of SVG generation (maud</code></a> is an HTML template engine on Rust macros):</p>
use </span>maud::{html, Markup};
</span>
</span>fn </span>render_svg</span>(</span>title</span>: &</span>str</span>, </span>author</span>: &</span>str</span>, </span>url</span>: &</span>str</span>, </span>photo</span>: &</span>str</span>) -> Markup {
</span>  </span>let </span>(w, h) = (</span>1200</span>, </span>630</span>);
</span>
</span>  </span>let</span> pic_r = </span>50</span>;
</span>  </span>let</span> pic_y = h - </span>128</span>;
</span>  </span>let</span> l1y = pic_y - (pic_r - </span>10</span>) / </span>2 </span>+ </span>48 </span>/ </span>2 </span>- </span>6</span>;
</span>  </span>let</span> l2y = pic_y - (pic_r - </span>10</span>) / </span>2 </span>+ </span>32 </span>/ </span>2 </span>+ </span>6 </span>+ </span>32</span>;
</span>
</span>  html!(svg xmlns="</span>http://www.w3.org/2000/svg</span>" viewBox=(format!("</span>0 0 </span>{} {}</span>", w, h)) width=(w) height=(h) {
</span>    style { ("</span>text { font-family: 'Open Sans', Arial, sans-serif; }</span>" ) }
</span>
</span>    rect x="</span>0</span>" y="</span>0</span>" width=(w) height=(h) fill="</span>black</span>" stroke="</span>white</span>" stroke-width="</span>16</span>" {}
</span>    text x=(</span>120</span>) y=(</span>200</span>) font-weight="</span>700</span>" font-size="</span>72</span>" fill="</span>white</span>" { (title) }
</span>    text x=(</span>200</span>) y=(l1y) font-weight="</span>700</span>" font-size="</span>48</span>" fill="</span>white</span>" { (author) }
</span>    text x=(</span>200</span>) y=(l2y) font-weight="</span>400</span>" font-size="</span>32</span>" fill="</span>white</span>" { (url) }
</span>    (</span>circle_avatar</span>(photo, </span>128</span>, pic_y, pic_r))
</span>  })
</span>}
</span>
</span>fn </span>circle_avatar</span>(</span>url</span>: &</span>str</span>, </span>cx</span>: </span>u32</span>, </span>cy</span>: </span>u32</span>, </span>radius</span>: </span>u32</span>) -> Markup {
</span>  html!({
</span>    defs {
</span>      clipPath id="</span>circle-photo</span>" { circle cx=(cx) cy=(cy) r=(radius) {} }
</span>    }
</span>
</span>    g clip-path="</span>url(#circle-photo)</span>" {
</span>      image href=(url) x=(cx-radius) y=(cy-radius) width=(radius * </span>2</span>) height=(radius * </span>2</span>) {}
</span>      circle cx=(cx) cy=(cy) r=(radius) stroke="</span>white</span>" stroke-width="</span>4</span>" fill="</span>none</span>" {}
</span>    }
</span>  })
</span>}
</span></code></pre>
And then the SVG image is generated in the Axum handler:</p>
use </span>axum::extract::{Query, Request};
</span>use </span>axum::http::{header, StatusCode};
</span>use </span>axum::response::IntoResponse;
</span>use </span>std::collections::HashMap;
</span>
</span>async </span>fn </span>svg_handler</span>(</span>req</span>: Request) -> Res<impl IntoResponse> {
</span>  </span>let</span> qs: Query<HashMap<String, String>> = Query::try_from_uri(req.</span>uri</span>())?;
</span>  </span>let</span> title = qs.</span>get</span>("</span>title</span>").</span>cloned</span>().</span>unwrap_or</span>("</span>This is default article title</span>".</span>to_string</span>());
</span>  </span>let</span> author = qs.</span>get</span>("</span>author</span>").</span>cloned</span>().</span>unwrap_or</span>("</span>Author</span>".</span>to_string</span>());
</span>  </span>let</span> url = qs.</span>get</span>("</span>url</span>").</span>cloned</span>().</span>unwrap_or</span>("</span>https://example.com</span>".</span>to_string</span>());
</span>  </span>let</span> photo = qs.</span>get</span>("</span>photo</span>").</span>cloned</span>().</span>unwrap_or</span>("</span>https://gravatar.com/avatar/</span>".</span>to_string</span>());
</span>
</span>  </span>let</span> svg = </span>render_svg</span>(&title, &author, &url, &photo);
</span>  Ok((StatusCode::</span>OK</span>, [(header::</span>CONTENT_TYPE</span>, "</span>image/svg+xml</span>")], svg))
</span>}
</span></code></pre>
This code will generate the following SVG:</p>
The endpoint can accept title</code>, author</code>, url</code> and photo</code> in the search params, and these values will be passed to SVG template. That's pretty much it, I thought, deployed the service to fly.io</a>, updated the blog links, went to check how new preview looked, and the thumbnail didn't display.</p>
It turned out that OG images cannot be SVG (which makes sense, because then different mobile clients would need to be able to render it, which is harder then showing prepared image).</p>
SVG to PNG</a></h2>
It's not a big deal, I thought. A lot things has already been rewritten in Rust, and there's probably a ready to use solution for SVG rendering, especially since I found several popular crates for this. But it turned out to be more complicated than expected. Rendering SVG to PNG is quite a complex task, and the SVG specification is huge, so libs not fully implement the specification. However, I only need basic SVG support. I tried several crates and settled on resvg</code>. I wrote a rendering function:</p>
use </span>resvg::{tiny_skia, usvg};
</span>
</span>fn </span>render_png</span>(</span>svg</span>: &</span>str</span>) -> Vec<</span>u8</span>> {
</span>  </span>let</span> tree = {
</span>    </span>let mut</span> opt = usvg::Options::default();
</span>    opt.</span>fontdb_mut</span>().</span>load_system_fonts</span>();
</span>    usvg::Tree::from_str(svg, &opt).</span>unwrap</span>()
</span>  };
</span>
</span>  </span>let</span> size = tree.</span>size</span>().</span>to_int_size</span>();
</span>  </span>let mut</span> pixmap = tiny_skia::Pixmap::new(size.</span>width</span>(), size.</span>height</span>()).</span>unwrap</span>();
</span>  resvg::render(&tree, tiny_skia::Transform::default(), &</span>mut</span> pixmap.</span>as_mut</span>());
</span>  pixmap.</span>encode_png</span>().</span>unwrap</span>()
</span>}
</span></code></pre>
And updated handlers:</p>
// in some reason axum not allow to pass Request as ref – code will not compile, so req.uri() used
</span>async </span>fn </span>prepare_svg</span>(</span>uri</span>: &axum::http::Uri) -> Res<String> {
</span>  </span>let</span> qs: Query<HashMap<String, String>> = Query::try_from_uri(uri)?;
</span>  </span>let</span> title = qs.</span>get</span>("</span>title</span>").</span>cloned</span>().</span>unwrap_or</span>("</span>This is default article title</span>".</span>to_string</span>());
</span>  </span>let</span> author = qs.</span>get</span>("</span>author</span>").</span>cloned</span>().</span>unwrap_or</span>("</span>Author</span>".</span>to_string</span>());
</span>  </span>let</span> url = qs.</span>get</span>("</span>url</span>").</span>cloned</span>().</span>unwrap_or</span>("</span>https://example.com</span>".</span>to_string</span>());
</span>  </span>let</span> photo = qs.</span>get</span>("</span>photo</span>").</span>cloned</span>().</span>unwrap_or</span>("</span>https://gravatar.com/avatar/</span>".</span>to_string</span>());
</span>  Ok(</span>render_svg</span>(&title, &author, &url, &photo).</span>into_string</span>())
</span>}
</span>
</span>async </span>fn </span>svg_handler</span>(</span>req</span>: Request) -> Res<impl IntoResponse> {
</span>  </span>let</span> svg = </span>prepare_svg</span>(req.</span>uri</span>()).await?;
</span>  Ok((StatusCode::</span>OK</span>, [(header::</span>CONTENT_TYPE</span>, "</span>image/svg+xml</span>")], svg))
</span>}
</span>
</span>async </span>fn </span>png_handler</span>(</span>req</span>: Request) -> Res<impl IntoResponse> {
</span>  </span>let</span> svg = </span>prepare_svg</span>(req.</span>uri</span>()).await?;
</span>  </span>let</span> png = </span>render_png</span>(&svg);
</span>  Ok((StatusCode::</span>OK</span>, [(header::</span>CONTENT_TYPE</span>, "</span>image/png</span>")], png))
</span>}
</span></code></pre>
I run it, and generally, everything works, but the avatar doesn't show up.</p>

Loading remote images</a></h2>
resvg</code> itself cannot load images from remote URLs used in SVG, but it can render images from a base64-encoded data string. Okay, I'll load the image and convert it myself:</p>
use </span>base64::Engine;
</span>use </span>reqwest::header::{HeaderMap, HeaderValue};
</span>
</span>async </span>fn </span>load_base64_image</span>(</span>url</span>: &</span>str</span>) -> Res<String> {
</span>  </span>let</span> ua = format!("</span>{}</span>/</span>{}</span>", env!("</span>CARGO_PKG_NAME</span>"), env!("</span>CARGO_PKG_VERSION</span>"));
</span>
</span>  </span>let mut</span> headers = HeaderMap::new();
</span>  headers.</span>insert</span>("</span>User-Agent</span>", HeaderValue::from_str(&ua)?);
</span>
</span>  </span>let</span> client = reqwest::Client::builder()
</span>    .</span>default_headers</span>(headers)
</span>    .</span>read_timeout</span>(std::time::Duration::from_secs(</span>10</span>))
</span>    .</span>build</span>()?;
</span>
</span>  </span>let</span> rep = client.</span>get</span>(url).</span>send</span>().await?;
</span>
</span>  </span>let</span> max_size = </span>1024 </span>* </span>1024 </span>* </span>5</span>; </span>// 5MB
</span>  </span>match</span> rep.</span>content_length</span>() {
</span>    None => </span>return </span>AppError::new("</span>Image size is unknown</span>"),
</span>    Some(len) </span>if</span> len > max_size => </span>return </span>AppError::new("</span>Image is too large</span>"),
</span>    _ => {}
</span>  };
</span>
</span>  </span>let</span> allowed = ["</span>image/jpeg</span>", "</span>image/png</span>", "</span>image/webp</span>", "</span>image/svg+xml</span>"];
</span>  </span>let</span> mime = </span>match</span> rep.</span>headers</span>().</span>get</span>(header::</span>CONTENT_TYPE</span>) {
</span>    None => </span>return </span>AppError::new("</span>Content type is unknown</span>"),
</span>    Some(ct) </span>if </span>!allowed.</span>contains</span>(&ct.</span>to_str</span>().</span>unwrap</span>()) => {
</span>      </span>return </span>AppError::new(&format!("</span>Content type is not allowed: </span>{:?}</span>", ct))
</span>    }
</span>    Some(ct) => ct.</span>to_str</span>().</span>unwrap</span>().</span>to_string</span>(),
</span>  };
</span>
</span>  </span>let</span> rep = rep.</span>bytes</span>().await?;
</span>  </span>let</span> rep = base64::engine::general_purpose::</span>STANDARD</span>.</span>encode</span>(rep);
</span>  Ok(format!("</span>data:</span>{mime}</span>;base64,</span>{rep}</span>"))
</span>}
</span>
</span>async </span>fn </span>prepare_svg</span>(</span>uri</span>: &axum::http::Uri) -> Res<String> {
</span>  </span>// ...
</span>  </span>let</span> photo = qs.</span>get</span>("</span>photo</span>").</span>cloned</span>().</span>unwrap_or</span>("</span>https://gravatar.com/avatar/</span>".</span>to_string</span>());
</span>  </span>let</span> photo = </span>load_base64_image</span>(&photo).await?;
</span>  Ok(</span>render_svg</span>(&title, &author, &url, &photo).</span>into_string</span>())
</span>}
</span></code></pre>
I check generation again – everything is fine, the avatar is present in the final render:</p>

Docker image</a></h2>
So, the service is ready. Now, just needed to pack code into a Docker image</a>. It's pretty much the same as usual, except that fonts files should be added to image too. I usually use Alpine images, and fonts can be added via apk</code>:</p>
FROM</span> alpine:latest
</span>RUN </span>apk add --no-cache ttf-opensans
</span>WORKDIR </span>/app
</span># ... copy and run bin file
</span></code></pre>
Fin</a></h2>
I added a template and copied themes from sagarhani/og-image-generator</code>. I also created a preview page where you can experiment with the generation. This is enough for my current needs, but if anyone needs more, Issues/PRs are welcome.</p>
The service is available at: https://ogp.fly.dev</a>

The code is on GitHub: https://github.com/vladkens/ogp</a></p>
The service is currently running without any restrictions. There is also the option to run self-hosted version with ghcr.io/vladkens/ogp</code> image.</p>
To integrate this service into a static website, you just need to add corresponding <meta></code> tags with service link and replace token with your data.</p>
<</span>meta
</span>  </span>property</span>="</span>og:image</span>"
</span>  </span>content</span>="</span>https://ogp.fly.dev/v0/png?title={title}&author={author}&photo={photo}&url={url}&theme={theme}</span>"
</span>/>
</span></code></pre>
Usage example for this blog: link generation</a>, meta tags</a>. To see link sharing at work click on links below ⬇️💀</p>


Deploy FastAPI application with SQLite on Fly.io
2024-10-21T00:00:00+00:00
</p>
Cloud solutions are good for medium and large projects, but too heavy for small personal projects. If you want to launch something small (a few api endpoins and a small repository), there are three options:</p>

Use the same approaches as for "big" projects (AWS ECS/EKS, RDS), but they are redundant, and infrastructure code can be larger than code of the actual project. Also it's expensive (~$100).</li>
Use serverless solutions (Lambda, Vercel). Most cloud providers have such solutions, but these services have difficulties with simple databases – they provide cheap vendor solutions (AWS) or require a managed database, which again is expensive (mostly nothing for serverless, ~$20 for DB)</li>
Use VPS with Docker. It is cheap (~$5 for small machine) and almost no need to manage infrastructure, but deployments sucks (needs private or self-hosted registry, SSH access from CI).</li>
</ul>
I usually write my small applications using SQLite, it's a handy little single file database that works in any programming language and can be copied to local machine to analyze data for example. So I was looking for some middleware solution that combines the serveless approach, ease of deployment and ability to use SQLite and found Fly.io.</p>
Setup</a></h2>
If you don't have an account in Fly.io – you need to create</a> it. Also CLI tool</a> called flyctl</code> required to manage projects. Fly.io can be deployed both locally and from CI.</p>
flyctl</code> makes deploy from project's root folder from Dockerfile, which is cool, because same Dockerfile can be used in other systems. For play with Fly.io, I prepared a simple FastAPI project that stores state in database – generic url shortener with click counting.</p>
Dockerfile</code>:</p>
FROM</span> python:3.13-alpine
</span>WORKDIR </span>/app
</span>
</span>COPY</span> ./requirements.txt .
</span>RUN </span>pip install --no-cache-dir --upgrade -r requirements.txt
</span>
</span>COPY</span> . /app
</span>
</span>ENV </span>HOST=0.0.0.0 PORT=8080
</span>EXPOSE </span>${PORT}
</span>CMD uvicorn main:app --host ${HOST} --port ${PORT}
</span></code></pre>
main.py</code>:</p>
import </span>asyncio
</span>import </span>random
</span>import </span>string
</span>from </span>urllib.parse </span>import </span>urlparse
</span>
</span>import </span>aiosqlite
</span>from </span>fastapi </span>import </span>FastAPI, HTTPException, Request
</span>from </span>fastapi.responses </span>import </span>RedirectResponse
</span>
</span>DB_PATH </span>= "</span>/data/app.db</span>"
</span>
</span>app = </span>FastAPI</span>()
</span>
</span>async def </span>get_db</span>() -> aiosqlite.Connection:
</span>    </span>if </span>db := </span>getattr</span>(get_db, "</span>_db</span>", </span>None</span>):
</span>        </span>if </span>db.is_alive:
</span>            </span>return </span>db
</span>
</span>    db = </span>await </span>aiosqlite.</span>connect</span>(</span>DB_PATH</span>, </span>loop</span>=asyncio.</span>get_event_loop</span>())
</span>    db.row_factory = aiosqlite.Row
</span>
</span>    qs = """
</span>    </span>CREATE TABLE </span>IF NOT EXISTS </span>links</span> (
</span>        created_at </span>INTEGER DEFAULT</span> (strftime(</span>'</span>%s</span>'</span>, </span>'</span>now</span>'</span>)),
</span>        short_code </span>TEXT PRIMARY KEY</span>,
</span>        full_url </span>TEXT </span>NOT </span>NULL</span>,
</span>        clicks </span>INTEGER DEFAULT </span>0
</span>    )
</span>    </span>"""
</span>
</span>    </span>await </span>db.</span>execute</span>(qs)
</span>    </span>await </span>db.</span>commit</span>()
</span>
</span>    </span>setattr</span>(get_db, "</span>_db</span>", db)
</span>    </span>return </span>db
</span>
</span>def </span>random_code</span>(</span>length</span>=</span>8</span>) -> str:
</span>    alphabet = string.ascii_letters + string.digits
</span>    </span>return </span>"".</span>join</span>(random.</span>choice</span>(alphabet) </span>for </span>x </span>in </span>range</span>(length))
</span>
</span>def </span>is_valid_url</span>(</span>url</span>: str) -> bool:
</span>    </span>try</span>:
</span>        parts = </span>urlparse</span>(url)
</span>        </span>return </span>all</span>([parts.scheme, parts.netloc])
</span>    </span>except </span>ValueError:
</span>        </span>return </span>False
</span>
</span>@app.</span>post</span>("</span>/</span>")
</span>async def </span>shorten</span>(</span>url</span>: str, </span>req</span>: Request):
</span>    </span>if </span>not </span>is_valid_url</span>(url):
</span>        </span>raise </span>HTTPException</span>(</span>status_code</span>=</span>400</span>, </span>detail</span>="</span>Invalid URL</span>")
</span>
</span>    host = req.headers.</span>get</span>("</span>host</span>")
</span>    </span>if </span>host is </span>None</span>:
</span>        </span>raise </span>HTTPException</span>(</span>status_code</span>=</span>500</span>, </span>detail</span>="</span>Missing host header</span>")
</span>
</span>    short_code = </span>random_code</span>()
</span>    db = </span>await </span>get_db</span>()
</span>    qs = "</span>INSERT INTO</span> links (short_code, full_url) </span>VALUES</span> (?, ?)</span>"
</span>    </span>await </span>db.</span>execute</span>(qs, (short_code, url))
</span>    </span>await </span>db.</span>commit</span>()
</span>
</span>    </span>return f</span>"</span>https://</span>{host}</span>/</span>{short_code}"
</span>
</span>@app.</span>get</span>("</span>/</span>")
</span>async def </span>list_links</span>():
</span>    db = </span>await </span>get_db</span>()
</span>    qs = "</span>SELECT</span> short_code, full_url, clicks </span>FROM</span> links </span>ORDER BY</span> created_at </span>DESC</span>"
</span>    </span>async with </span>db.</span>execute</span>(qs) </span>as </span>cursor:
</span>        </span>return await </span>cursor.</span>fetchall</span>()
</span>
</span>@app.</span>get</span>("</span>/</span>{short_code}</span>")
</span>async def </span>redirect</span>(</span>short_code</span>: str):
</span>    db = </span>await </span>get_db</span>()
</span>    qs = """
</span>    </span>UPDATE</span> links </span>SET</span> clicks </span>=</span> clicks </span>+ </span>1 </span>WHERE</span> short_code </span>=</span> ?
</span>    RETURNING full_url
</span>    </span>"""
</span>
</span>    </span>async with </span>db.</span>execute</span>(qs, (short_code,)) </span>as </span>cursor:
</span>        </span>if </span>row := </span>await </span>cursor.</span>fetchone</span>():
</span>            </span>return </span>RedirectResponse</span>(row["</span>full_url</span>"])
</span>
</span>    </span>raise </span>HTTPException</span>(</span>status_code</span>=</span>404</span>)
</span></code></pre>
requirements.txt</code>:</p>
aiosqlite
</span>fastapi
</span>uvicorn
</span></code></pre>
Deploy</a></h2>
To deploy our code, first we need to create a Fly.io project. This can be done either in the web interface or with flyctl</code>. To create proejct with CLU tool in root folder (where code located) flyctl launch should be runned. This command will offer to select desired hardware and will create fly.toml</code> file:</p>
fly</span> launch</span> --build-only
</span></code></pre>
You can modify project in future by changing parameters in this file or via web ui. The basic fly.toml</code> looks fine, but SQLite requires Storage, which can be created with:</p>
fly</span> volumes create sqlite_data</span> -s</span> 1</span> -r</span> ams
</span></code></pre>
where -s 1</code> sets volume size to 1 GB (default is 3 GB), and -r</code> is region in which volume will be created (use same region in which Fly.io project is created). You can always change storage size later.</p>
The last thing to do is to add a mounts</code> section to fly.toml</code>, which attaches the volume to the application:</p>
[mounts]
</span>source </span>= "</span>sqlite_data</span>"
</span>destination </span>= "</span>/data</span>"
</span></code></pre>
sqlite_data</code> is the name of the storage, /data</code> is the path where volume will be connected. This is essentially same as docker run --mount source=sqlite_data,target=/data</code> or corresponding Docker Compose section.</p>
SQLite cannot be writable from more than one app, and Fly.io by default creates 2 instances for an app, so we can specify the number of replicas as one just in case:</p>
fly</span> scale count 1
</span></code></pre>
All configurations are done now and we can deploy our app with command:</p>
fly</span> deploy
</span></code></pre>
The app should boot successfully and the public DNS name will be printed to console. Now we can check it out by posting some url to shortener:</p>
❯</span> curl</span> -X</span> POST '</span>https://fly-fastapi-sqlite.fly.dev/?url=https://example.com</span>'
</span>https://fly-fastapi-sqlite.fly.dev/8ydmfAcK
</span></code></pre>
Then we can visit this link, it should redirect to https://example.com</code>. Finally, we can check that clicks are updated:</p>
❯</span> curl</span> -s </span>'</span>https://fly-fastapi-sqlite.fly.dev/</span>' | </span>jq
</span>[
</span>  {
</span>    "</span>short_code</span>"</span>: </span>"</span>8ydmfAcK</span>",
</span>    "</span>full_url</span>"</span>: </span>"</span>https://example.com</span>",
</span>    "</span>clicks</span>"</span>:</span> 1
</span>  }
</span>]
</span></code></pre>
To check that database state saved between deployments, we can perform new deployment with fly deploy</code> and check that links list remained same as above (1 link, 1 click).</p>
Migrations</a></h2>
If you are using an external solution for migrations, rather than running them from code when app starts, then only way to run migration is to put it in Dockerfile as part of the RUN command.</p>
Backup</a></h2>
We can connect to machine with fly ssh console</code> and then in /data</code> folder interact with database file. Also we can copy database file to local machine with:</p>
fly</span> ssh sftp get /data/app.db ./app-backup.db
</span></code></pre>
Conclusion</a></h2>
Fly.io is a simple and convenient service for deploying applications. Deploy works from Docker Containers, additional services include PSQL, Redis, S3 like storage (unlike Vercel). It's cheap, the cheapest service costs 3 dollars (1 shared CPU / 256 MB) – maybe even less, if you have little traffic – container shuts down after a few minutes without activity and automatically turns on when traffic appears.</p>
On downside, there is no built-in solution for scheduled tasks – instead, the official solution is to set up a separate server with crontab</code> and run tasks from it – it's kind of creepy.</p>


https://fly.io/docs/rails/advanced-guides/sqlite3/</a></li>
https://matthewsetter.com/deploy-go-sqlite-app-flydotapp/</a></li>
https://programmingmylife.com/2023-11-06-using-sqlite-for-a-django-application-on-flyio.html</a></li>
</ol>


Setting up WireGuard VPN at AWS with Terraform
2024-10-08T00:00:00+00:00
All resources in AWS work inside private VPC. Sometimes you may need to access these resources from local computer (e.g. to interact with database). Some resources, like RDS, have the option to enable public access to them – but this is unsecure. Of course you can configure Security Group to allow access to public resource only from allowed IPs to make this setup a bit better, but still in this case all your colleagues must to have static IPs, which is not always true.</p>
Objective</a></h2>
We need to provide a quick and easy solution to give access to internal AWS VPC resources for our team. In general, there are two ways to do this:</p>

Bastion / Jump Host – EC2 instance with SSH access for each member</li>
Private VPN – EC2 instance with VPN config for each member</li>
</ol>
In both cases we need to to setup a custom EC2 instance, so second variant seems more convenient to me.</p>
Preparation</a></h2>
In addition to the traditional Terraform client, we will need WireGuard CLI tools to generate a key pair. You can install WireGuard tools on macOS via brew</code>:</p>
brew</span> install wireguard-tools
</span></code></pre>
Initial Terraform</a></h2>
First, as usual, we need to configure Terraform Provider and get VPC and Public Subnet data where we want to host our VPN server. On this step you can export VPC / Public Subnet from another layer or pass them via variables. I will access my VPC and Public Subnet by name.</p>
terraform</span> {
</span>  required_providers {
</span>    </span>aws </span>= { </span>source </span>= "</span>hashicorp/aws</span>", </span>version </span>= "</span>~> 5.70</span>" }
</span>  }
</span>}
</span>
</span>provider </span>"</span>aws</span>" {
</span>  </span>profile </span>= "</span>default</span>"   </span># Change this to your AWS profile
</span>  </span>region  </span>= "</span>us-east-1</span>" </span># Change this to your AWS region
</span>}
</span>
</span># MARK: Getting VPC and Subnets
</span>
</span>locals</span> {
</span>  </span>vpc_name </span>= "</span>my_vpc</span>" </span># Change this to yor VPC name
</span>}
</span>
</span>data </span>"</span>aws_vpc</span>" "</span>vpc</span>" {
</span>  filter {
</span>    </span>name   </span>= "</span>tag:Name</span>"
</span>    </span>values </span>= [</span>local</span>.vpc_name]
</span>  }
</span>}
</span>
</span>data </span>"</span>aws_subnets</span>" "</span>public</span>" {
</span>  filter {
</span>    </span>name   </span>= "</span>vpc-id</span>"
</span>    </span>values </span>= [</span>data</span>.aws_vpc.vpc.id]
</span>  }
</span>
</span>  </span>tags </span>= { </span>Tier </span>= "</span>Public</span>" } </span># This is custom tag
</span>}
</span></code></pre>
WireGuard Server</a></h2>
To setup WireGuard server we need to know AMI ID of OS will be running on EC2 instance, prepare key pairs for the server and for client (peers), create a security group to control network access, create and assing Elastic IP to be sure that the server IP will be same.</p>
Generate key pairs</a></h3>
First we need to create key pairs which can be generated with this command:</p>
bash -c </span>'</span>priv=$(wg genkey); pub=$(echo $priv | wg pubkey); printf "$priv\n$pub\n"</span>'
</span></code></pre>
In output first line is privkey, second is pubkey.</p>
I will generate it twice, one for server, second for first client (peer). Then for each new peer you will need to generate new key pairs. I will same generated keys to secrets.yaml</code> file, but you can use different ways to store secrets.</p>
Generate secrets for server and first peer (do not use this keys in your setup):</p>
❯</span> bash</span> -c </span>'</span>priv=$(wg genkey); pub=$(echo $priv | wg pubkey); printf "$priv\n$pub\n"</span>'
</span>0McsAe5Y/oupwJeju+94ZFl1bmp2KIX2bBQGk0cSWXQ</span>=
</span>KeV+BlZMfCgRhdfaHqpdPp23Nu/otir+6VR02DER7lU</span>=
</span>
</span>❯</span> bash</span> -c </span>'</span>priv=$(wg genkey); pub=$(echo $priv | wg pubkey); printf "$priv\n$pub\n"</span>'
</span>SK71fHXpTo2cGKvlLLTlu+0AqpwaNWG30rzGXFIryk0</span>=
</span>wf3j1JLhXw/u6iDZf9Qhq0lU34exF8mrHdGzz/u+xFU</span>=
</span></code></pre>
Then save generated keys to secrets.yaml</code></p>
wg_server</span>:
</span>  </span>privkey</span>: </span>0McsAe5Y/oupwJeju+94ZFl1bmp2KIX2bBQGk0cSWXQ=
</span>  </span>pubkey</span>: </span>KeV+BlZMfCgRhdfaHqpdPp23Nu/otir+6VR02DER7lU=
</span>
</span>wg_peers</span>:
</span>  - </span>name</span>: </span>user1
</span>    </span>addr</span>: </span>172.16.16.1
</span>    </span>privkey</span>: </span>SK71fHXpTo2cGKvlLLTlu+0AqpwaNWG30rzGXFIryk0=
</span>    </span>pubkey</span>: </span>wf3j1JLhXw/u6iDZf9Qhq0lU34exF8mrHdGzz/u+xFU=
</span></code></pre>
Setup script</a></h3>
Then we need create WireGuard setup script which will be runned during EC2 creation. User data will be generated from this script and populated from secrets.yaml</code> file. So create templates</code> directory with two files in it: wg-init.tpl</code> (general setup script) and wg-peer.tpl</code> (peer info).</p>
#!/bin/bash
</span>sudo</span> apt update && </span>apt -y</span> install net-tools wireguard
</span>
</span># https://github.com/vainkop/terraform-aws-wireguard/blob/master/templates/user-data.txt
</span>sudo</span> mkdir</span> -p</span> /etc/wireguard
</span>sudo</span> cat > /etc/wireguard/wg0.conf <<- </span>EOF
</span>[Interface]
</span>PrivateKey = </span>$</span>{</span>wg_privkey</span>}
</span>ListenPort = </span>$</span>{</span>wg_port</span>}
</span>Address = </span>$</span>{</span>wg_cidr</span>}
</span>PostUp = sysctl -w -q net.ipv4.ip_forward=1
</span>PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUTING -o ENI -j MASQUERADE
</span>PostDown = sysctl -w -q net.ipv4.ip_forward=0
</span>PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD -o %i -j ACCEPT; iptables -t nat -D POSTROUTING -o ENI -j MASQUERADE
</span>
</span>$</span>{</span>wg_peers</span>}
</span>EOF
</span>
</span># Make sure we replace the "ENI" placeholder with the actual network interface name
</span>export </span>ENI</span>=$</span>(</span>ip</span> route get 8.8.8.8 </span>| </span>grep</span> 8.8.8.8 </span>| </span>awk </span>'</span>{print $5}</span>'</span>)
</span>sudo</span> sed</span> -i </span>"</span>s/ENI/</span>$</span>ENI</span>/g</span>" /etc/wireguard/wg0.conf
</span>
</span>sudo</span> wg-quick up wg0
</span>sudo</span> systemctl enable wg-quick@wg0
</span></code></pre>
[Peer]
</span># ${peer_name}
</span>PublicKey</span> = ${</span>peer_pubkey</span>}
</span>AllowedIPs</span> = ${</span>peer_addr</span>}
</span></code></pre>
Next we need to prepare user data script in terraform file from this templates.</p>
# MARK: Config
</span>
</span>locals</span> {
</span>  </span>wg_cidr </span>= "</span>172.16.16.0/20</span>" </span># CIDR of WireGuard Server
</span>  </span>wg_port </span>= </span>51820            </span># Port of WireGuard Server
</span>  </span>secrets </span>= yamldecode(file("</span>secrets.yaml</span>"))
</span>
</span>  </span>userdata </span>= templatefile("</span>templates/wg-init.tpl</span>", {
</span>    </span>wg_cidr    </span>= </span>local</span>.wg_cidr
</span>    </span>wg_port    </span>= </span>local</span>.wg_port
</span>    </span>wg_privkey </span>= </span>local</span>.secrets.wg_server.privkey
</span>    </span>wg_peers </span>= join("</span>\n</span>", [
</span>      for p in </span>local</span>.secrets.wg_peers :
</span>      templatefile("</span>templates/wg-peer.tpl</span>", {
</span>        </span>peer_name   </span>= p.name
</span>        </span>peer_pubkey </span>= p.pubkey
</span>        </span>peer_addr   </span>= p.addr
</span>      })
</span>    ])
</span>  })
</span>}
</span></code></pre>
Server setup</a></h3>
Then we need to select Ubuntu AMI, create Security Group and Elastic IP.</p>
# MARK: Server dependencies
</span>
</span>data </span>"</span>aws_ami</span>" "</span>ubuntu</span>" {
</span>  </span>owners      </span>= ["</span>099720109477</span>"] </span># Canonical
</span>  </span>most_recent </span>= </span>true
</span>  filter {
</span>    </span>name   </span>= "</span>name</span>"
</span>    </span>values </span>= ["</span>ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-arm64-server-*</span>"]
</span>  }
</span>}
</span>
</span>resource </span>"</span>aws_eip</span>" "</span>wireguard</span>" {
</span>  </span>tags </span>= { </span>Name </span>= "</span>wireguard</span>" }
</span>}
</span>
</span>resource </span>"</span>aws_security_group</span>" "</span>wireguard</span>" {
</span>  </span>name        </span>= "</span>${</span>local</span>.vpc_name}-wireguard</span>"
</span>  </span>description </span>= "</span>SG for WireGuard VPN Server</span>"
</span>  </span>vpc_id      </span>= </span>data</span>.aws_vpc.vpc.id
</span>  ingress {
</span>    </span>from_port   </span>= </span>local</span>.wg_port
</span>    </span>to_port     </span>= </span>local</span>.wg_port
</span>    </span>protocol    </span>= "</span>udp</span>"
</span>    </span>cidr_blocks </span>= ["</span>0.0.0.0/0</span>"]
</span>    </span>description </span>= "</span>Allow WireGuard Traffic</span>"
</span>  }
</span>  egress {
</span>    </span>from_port   </span>= </span>0
</span>    </span>to_port     </span>= </span>0
</span>    </span>protocol    </span>= "</span>-1</span>"
</span>    </span>cidr_blocks </span>= ["</span>0.0.0.0/0</span>"]
</span>  }
</span>}
</span></code></pre>
Next we can setup a server and assing Elastic IP to it:</p>
resource </span>"</span>aws_instance</span>" "</span>wireguard</span>" {
</span>  </span>count                       </span>= </span>1
</span>  </span>ami                         </span>= </span>data</span>.aws_ami.ubuntu.id
</span>  </span>instance_type               </span>= "</span>t4g.nano</span>"
</span>  </span>subnet_id                   </span>= </span>data</span>.aws_subnets.public.ids[</span>0</span>]
</span>  </span>vpc_security_group_ids      </span>= [aws_security_group.wireguard.id]
</span>  </span>user_data                   </span>= </span>local</span>.userdata
</span>  </span>user_data_replace_on_change </span>= </span>true
</span>  </span>tags                        </span>= { </span>Name </span>= "</span>wireguard</span>" }
</span>}
</span>
</span>resource </span>"</span>aws_eip_association</span>" "</span>wireguard</span>" {
</span>  </span>instance_id   </span>= aws_instance.wireguard[</span>0</span>].id
</span>  </span>allocation_id </span>= aws_eip.wireguard.id
</span>}
</span></code></pre>
Client configuration</a></h2>
To access to WireGuard server we need to have WireGuard client app</a> and valid client config with .conf</code> extension.</p>
We can generated this configs with Terraform Templates as well. So first, lets add new template file called client-conf.tpl</code>:</p>
[Interface]
</span>Address</span> = ${</span>client_addr</span>}/32
</span>PrivateKey</span> = ${</span>client_privkey</span>}
</span>DNS</span> = ${</span>client_dns</span>}
</span>
</span>[Peer]
</span>Endpoint</span> = ${</span>server_addr</span>}
</span>PublicKey</span> = ${</span>server_pubkey</span>}
</span>AllowedIPs</span> = ${</span>client_routes</span>}
</span>PersistentKeepalive</span> = 25
</span></code></pre>
And add this code to generate client config to out terraform file:</p>
# MARK: Export client configuration
</span>
</span>locals</span> {
</span>  </span># https://docs.aws.amazon.com/vpc/latest/userguide/AmazonDNS-concepts.html#AmazonDNS
</span>  </span>client_dns </span>= ["</span>169.254.169.253</span>", "</span>1.1.1.1</span>", "</span>1.0.0.1</span>"]
</span>  </span>client_routes </span>= [
</span>    </span>data</span>.aws_vpc.vpc.cidr_block, </span># All IPs in the VPC
</span>    "</span>169.254.169.253/32</span>",        </span># AWS DNS
</span>    </span># "0.0.0.0/32",                # Route all traffic through VPN (uncomment if you want this)
</span>  ]
</span>}
</span>
</span>resource </span>"</span>local_file</span>" "</span>peer_conf</span>" {
</span>  </span>for_each </span>= { for index, p in </span>local</span>.secrets.wg_peers : p.</span>name </span>=> p }
</span>  </span>filename </span>= "</span>generated/${each.value.name}.conf</span>"
</span>  </span>content </span>= templatefile("</span>templates/client-conf.tpl</span>", {
</span>    </span>server_addr    </span>= "</span>${aws_eip.wireguard.public_ip}:${</span>local</span>.wg_port}</span>",
</span>    </span>server_pubkey  </span>= </span>local</span>.secrets.wg_server.pubkey,
</span>    </span>client_addr    </span>= each.value.addr,
</span>    </span>client_privkey </span>= each.value.privkey,
</span>    </span>client_dns     </span>= join("</span>,</span>", </span>local</span>.client_dns),
</span>    </span>client_routes  </span>= join("</span>,</span>", </span>local</span>.client_routes),
</span>  })
</span>}
</span></code></pre>
Execute</a></h2>
Finally we can run terraform init</code> & terraform apply</code>. In less then a minute VPN server will be created and user client will be exported to generated/user1.conf</code>. Then we can share this file with our teammates.</p>
To check VPN connection working fine you can export .conf</code> into WireGuard client. Then for example I check my RDS instance before VPN enabled and after.</p>
❯</span> dig +short my-rds.4oexsiqydp8q.us-east-1.rds.amazonaws.com
</span>ec2-34-226-xx-xx.compute-1.amazonaws.com.
</span>34.226.xx.xx
</span>
</span>❯</span> dig +short my-rds.4oexsiqydp8q.us-east-1.rds.amazonaws.com
</span>ec2-34-226-xx-xx.compute-1.amazonaws.com.
</span>10.10.20.80
</span></code></pre>
Adding clients</a></h2>
Adding new client configuration is quite simple. We need to generate new key pair:</p>
❯</span> bash</span> -c </span>'</span>priv=$(wg genkey); pub=$(echo $priv | wg pubkey); printf "$priv\n$pub\n"</span>'
</span>ALqgaUiWd0rtcLza2K143x5RwS0Y5zwh3YwHx/L7nV0</span>=
</span>u1yQmh/G7n+S8aCp0PRuRuAuacmqYNzHLCPOcl9aS28</span>=
</span></code></pre>
Then update secrets.yaml</code> file:</p>
wg_peers</span>:
</span>  </span># ...
</span>  - </span>name</span>: </span>user2
</span>    </span>addr</span>: </span>172.16.16.2
</span>    </span>privkey</span>: </span>ALqgaUiWd0rtcLza2K143x5RwS0Y5zwh3YwHx/L7nV0=
</span>    </span>pubkey</span>: </span>u1yQmh/G7n+S8aCp0PRuRuAuacmqYNzHLCPOcl9aS28=
</span></code></pre>
And run terraform apply</code> again. Server will be re-created (!) with new configuration. Note that existing clients will be disconnected during deploy.</p>
Summary</a></h2>
WireGuard provides easy solution for creating VPN server. We created cheap and fast setup to access to internal AWS VPC with ability to create different clients configurations, which can be shared with out team members.</p>
PS. Source files of this article can be found here</a>.</em></p>


Fast multi-arch Docker build for Rust projects
2024-10-04T00:00:00+00:00
</p>
Rust allows programs to run very fast and memory efficient, but it has a cost – compile time. In web development, it is standard practice to deliver programs as Docker images and then run them in Kubernetes / Amazon ECS</a> / Docker Compose / etc. After the popularity of ARM processors in recent years, programmers have been faced with the additional step of preparing multi-arch Docker images (meaning that the same image should be able to run natively on both x86-64/amd64 and aarch64/arm64 processors).</p>
Preparing Docker images for interpreted languages is usually not a problem – Docker Buildx is used, which in turn runs inside the emulator and builds the image for each architucture natively. NodeJS / Python just install dependencies via npm / pip, copy the project code, does a little polishing and that's pretty much it. Even in compiled Go this approach works fine because Go has an extensive standard library. In the case of Rust, compiling even a simple web application is a rebuilding of the “universe” – almost any web application requires: HTTP-server, HTTP-client (which in turn requires a library to work with cryptography for https), asynchronous runtime (tokie, etc), serialization/desirialization (serde, serde_json); which in Rust should be installed as external libraries (crates) and should be compiled each time when program builded.</p>
Although Rust compiler has a lot of work to do, it can do it quickly. Even not strongest CI, will build an average project in a couple of minutes. However, this is true only in case of building on native architecture (for example building amd64 binary on an amd64 processor). As soon as we need to build a multi-arch image, we have to do emulation and the compilation speed drops dramatically. For example, on my simple public project – ghstats</a>, building multi-arch image from scratch took about 50 minutes, when same build for native architecture takes 2-3 minutes.</p>
Building time can be optimized by proper usage of Docker layers, so that the step with rebuilding dependencies occurs only when they are actually changed. So only the first build will be long and following builds will be fast. Unfortunately, Rust infrastructure have a problem in this point –
any change in Cargo.toml</code> file (for example version number) will invalidate Docker layer and triggers full rebuild of all dependencies.</p>
Problem definition</a></h2>
So there are two problems with building multi-arch Docker images for Rust projects:</p>

Layer invalidation and full rebuild on any change in Cargo.toml </code></li>
Very slow multi-arch build due to emulation</li>
</ol>
Better dependencies building</a></h2>
Easiest way to solve the first problem is to use cargo-chef</code> which was created exacly for it. cargo-chef</code> converts Cargo.toml</code> & Cargo.lock</code> into a special recipe.json</code> file that will remain unchanged until project dependencies are unchanged. Then we can use this json file to cache expensive Docker layer that compiles dependencies.</p>
Dockerfile with cargo-chef</code> will use multi-stage build splited to 5 parts:</p>

Installing cargo-chef</code> and build dependencies (OpenSSL, linux-headers, etc)</li>
Preparing recipe.json</code> file with project dependencies description</li>
Installing & builiding project dependencies from recipe.json</code></li>
Building whole project</li>
Copy binaries to runtime image and final polishing</li>
</ol>
Example of Dockerfile with cargo-chef</code> and multi-stage build (I use alpine in my projects, but you can use other base-images).</p>
# (1) installing cargo-chef & build deps
</span>FROM</span> rust:alpine </span>AS </span>chef
</span>WORKDIR </span>/app
</span>RUN </span>cargo install --locked cargo-chef
</span>RUN </span>apk add --no-cache musl-dev openssl-dev
</span>
</span># (2) prepating recipe file
</span>FROM</span> chef </span>AS </span>planner
</span>COPY</span> . .
</span>RUN </span>cargo chef prepare --recipe-path recipe.json
</span>
</span># (3) building project deps, cache magic happen on COPY command
</span>FROM</span> chef </span>AS </span>builder
</span>COPY</span> --from=</span>planner</span> /app/recipe.json recipe.json
</span>RUN </span>cargo chef cook --recipe-path recipe.json --release
</span>
</span># (4) actual project build
</span>COPY</span> . .
</span>RUN </span>cargo build -r
</span>
</span># (5) runtime image, you can use any base image you want
</span>FROM</span> alpine:latest </span>AS </span>runtime
</span>WORKDIR </span>/app
</span>COPY</span> --from=</span>builder</span> /app/target/release/prog /app/prog
</span>CMD "</span>/app/prog</span>"
</span></code></pre>
This approach will speed up Docker image build many times while project dependencies are not changed. Because Docker builds stages separately, planner</em> stage will be executed each time (but cargo-chef</code> is fast!) and builder</em> stage will be partially cached until recipe.json</code> file are same.</p>
If you need single architecture build this will work fine. But if you need multi-arch images, this approach still be slow.</p>
Multi-arch image with cross-compilation</a></h2>
Multi-arch builds using Docker Buildx run very slow due to QEMU emulation. If we get rid of emulation, compilation will be at full speed. Rust has built-in cross-compilation for other architectures, so we adapt it in Docker build.</p>
Cross-compilation in Rust itself works fine, but some crates are based on C libraries (OpenSSL, SQLite, etc). Compiling and linking C code is quite complicated and not always clear (usually you have to look for error codes somewhere on Stack Overflow or Github Issues until you get the right set of compilers/header files added). There is another tool that surprisingly solves the problem of cross-compiling C code very well – Zig (actually this is programming language, but they have build toolchain as well).</p>
To connect Zig build toolchain with Rust I will use cargo-zigbuild</code> crate. I other Docker file looks pretty same to our cargo-chef</code> variant, expect I added second target in Cargo to build and cargo build</code> replaced with cargo zigbuild</code>.</p>
# (1) this stage will be run always on current arch
</span># zigbuild & Cargo targets added
</span>FROM</span> --platform=$BUILDPLATFORM rust:alpine AS chef
</span>WORKDIR </span>/app
</span>ENV </span>PKG_CONFIG_SYSROOT_DIR=/
</span>RUN </span>apk add --no-cache musl-dev openssl-dev zig
</span>RUN </span>cargo install --locked cargo-zigbuild cargo-chef
</span>RUN </span>rustup target add x86_64-unknown-linux-musl aarch64-unknown-linux-musl
</span>
</span># (2) nothing changed
</span>FROM</span> chef </span>AS </span>planner
</span>COPY</span> . .
</span>RUN </span>cargo chef prepare --recipe-path recipe.json
</span>
</span># (3) building project deps: need to specify all targets; zigbuild used
</span>FROM</span> chef </span>AS </span>builder
</span>COPY</span> --from=</span>planner</span> /app/recipe.json recipe.json
</span>RUN </span>cargo chef cook --recipe-path recipe.json --release --zigbuild \
</span>    --target x86_64-unknown-linux-musl --target aarch64-unknown-linux-musl
</span>
</span># (4) actuall project build for all targets
</span># binary renamed to easier copy in runtime stage
</span>COPY</span> . .
</span>RUN </span>cargo zigbuild -r \
</span>    --target x86_64-unknown-linux-musl --target aarch64-unknown-linux-musl && \
</span>    mkdir /app/linux && \
</span>    cp target/aarch64-unknown-linux-musl/release/prog /app/linux/arm64 && \
</span>    cp target/x86_64-unknown-linux-musl/release/prog /app/linux/amd64
</span>
</span># (5) this staged will be emulated as was before
</span># TARGETPLATFORM usage to copy right binary from builder stage
</span># ARG populated by docker itself
</span>FROM</span> alpine:latest </span>AS </span>runtime
</span>WORKDIR </span>/app
</span>ARG </span>TARGETPLATFORM
</span>COPY</span> --from=</span>builder</span> /app/${TARGETPLATFORM} /app/prog
</span>CMD "</span>/app/prog</span>"
</span></code></pre>
In general that all. Such build approach will work faster. For my project:</p>

initial build 50 min → 13 min (3.8x)</li>
code updated 7 min → 3 min (2.3x)</li>
</ol>
Additional performance can be achieved by moving first step into separate base image, then our main Docker image will not need to build cargo-zibguild</code> and cargo-chef</code> (-2 min from initial build).</p>

Note: If you already know your server architecture (if you build your docker for a specific server), you can remove everything link to the other architecture. This will lead you to have performance improvement in both initial build and updates.</p>
</blockquote>


https://www.docker.com/blog/faster-multi-platform-builds-dockerfile-cross-compilation-guide</a></li>
</ol>


apigen-ts – Simple TypeScript API Client Generator
2024-09-04T00:00:00+00:00

OpenAPI / Swagger is a good way to describe API contracts between backend / frontend or different services. Nowadays frontend is almost always TypeScript + some framework (React, Vue, Angular, etc). Backend in turn can be in NodeJS (and then in principle you can share contracts with code reuse), or in another language (e.g. Python / Go / Scala / Rust) and then code reuse and API synchronization becomes a problem.</p>
OpenAPI can solve this problem through codegen – backend should qualitatively describe the available API methods and data types, after that you can use the client generation utility and get ready typed TypeScript code.</p>
OpenAPI has a standard tool for such generation, but it has a number of problems:</p>

Tool is written in Java, which requires special software to be installed on dev computer / CI</li>
Generate client is extensive and consists of many .ts</code> files, which makes it not user friendly (in my opinion)</li>
</ul>
In JavaScript ecosystem it is common to use npm (or yarn / pnpm – whatever you like more) as a package manager. But the point is the same: packages are written in JavaScript, installed from one place, versioned.</p>
In npm you can find several alternative apigen utilities written in TypeScript which provides a better development experience. I used one of those too (swagger-typescript-api), until they changed interfaces a few times and I had to fix my several projects. I tried to find an alternative in npm, but all available generators (as of late 2023) also have their own problems: generate a lot of files, are difficult to configure, break compatibility between versions, require to install additional packages.</p>
Meet apigen-ts</a></h2>
I have a couple of projects in NestJS / FastAPI in support mode, for which I just need a stable API client generation utility that is simple and won't change often. So I wrote my own TypeScript HTTP API generation package from Swagger specification – apigen-ts</a>.</p>
Originally I wrote apigen-ts</code> for personal projects, but then I share it to my friends and they also started to use apigen-ts</code> in their projects. Now I want to share it with the community, maybe it will be useful for someone too. So library features:</p>

supports Swagger / OpenAPI v2 / v3 / v3.1 (parsed with redocly</a>)</li>
generates a typed TypeScript client</li>
single output file, single api class</li>
exports data models for use in your code</li>
automatic code formatting with prettier (peer dependency)</li>
ability to automatic dates parsing</a></li>
</ul>
Usage</a></h2>
Usage is as simple as possible and is divided into two steps: generating a client from specification (local file or remote url; yaml or json), using the generated typed client in the app.</p>
Generation of an arbitrary client can be performed by command:</p>
npx -y</span> apigen-ts https://petstore3.swagger.io/api/v3/openapi.json ./api-client.ts
</span></code></pre>
Or you can install apigen-ts</code> as a dev dependency (npm i -D apigen-ts</code>) and create a task to generate the client in package.json</code>:</p>
{
</span>  "</span>name</span>": "</span>my-frontend</span>",
</span>  "</span>scripts</span>": {
</span>    "</span>apigen</span>": "</span>apigen-ts http://127.0.0.1:3000/docs-json ./api-client.ts --parse-dates && tsc --noEmit</span>"
</span>  }
</span>}
</span></code></pre>
Then use: npm run apigen</code> or yarn apigen</code>.</p>
Generated file contains a client class named ApiClient</code> and TypeScript types used in API endpoints. Next you can use them in app code in the following way:</p>
import </span>{ </span>ApiClient</span>, </span>Pet </span>} </span>from </span>"</span>./api-client</span>";
</span>
</span>const </span>api </span>= new ApiClient({
</span>  baseUrl: "</span>https://example.com</span>",
</span>  headers: { Authorization: "</span>secret-token</span>" },
</span>});
</span>
</span>// GET https://example.com/pet/findByStatus?status=sold
</span>await </span>api</span>.</span>pet</span>.</span>findPetsByStatus</span>({ status: "</span>sold</span>" }); </span>// -> Pet[]
</span></code></pre>
Generator tries to create method names in format like {domain}.{operation}</code>. The types used in Swagger spec can be imported directly into code (in example above Pet</code> is generated type). This approach makes it easy to track changes in contract and fix the application code where types are updated or methods are changed (of course if you describe your Swagger well). Just run new api generation and check frontend code for errors afterwards with TypeScript.</p>
Also apigen-ts</code> works well in CI pipelines, when on new commits to the backend part you run frontend api client generation and check code compatibility.</p>
The generated client uses native fetch API (does not require additional packages) and can be used both in browser and NodeJS environment. If you use prettier</code> in your project, the generated client will be automatically formatted according to project settings.</p>
More configuration options and customization abilities can be in readme</a>.</p>
Conclusion</a></h2>
Backend ↔ frontend (or backend ↔ backend) communication is monotonous and repetitive work from project to project. Do not waste time and effort on manual writing of api clients. Instead use simple tools to automate process and keep code quality.</p>

You can support this project with a star ⭐️ or leave an issue / feature request on Github page</a>.</p>


Table sorting and pagination with HTMX
2024-08-18T00:00:00+00:00
I recently tried HTMX for my new project – ghstats</a> – a dashboard of Github repository traffic in a single interface for longer than 14 days. This project was planned as a self-hosted service, so I was thinking about a really simple and memory-less tech stack. Last time I played with Rust, so I decided to use it instead of NodeJS / Python. Of course, if I'm generating static HTML on the server side, I have two options for implementing table sorting and pagination: use query parameters and do everything on backend, or use some JavaScript to call data from the backend and render the table on the client side. But HTMX offers a new, third way: write all the logic on the backend and replace the necessary parts of HTML with just a few tag attributes. Let's see how it works.</p>
Initial setup</a></h2>
To demonstrate table sorting and pagination, I'll be using the same stack I used for my service: axum</code> as a backend framework and maud</code> for HTML templating (it's a DSL over Rust templates).</p>
Let's create new project with:</p>
cargo</span> init htmx-example && </span>cd-html-example
</span></code></pre>
Then install dependencies:</p>
cargo</span> add tokio axum maud</span> --features</span> tokio/full,maud/axum
</span></code></pre>
and do initial setup in src/main.rs</code> file:</p>
use </span>axum::{response::IntoResponse, routing::get, Router};
</span>use </span>maud::html;
</span>
</span>async </span>fn </span>index_page</span>() -> impl IntoResponse {
</span>  html! {
</span>    h1 { "</span>Hello, World!</span>" }
</span>  }
</span>}
</span>
</span>#[</span>tokio</span>::</span>main</span>]
</span>async </span>fn </span>main</span>() {
</span>  </span>let</span> service = Router::new().</span>route</span>("</span>/</span>", </span>get</span>(index_page)).</span>into_make_service</span>();
</span>  </span>let</span> listener = tokio::net::TcpListener::bind("</span>127.0.0.1:8080</span>").await.</span>unwrap</span>();
</span>  axum::serve(listener, service).await.</span>unwrap</span>();
</span>}
</span></code></pre>
Then we cant run this with cargo watch -x run</code> and open http://127.0.0.1:8080</code> in the browser. "Hello, World!" should be displayed.</p>
Creating a static table</a></h2>
Let's add crates to generate random data:</p>
cargo</span> add fake rand rand_chacha</span> --features</span> fake/derive
</span></code></pre>
and define simple Contact</code> struct (like our application is simple CMS).</p>
use </span>fake::faker::company::en::*;
</span>use </span>fake::faker::internet::en::*;
</span>use </span>fake::faker::name::en::*;
</span>use </span>fake::faker::phone_number::en::*;
</span>use </span>fake::{Dummy, Fake, Faker};
</span>use </span>rand::prelude::*;
</span>use </span>rand_chacha::ChaCha8Rng;
</span>
</span>#[</span>derive</span>(Debug, Clone, Dummy)]
</span>struct </span>Contact {
</span>  </span>id</span>: </span>u64</span>,
</span>
</span>  #[</span>dummy</span>(faker = "</span>Name()</span>")]
</span>  </span>name</span>: String,
</span>
</span>  #[</span>dummy</span>(faker = "</span>CompanyName()</span>")]
</span>  </span>company</span>: String,
</span>
</span>  #[</span>dummy</span>(faker = "</span>FreeEmail()</span>")]
</span>  </span>email</span>: String,
</span>
</span>  #[</span>dummy</span>(faker = "</span>PhoneNumber()</span>")]
</span>  </span>phone</span>: String,
</span>}
</span>
</span>fn </span>get_contacts</span>(</span>n</span>: </span>usize</span>) -> Vec<Contact> {
</span>  </span>let mut</span> items = Vec::with_capacity(n);
</span>  </span>let mut</span> rng = ChaCha8Rng::seed_from_u64(</span>42</span>);
</span>  </span>for </span>_ in </span>0</span>..n {
</span>    items.</span>push</span>(Faker.</span>fake_with_rng</span>(&</span>mut</span> rng));
</span>  }
</span>  items
</span>}
</span>
</span>async </span>fn </span>index_page</span>() -> impl IntoResponse {
</span>	</span>let</span> contacts = </span>get_contacts</span>(</span>100</span>);
</span>	println!("</span>>> </span>{:?}</span>", contacts[</span>0</span>]);
</span>	println!("</span>>> </span>{:?}</span>", contacts[</span>1</span>]);
</span>
</span>  html! {
</span>    h1 { "</span>Hello, World!</span>" }
</span>  }
</span>}
</span></code></pre>
Now on each request same random contacts will be generated, so we can use this data in our table.
get_contacts</code> function emulates database query in real environment.</p>
I will also add base layout for page with PicoCSS</a> framework to have nice look:</p>
fn </span>base</span>(</span>html</span>: maud::Markup) -> impl IntoResponse {
</span>  html! {
</span>    html {
</span>      head {
</span>        meta charset="</span>utf-8</span>" {}
</span>        title { "</span>Contacts</span>" }
</span>
</span>        link rel="</span>stylesheet</span>" href="</span>https://unpkg.com/@picocss/pico@2.0</span>" {}
</span>        script src="</span>https://unpkg.com/htmx.org@2.0</span>" {}
</span>      }
</span>      body class="</span>container</span>" {
</span>        { (html) }
</span>      }
</span>    }
</span>  }
</span>}
</span></code></pre>
And finally update index_page</code> to render contact table:</p>
async </span>fn </span>index_page</span>() -> impl IntoResponse {
</span>  </span>let</span> contacts = </span>get_contacts</span>(</span>100</span>);
</span>
</span>  </span>base</span>(html! {
</span>    table {
</span>      thead {
</span>        tr {
</span>          th { "</span>ID</span>" }
</span>          th { "</span>Name</span>" }
</span>          th { "</span>Company</span>" }
</span>          th { "</span>Email</span>" }
</span>          th { "</span>Phone</span>" }
</span>        }
</span>      }
</span>      tbody {
</span>        @</span>for</span> contact in contacts {
</span>          tr {
</span>            td { (contact.id) }
</span>            td { (contact.name) }
</span>            td { (contact.company) }
</span>            td { (contact.email) }
</span>            td { (contact.phone) }
</span>          }
</span>        }
</span>      }
</span>    }
</span>  })
</span>}
</span></code></pre>
We can request again http://127.0.0.1:8080</code> and see our table with random contacts (which on each request at same).</p>
Sorting table</a></h2>
Now is the time to add sorting to our table. We will use HTMX to send requests to the server and get sorted HTML back. First, we need to add sorting logic to our backend. We will use query parameters to pass sorting column and direction.</p>
To make it easier, I will add struct TableFilter</code> which will represent sorting and pagination parameters, for code simplicity I will parse Request manually rather than using Query</code> extractor and serde</code> crate:</p>
use </span>std::collections::HashMap;
</span>use </span>axum::extract::{Query, Request};
</span>
</span>#[</span>derive</span>(Debug)]
</span>struct </span>TableFilter {
</span>  </span>sort</span>: String,
</span>  </span>order</span>: String,
</span>  </span>page</span>: </span>u32</span>,
</span>  </span>per_page</span>: </span>u32</span>,
</span>}
</span>
</span>fn </span>parse_table_filter</span>(</span>req</span>: &Request) -> TableFilter {
</span>  </span>let</span> qs: Query<HashMap<String, String>> = Query::try_from_uri(req.</span>uri</span>()).</span>unwrap</span>();
</span>  </span>let</span> sort = qs.</span>get</span>("</span>sort</span>").</span>unwrap_or</span>(&"</span>name</span>".</span>to_string</span>()).</span>to_string</span>();
</span>  </span>let</span> order = qs.</span>get</span>("</span>order</span>").</span>unwrap_or</span>(&"</span>desc</span>".</span>to_string</span>()).</span>to_string</span>();
</span>  </span>let</span> page = qs.</span>get</span>("</span>page</span>").</span>unwrap_or</span>(&"</span>1</span>".</span>to_string</span>()).</span>parse</span>().</span>unwrap</span>();
</span>  </span>let</span> per_page = qs.</span>get</span>("</span>per_page</span>").</span>unwrap_or</span>(&"</span>10</span>".</span>to_string</span>()).</span>parse</span>().</span>unwrap</span>();
</span>  TableFilter { sort, order, page, per_page }
</span>}
</span>
</span></code></pre>
Next, we need to create a function to render th</code> with HTMX attributes to handle sorting:</p>

</span>fn </span>th</span>(</span>title</span>: &</span>str</span>, </span>qs</span>: &TableFilter) -> maud::Markup {
</span>  </span>let</span> id = title.</span>to_lowercase</span>();
</span>  </span>let</span> order = </span>if</span> qs.sort == id && qs.order == "</span>desc</span>" { "</span>asc</span>" } </span>else </span>{ "</span>desc</span>" };
</span>  </span>let</span> url = format!("</span>/?sort=</span>{}</span>&order=</span>{}</span>&page=1&per_page=</span>{}</span>", id, order, qs.per_page);
</span>
</span>  html! {
</span>    th scope="</span>col</span>" style="</span>cursor: pointer;</span>"
</span>      hx-trigger="</span>click</span>"
</span>      hx-get=(url)
</span>      hx-target="</span>#contacts_table</span>"
</span>      hx-swap="</span>outerHTML</span>"
</span>    {
</span>      (title)
</span>      @</span>if</span> qs.sort == id {
</span>        span style="</span>margin-left: 0.5em;</span>" {
</span>          @</span>if</span> qs.order == "</span>asc</span>" { "</span>↑</span>" } @</span>else </span>{ "</span>↓</span>" }
</span>        }
</span>      }
</span>    }
</span>  }
</span>}
</span></code></pre>
Update get_contacts</code> to accept TableFilter</code> and sort items / do pagination:</p>

</span>fn </span>get_contacts</span>(</span>n</span>: </span>usize</span>, </span>qs</span>: &TableFilter) -> (Vec<Contact>, </span>u32</span>) {
</span>  </span>let mut</span> items: Vec<Contact> = Vec::with_capacity(n);
</span>  </span>let mut</span> rng = ChaCha8Rng::seed_from_u64(</span>42</span>);
</span>  </span>for </span>_ in </span>0</span>..n {
</span>    items.</span>push</span>(Faker.</span>fake_with_rng</span>(&</span>mut</span> rng));
</span>  }
</span>
</span>  items.</span>sort_by</span>(|</span>a</span>, </span>b</span>| {
</span>    </span>let</span> cmp = </span>match</span> qs.sort.</span>as_str</span>() {
</span>      "</span>id</span>" => a.id.</span>cmp</span>(&b.id),
</span>      "</span>name</span>" => a.name.</span>cmp</span>(&b.name),
</span>      "</span>company</span>" => a.company.</span>cmp</span>(&b.company),
</span>      "</span>email</span>" => a.email.</span>cmp</span>(&b.email),
</span>      "</span>phone</span>" => a.phone.</span>cmp</span>(&b.phone),
</span>      _ => a.id.</span>cmp</span>(&b.id),
</span>    };
</span>
</span>    </span>if</span> qs.order == "</span>asc</span>" {
</span>      cmp
</span>    } </span>else </span>{
</span>      cmp.</span>reverse</span>()
</span>    }
</span>  });
</span>
</span>  </span>let</span> pages = (items.</span>len</span>() as </span>f64 </span>/ qs.per_page as </span>f64</span>).</span>ceil</span>() as </span>u32</span>;
</span>  </span>let</span> range_s = (qs.page as </span>usize </span>- </span>1</span>) * qs.per_page as </span>usize</span>;
</span>  </span>let</span> range_e = qs.page as </span>usize </span>* qs.per_page as </span>usize</span>;
</span>  </span>let</span> items = items[range_s..range_e].</span>to_vec</span>();
</span>
</span>  (items, pages)
</span>}
</span></code></pre>
And finally, we can update our index_page</code> to use TableFilter</code> and th</code>:</p>
async </span>fn </span>index_page</span>(</span>req</span>: Request) -> impl IntoResponse {
</span>  </span>let</span> qs = </span>parse_table_filter</span>(&req);
</span>  </span>let </span>(contacts, pages) = </span>get_contacts</span>(</span>100</span>, &qs);
</span>
</span>  </span>base</span>(html! {
</span>  	</span>// id added to tell HTMX where to put new data
</span>    table id="</span>contacts_table</span>" {
</span>      thead {
</span>        tr {
</span>          (</span>th</span>("</span>ID</span>", &qs))
</span>          (</span>th</span>("</span>Name</span>", &qs))
</span>          (</span>th</span>("</span>Company</span>", &qs))
</span>          (</span>th</span>("</span>Email</span>", &qs))
</span>          (</span>th</span>("</span>Phone</span>", &qs))
</span>        }
</span>      }
</span>      </span>// same as before
</span>    }
</span>  })
</span>}
</span></code></pre>
That's it! Now we can sort our table by clicking on the column header. HTMX will send a request to the server with sorting parameters and replace the table with the new sorted data. Let's me explain HTMX attributes:</p>

hx-trigger</code> - event which will trigger the request, in our case it's click</code></li>
hx-get</code> - URL to send the request and HTTP method (can be GET or POST)</li>
hx-target</code> - CSS selector to replace with new data</li>
hx-swap</code> - how to replace the target, in our case it's outerHTML</code> which will replace the whole table (by default it's inserting new data into the target)</li>
</ul>
All this attributes handled automatically by HTMX, so we just need to include HTMX script in <head></code>.</p>
Pagination</a></h2>
In previous step we already parse pagination filters to struct, so lets add code to render pagination links. I will use code from my another react</code> project, which originally from StackOverflow answer:</p>
// https://stackoverflow.com/a/70263913
</span>fn </span>calc_pagination</span>(</span>page</span>: </span>usize</span>, </span>total</span>: </span>usize</span>, </span>len</span>: </span>usize</span>) -> Vec<Option<</span>usize</span>>> {
</span>  </span>use </span>std::cmp::{max, min};
</span>
</span>  </span>let</span> len = </span>if</span> len == </span>0 </span>{ </span>5 </span>} </span>else </span>{ len };
</span>  </span>let</span> total = </span>max</span>(total, page);
</span>  </span>let</span> start =
</span>    </span>max</span>(</span>1</span>, </span>min</span>(page as </span>isize </span>- ((len - </span>3</span>) as </span>isize </span>/ </span>2</span>), total as </span>isize </span>- len as </span>isize </span>+ </span>2</span>))
</span>      as </span>usize</span>;
</span>  </span>let</span> end = </span>min</span>(total, </span>max</span>(page + (len - </span>2</span>) / </span>2</span>, len - </span>1</span>));
</span>
</span>  </span>let mut</span> result = Vec::new();
</span>
</span>  </span>if</span> start > </span>2 </span>{
</span>    result.</span>push</span>(Some(</span>1</span>));
</span>    result.</span>push</span>(None);
</span>  } </span>else if</span> start > </span>1 </span>{
</span>    result.</span>push</span>(Some(</span>1</span>));
</span>  }
</span>
</span>  </span>for</span> i in start..=end {
</span>    result.</span>push</span>(Some(i));
</span>  }
</span>
</span>  </span>if</span> end < total - </span>1 </span>{
</span>    result.</span>push</span>(None);
</span>    result.</span>push</span>(Some(total));
</span>  } </span>else if</span> end < total {
</span>    result.</span>push</span>(Some(total));
</span>  }
</span>
</span>  result
</span>}
</span></code></pre>
Let's define two more helper function which will render pagination link and delimiter (like ...</code>):</p>
fn </span>pagination_link</span>(</span>qs</span>: &TableFilter, </span>i</span>: </span>u32</span>) -> maud::Markup {
</span>  </span>let</span> url = format!("</span>/?sort=</span>{}</span>&order=</span>{}</span>&page=</span>{}</span>&per_page=</span>{}</span>", qs.sort, qs.order, i, qs.per_page);
</span>  html! {
</span>    li {
</span>      a href=(url)
</span>        hx-trigger="</span>click</span>"
</span>        hx-get=(url)
</span>        hx-target="</span>#contacts_table</span>"
</span>        hx-swap="</span>outerHTML</span>"
</span>      { (i) }
</span>    }
</span>  }
</span>}
</span>
</span>fn </span>pagination_delim</span>() -> maud::Markup {
</span>  html! {
</span>    li {
</span>      span { "</span>...</span>" }
</span>    }
</span>  }
</span>}
</span></code></pre>
And finally we can write pagination function, which will return list of links and delimiters:</p>
fn </span>pagination</span>(</span>pages</span>: </span>u32</span>, </span>qs</span>: &TableFilter) -> maud::Markup {
</span>  </span>let</span> items = </span>calc_pagination</span>(qs.page as </span>usize</span>, pages as </span>usize</span>, </span>5</span>);
</span>
</span>  html!(
</span>    nav {
</span>      ul {
</span>        @</span>for</span> item in items {
</span>          @</span>if let </span>Some(i) = item {
</span>            (</span>pagination_link</span>(qs, i as </span>u32</span>))
</span>          } @</span>else </span>{
</span>            (</span>pagination_delim</span>())
</span>          }
</span>        }
</span>      }
</span>    }
</span>  )
</span>}
</span>
</span>async </span>fn </span>index_page</span>(</span>req</span>: Request) -> impl IntoResponse {
</span>	</span>// ...
</span>	</span>base</span>(html! {
</span>		</span>// ..
</span>		</span>// add pagination call after table
</span>		(</span>pagination</span>(pages, &qs))
</span>	})
</span>}
</span></code></pre>
Partial HTML updates</a></h2>
Before this step, we always replaced the whole page with new HTML content. This can be done without HTMX at all, so what the point of using it? It's right, we have very simple layout before. Let's make it more complex to show nice feature of HTMX – partial updates.</p>
First of all let's move table generation into separate function:</p>
fn </span>get_contacts_table</span>(</span>contacts</span>: Vec<Contact>, </span>qs</span>: &TableFilter, </span>pages</span>: </span>u32</span>) -> maud::Markup {
</span>  html! {
</span>  	</span>// id moved from table to div
</span>  	div id="</span>contacts_table</span>" {
</span>	    table {
</span>	      thead {
</span>	        tr {
</span>	          (</span>th</span>("</span>ID</span>", &qs))
</span>	          (</span>th</span>("</span>Name</span>", &qs))
</span>	          (</span>th</span>("</span>Company</span>", &qs))
</span>	          (</span>th</span>("</span>Email</span>", &qs))
</span>	          (</span>th</span>("</span>Phone</span>", &qs))
</span>	        }
</span>	      }
</span>	      tbody {
</span>	        @</span>for</span> contact in contacts {
</span>	          tr {
</span>	            td { (contact.id) }
</span>	            td { (contact.name) }
</span>	            td { (contact.company) }
</span>	            td { (contact.email) }
</span>	            td { (contact.phone) }
</span>	          }
</span>	        }
</span>	      }
</span>	    }
</span>
</span>	    (</span>pagination</span>(pages, &qs))
</span>   	}
</span>  }
</span>}
</span></code></pre>
Now we can call this code in any other place and have contacts table. Now let's add some navigation and other content to our main page:</p>
async </span>fn </span>index_page</span>(</span>req</span>: Request) -> impl IntoResponse {
</span>  </span>let</span> qs = </span>parse_table_filter</span>(req);
</span>  </span>let </span>(contacts, pages) = </span>get_contacts</span>(</span>1000</span>, &qs);
</span>
</span>  </span>base</span>(html! {
</span>    nav {
</span>      ul {
</span>        li { a href="</span>/dashboard</span>" { "</span>Dashboard</span>" } }
</span>        li { a href="</span>/</span>" { "</span>Contacts</span>" } }
</span>        li { a href="</span>/settings</span>" { "</span>Settings</span>" } }
</span>      }
</span>    }
</span>
</span>    h1 { "</span>Contacts</span>" }
</span>    (</span>get_contacts_table</span>(contacts, &qs, pages))
</span>  })
</span>}
</span></code></pre>
Right now in the application we have a simple navigation. But when we interact with the table, a full page will be returned, and the result will look like a nested page in a nested page... This is wrong. Let's fix this with HTMX (and this is actually where HTMX comes power).</p>
When browser loads the page it load as regular page with default headers. When request send by HTMX it will set extra headers like HX-Request</code> and HX-Target</code>. We can use this headers to detect which part of HTML should be returned. Let's add this code to our index_page</code>:</p>
fn </span>get_hx_target</span>(</span>req</span>: &Request) -> Option<&</span>str</span>> {
</span>  </span>match</span> req.</span>headers</span>().</span>get</span>("</span>hx-target</span>") {
</span>    Some(x) => Some(x.</span>to_str</span>().</span>unwrap_or_default</span>()),
</span>    None => None,
</span>  }
</span>}
</span>
</span>async </span>fn </span>index_page</span>(</span>req</span>: Request) -> impl IntoResponse {
</span>  </span>let</span> qs = </span>parse_table_filter</span>(&req);
</span>  </span>let </span>(contacts, pages) = </span>get_contacts</span>(</span>1000</span>, &qs);
</span>
</span>  </span>match </span>get_hx_target</span>(&req) {
</span>    Some("</span>contacts_table</span>") => </span>return </span>get_contacts_table</span>(contacts, &qs, pages).</span>into_response</span>(),
</span>    _ => {}
</span>  }
</span>
</span>  </span>base</span>(html! {
</span>  	</span>// ... this part is the same
</span>  })
</span>  .</span>into_response</span>() </span>// added into_response() to relax rust type inference
</span>}
</span></code></pre>
So in this code we check if hx-target</code> header is present. If not – we return full page. If it's present and equal to contacts_table</code> – we return only table. This is how partial updates works in HTMX. We can split page to smaller parts and update it independently.</p>
Conclusion</a></h2>
HTMX is good utility for simple application which allows to add some interactivity without writing a lot of JavaScript code. It easy to use and can be integrated with any backend language, because not require any special server-side support. It's also very lightweight and can be used in any project without any additional dependencies.</p>
PS. Source files of this article can be found here</a>.</em></p>


How to get macOS power metrics with Rust?
2024-08-13T00:00:00+00:00
In this post, I’m going to talk about how macmon</a> works, specifically where it gets system metrics.</p>
MacOS has a built-in powermetics</code> utility that can show the current CPU core frequencies, utilisation, and power consumption. This program should take these values from the system somewhere. So the first thing to check is what shared libraries and function calls (symbols) are used.</p>
There are several programs can do this:</p>

otool</code> — show list of shared libraries</li>
nm</code> — see the list of used symbols</li>
strings</code> — just prints all strings from a binary file, but it may be useful to understand what is going on in the program.</li>
</ul>
> otool -L /usr/bin/powermetrics
</span> /usr/lib/libIOReport.dylib (compatibility version 1.0.0, current version 1.0.0)
</span> /usr/lib/libpmsample.dylib (compatibility version 1.0.0, current version 2.0.0)
</span> /usr/lib/libpmenergy.dylib (compatibility version 1.0.0, current version 2.0.0)
</span> /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
</span>...
</span>
</span>> nm -a /usr/bin/powermetrics
</span>/usr/bin/powermetrics (for architecture arm64e):
</span>  ...
</span>  U _IOPMCopyPowerStateInfo
</span>  U _IORegistryEntryCreateCFProperties
</span>  U _IORegistryEntryCreateCFProperty
</span>  U _IORegistryEntryFromPath
</span>  ...
</span></code></pre>
This data provides some starting point. Next I just did a search on Github to see how these libs might have been used in other repositories.</p>
Getting CPU / GPU Usage</a></h2>
The first thing I found is that macOS has an IOReport</code> shared library that returns a lot of hardware information. Including it can do sampling and return CPU/GPU & energy usage.</p>
IOReport</code> operates on a subscription basis. It is possible to select interesting data channels (or all available) and perform sampling on it in order to obtain actual metrics. The channel has a group / subgroup name (sometimes may not exist). I’m interesting in “Energy Model”, “CPU Stats” / “CPU Core Performance States” and “GPU Stats” / “GPU Performance States”. These values were found by comparing available channels names with the list of those strings used in powermetrics</code> (strings /usr/bin/powermetrics</code>).</p>
To subscribe to a channel and receive updates, it is first necessary to get internal channel struct as CFDictionary</code> using IOReportCopyChannelsInGroup</code> and merge all the dicts into one using IOReportMergeChannels</code>. Then merged dictionary can be used to create an subscription using IOReportCreateSubscription</code>. Once subscription object obtained, updates can be received with IOReportCreateSamples</code> and IOReportCreateSamplesDelta</code>. All together it looks something like this:</p>
// simplified code without error handling
</span>use </span>std::ptr::null;
</span>
</span>let mut</span> channels = vec![];
</span>channels.</span>push</span>(IOReportCopyChannelsInGroup("</span>Energy Model</span>", </span>null</span>(), </span>0</span>, </span>0</span>, </span>0</span>));
</span>channels.</span>push</span>(IOReportCopyChannelsInGroup("</span>CPU Stats</span>", "</span>CPU Core Performance States</span>", </span>0</span>, </span>0</span>, </span>0</span>));
</span>
</span>let</span> chan = channels[</span>0</span>];
</span>for</span> i in </span>1</span>..channels.</span>len</span>() {
</span>  IOReportMergeChannels(chan, channels[i], </span>null</span>());
</span>}
</span>
</span>let</span> size = CFDictionaryGetCount(chan);
</span>let</span> chan = CFDictionaryCreateMutableCopy(kCFAllocatorDefault, size, chan);
</span>
</span>let mut</span> subs: MaybeUninit<CFMutableDictionaryRef> = MaybeUninit::uninit();
</span>IOReportCreateSubscription(std::ptr::null(), chan, s.</span>as_mut_ptr</span>(), </span>0</span>, std::ptr::null());
</span>subs.</span>assume_init</span>();
</span>
</span>let</span> sample1 = IOReportCreateSamples(subs, chan, </span>null</span>());
</span>std::thread::sleep(std::time::Duration::from_millis(</span>100</span>)); </span>// 100ms
</span>let</span> sample2 = IOReportCreateSamples(subs, chan, </span>null</span>());
</span>
</span>let</span> rs = IOReportCreateSamplesDelta(sample1, sample2, </span>null</span>());
</span>CFShow(rs);
</span>
</span>// rs is CFDictionary, so we can read metrics by keys from it
</span></code></pre>
The basic dict functions are exported from core_foundation</a> crate. IOReport</code> functions are C-bindings to a private macOS API:</p>
#[</span>link</span>(name = "</span>IOReport</span>", kind = "</span>dylib</span>")]
</span>extern </span>"</span>C</span>" {
</span>  </span>fn </span>IOReportCopyChannelsInGroup</span>(</span>a</span>: CFStringRef, </span>b</span>: CFStringRef, </span>c</span>: </span>u64</span>, </span>d</span>: </span>u64</span>, </span>e</span>: </span>u64</span>) -> CFDictionaryRef;
</span>  </span>fn </span>IOReportMergeChannels</span>(</span>a</span>: CFDictionaryRef, </span>b</span>: CFDictionaryRef, </span>nil</span>: CFTypeRef);
</span>  </span>fn </span>IOReportCreateSubscription</span>(</span>a</span>: CVoidRef, </span>b</span>: CFMutableDictionaryRef, </span>c</span>: </span>*mut</span> CFMutableDictionaryRef, </span>d</span>: </span>u64</span>, </span>b</span>: CFTypeRef) -> IOReportSubscriptionRef;
</span>  </span>fn </span>IOReportCreateSamples</span>(</span>a</span>: IOReportSubscriptionRef, </span>b</span>: CFMutableDictionaryRef, </span>c</span>: CFTypeRef) -> CFDictionaryRef;
</span>  </span>fn </span>IOReportCreateSamplesDelta</span>(</span>a</span>: CFDictionaryRef, </span>b</span>: CFDictionaryRef, </span>c</span>: CFTypeRef) -> CFDictionaryRef;
</span>}
</span></code></pre>
Full implementation can be found here</a>.</p>
Note: For CPU I get values separately for each core. In general, there is also a group “CPU Complex Performance States” to get values for the whole cluster at once, but the group results sometimes show 100% load of the cluster, when in reality there is no load. This is caused by some internal bug. For GPU there is no information for each core separately, so the load information is obtained for the whole GPU at once — good luck no errors were noticed.</em></p>
Parsing CPU / GPU Usage</a></h2>
IOReportCreateSamplesDelta</code> returns an object with one IOReportChannels</code> field, which containts array of metrics objects. Each metrics object have group</code> and subgroup</code> (on which subscription was created), and extra fields: channel</code> name, value unit</code> and value</code> itself.</p>
The values for “CPU Stats” / “GPU Stats” are returned in array of tuples (string, int):</p>

string: frequency level name (first always IDLE)</li>
int: time been on this level (should be in nanoseconds, but not sure)</li>
</ul>
In general, internally CPU can work at different frequencies and voltages. The values of these frequencies are known in advance and are always the same. Under load CPU works at higher levels (higher frequency, higher voltage). During idle work it constantly jumps between modes (in any case OS always has some background activity). Simple tasks are mostly executed on E-cluster, more complex ones are switched to P-cluster.</p>
Each cluster has different DVFS config (Dynamic Voltage and Frequency Scaling), but all cores within same cluster uses same DVFS config. Same approach is used for GPUs.</p>
Usually GUI interfaces show single number as current core frequency, but in reality it is calculated as average of time each core work on DVFS level. So the average frequency of a core can be calculated like this:</p>
// predefined DVFS values for E-cluster
</span>let</span> frequencies = [</span>600</span>, </span>912</span>, </span>1284</span>, </span>1752</span>, </span>2004</span>, </span>2256</span>, </span>2424</span>]; </span>// 7 items
</span>// time was on each level (first value is idle)
</span>let</span> residencies = [</span>17563069</span>, </span>0</span>, </span>5787276</span>, </span>156965</span>, </span>419037</span>, </span>100647</span>, </span>106031</span>, </span>41735</span>]; </span>// 8 items
</span>
</span>let</span> total_time = residencies.</span>iter</span>().sum::<</span>u64</span>>();
</span>let</span> usage_time = residencies.</span>iter</span>().</span>skip</span>(</span>1</span>).sum::<</span>u64</span>>();
</span>
</span>let mut</span> freq = </span>0</span>f64</span>;
</span>for</span> i in </span>0</span>..frequencies.</span>len</span>() {
</span> </span>let</span> percent = residencies[i + </span>1</span>] as </span>f64 </span>/ usage_time as </span>f64</span>;
</span> freq += percent * frequencies[i] as </span>f64</span>;
</span>}
</span>
</span>println!("</span>E-cluster freq: </span>{:.2}</span> MHz</span>", freq);
</span></code></pre>
DVFS values for target system can be read from AppleARMIODevice</code>. Full code can be found here</a>. For M1/M2/M3 processors looks names are: voltage-states1-sram</code> for E-Cluster, voltage-states5-sram</code> for P-Cluster and voltage-states9</code> for GPU. For Max / Ultra chips is also exists voltage-states11-sram</code> or something like this, because this chips have two or more P-Clusters but fortunately this values are same for all clusters of same group (at least for now).</p>
Parsing Energy Usage</a></h2>
In the channel earlier, we also asked for metrics for the “Energy Model” group. These are easier to read because it’s just a number. Value unit can be mJ / uJ / nJ</code>, so it just needs to be converted to Watts. This is easily done using the formula: P(W) = E(J) / t(s)</code>, where s</code> is sampling time. Full code here</a>.</p>
let</span> joules_raw = IOReportSimpleGetIntegerValue(metric_dict, </span>0</span>) as </span>f32</span>;
</span>let</span> joules = </span>match</span> metric_unit.</span>as_str</span>() {
</span>  "</span>mJ</span>" => joules_raw / 1e3f32,
</span>  "</span>uJ</span>" => joules_raw / 1e6f32,
</span>  "</span>nJ</span>" => joules_raw / 1e9f32,
</span>  _ => Err(format!("</span>Unknown energy unit: </span>{}</span>", metric_unit).</span>into</span>()),
</span>};
</span>
</span>let</span> sec = (duration as </span>f32 </span>/ </span>1000.0</span>);
</span>let</span> watts = joules / sec;
</span></code></pre>
Getting RAM Usage</a></h2>
Getting RAM / SWAP usage is done via libc</code> in principle I think similar to any UNIX-like system. Nothing specific here, except to figure out how to calculate the value in bytes, since internally memory is measured in pages.</p>
MacOS uses the vm_statistics64</code></a> structure, which has quite a few fields. By experimentation I found a value that is similar to what Activity Monitor shows. To get current memory host_statistics64</code> need to be called. For total memory available and SWAP info sysconf</code> call is enough. Full code can be found here</a>. Short example with libc</a> crate:</p>
let mut</span> count: </span>u32 </span>= libc::</span>HOST_VM_INFO64_COUNT </span>as _;
</span>let mut</span> stats = std::mem::zeroed::<libc::vm_statistics64>();
</span>
</span>libc::host_statistics64(
</span>  libc::mach_host_self(),
</span>  libc::</span>HOST_VM_INFO64</span>,
</span>  &</span>mut</span> stats as </span>*mut </span>_ as </span>*mut </span>_, </span>// cast to void (untyped) pointer
</span>  &</span>mut</span> count,
</span>);
</span>
</span>let</span> page_size_kb = libc::sysconf(libc::</span>_SC_PAGESIZE</span>) as </span>u64</span>;
</span>let</span> usage = page_size_kb * (
</span>  + stats.active_count as </span>u64
</span>  + stats.inactive_count as </span>u64
</span>  + stats.wire_count as </span>u64
</span>  + stats.speculative_count as </span>u64
</span>  + stats.compressor_page_count as </span>u64
</span>  - stats.purgeable_count as </span>u64
</span>  - stats.external_page_count as </span>u64</span>);
</span></code></pre>
Getting Temperature Values</a></h2>
This appeared to be the most complicated part of the program. Initially I wanted to get values for each core separately and show them in a separate tab in the interface. On macOS there are GUI programs (Stats, Macs Fan Control) that can do this, but after a detailed study I realised that the values in these programs are shown by guessing which sensor is responsible for which core.</p>
Temperature value can be obtained from two places: IOHID for M1 on macOS 12–13 and SMC for M1/M2/M3 on macOS 14–15.</p>
IOHID</code> approach is simpler — it’s just a function call that returns a CFDictionary</code> with values. There seems only one person in open-source</a> who understands how it works, because all other repositories I found referenced it.</p>
SMC (System Management Controller) is more complicated. It works through something similar to RPC calls with large structures where fields are optionally filled in depending on which RPC method called. SMC looks like a more classic approach, but it looks like Apple didn’t have time to implement all the functionality during migration to ARM architecture. To make macmon</code> works everywhere, I had to implement both approaches.</p>
IOHID</code> implementation can be found here</a>, there is really nothing to talk about technically, except that the parameter values are found "magically".</p>
In SMC, it is possible to obtain a sensor’s value using a key. The key itself uses FourCC notation, which is essentially int32</code> but appears human-readable in code (for example, Tp01 -> 0x54703031</code>, Tp02 -> 0x54703032</code>, etc.). This key is used to obtain KeyInfo</code>, from which KeyData</code> can be retrieved. Therefore, to get the sensor value, two RPC calls are needed, which is inefficient and costly. Hence, it is necessary to cache the KeyInfo</code> value beforehand.</p>
The SMC initialisation process looks like: obtain all possible keys using the special key #KEY</code> (0x234b4559</code>), then filter out only those KeyInfo</code> that start with Tp</code> (CPU related) and Tg</code> (GPU related), and then make requests only to them – resulting in one RPC call per sensor.</p>
Since I do not know which sensors correspond to which core or other sensors on the chip, the program simply calculates average of all values per type.</p>
SMC implementation can be found here</a> (utility to perform RPC calls), here</a> (initialisation) and here</a> (actual metrics gathering).</p>
Combine all together</a></h2>
There’s nothing special about this part. Once per second, the current metric values are obtained from sources described above. The only thing worth noting is that SMC is used by default, if its value is not available, then IOHID is used (fallback for M1 on macOS 13). Then these values are combined into a single structure and return to caller.</p>
Then these values are aggregated in the "frontend" code and output to the console using ratatui</a> crate or in plain text in raw mode. But this is already a material for a separate article.</p>

You can try macmon</a> with Homebrew:</p>
brew</span> install vladkens/tap/macmon
</span></code></pre>


macmon – Mac Usage Monitor in Terminal
2024-08-06T00:00:00+00:00
</p>
When MacBook moved to ARM architecture (M1 processor), old system APIs stopped working and with them existing monitoring tools. However, macOS has a built-in console utility called powermetrics</code>. It can show real-time resource consumption, but it does it in plain text.</p>
asitop</code> is the first utility that allows users to view resource and energy consumption visually in a terminal. But it works on top of powermetrics</code> which requires sudo</code> to work. The program is written in Python and has compatibility issues with new M-processors.</p>
macmon</code> attempts to solve these issues. Coded in Rust as a single binary file it uses an undocumented Apple API (the same API that uses powermetrics, but doesn’t require sudo) to get metrics on system utilization, power consumption and temperature.</p>
Installation</a></h2>
brew</span> install vladkens/tap/macmon
</span></code></pre>
Features</a></h2>

No Sudo Required</em>: Run seamlessly without administrative privileges.</li>
Real-Time Metrics</em>: Monitor CPU/GPU/ANE power usage live.</li>
Detailed Utilisation</em>: Track CPU usage per cluster and RAM/Swap usage.</li>
Historical Data: View</em> historical charts with average and max values.</li>
Customizable Interface</em>: Switch between six color variants.</li>
Compact Design</em>: Fits neatly in a small terminal window.</li>
Built with Rust</em>: Ensuring performance and reliability.</li>
</ul>
Interface Overview</a></h2>
</p>
The top line shows the chip model, number of Effective (E) and Performance (P) cores, number of GPU cores, and RAM.</p>
The second line shows the chart of E and P cluster utilization of the CPU. MacOS uses E-cores for normal tasks, but when something complex needs to be done, process switches to P-cores.</p>
The third line shows memory usage and GPU loading (during gaming, video processing or LLM running 😈).</p>
And finally, the fourth line, the most interesting one, shows the power consumption for compute tasks, by system in general and separately for CPU, GPU and ANE (Apple Neural Engine). It also shows the average temperature of CPU and GPU.</p>

macmon</code> is an open-source program. You can view, star, request features or report bugs on Github</a>.</p>


REST API Deployment on AWS Lambda with Terraform (2023)
2023-10-12T00:00:00+00:00
In this article I will show the way how to deploy AWS Lambda using Terraform, with support for code updates in a single workflow.</p>
The problem with AWS Lambda is that at the time of its creation we need to have prepared code in zip archive or Docker image uploaded to ECR, unlike ECS / Kubernetes service, where we just specify the path to the repository, which can be empty at the creation stage. So to make an AWS Lambda deploy using Terraform we will need to prepare the code in the same Terraform recipe.</p>
In this article I will use the Docker image as source for AWS Lambda, as it is essentially the only one normal way to run any complex applications in Lambda that have external dependencies (libraries). There is a way to package the dependencies in a zip archive too, but if the architecture of the computer where the dependencies were built is different from the one where AWS Lambda runs, there will be problems. That’s why Docker is the most clear and reliable option.</p>
So this article will explore the creation of such resources:</p>

Simple HTTP server on Python with FastAPI</li>
Docker image to that application</li>
Creating ECR storage</li>
Build & push Docker image in Terraform file (null_resource)</li>
Creating AWS Lambda</li>
Creating access to Lambda from public url</li>
Updating Lambda on code change</li>
</ul>
Preparation</a></h2>
As usual, we start our work by creating a new project and declaring terraform dependencies:</p>
mkdir</span> lambda-demo; </span>cd</span> lambda-demo; </span>touch</span> main.tf
</span></code></pre>
Describe Terraform dependencies in main.tf</code> file</p>
terraform</span> {
</span>  required_providers {
</span>    </span>aws </span>= { </span>source </span>= "</span>hashicorp/aws</span>", </span>version </span>= "</span>5.17.0</span>" }
</span>  }
</span>}
</span>
</span>provider </span>"</span>aws</span>" {
</span>  </span>profile </span>= "</span>default</span>"
</span>  </span>region  </span>= "</span>us-east-1</span>"
</span>}
</span></code></pre>
Creating HTTP application with FastAPI</a></h2>
Let’s create subfolder for our code and create 3 files:</p>
mkdir</span> code; </span>touch</span> code/app.py code/requirements.txt code/Dockerfile
</span></code></pre>
Next, in the app.py</code> file add the application code:</p>
import </span>os
</span>import </span>uvicorn
</span>from </span>fastapi </span>import </span>FastAPI
</span>from </span>mangum </span>import </span>Mangum
</span>
</span>
</span>app = </span>FastAPI</span>()
</span>
</span>
</span>@app.</span>get</span>("</span>/</span>")
</span>def </span>index</span>():
</span>    </span>return </span>"</span>Hello, from AWS Lambda!</span>"
</span>
</span>
</span>handler = </span>Mangum</span>(app, </span>lifespan</span>="</span>off</span>")
</span>
</span>if </span>__name__ == "</span>__main__</span>":
</span>    uvicorn_app = </span>f</span>"{os.path.</span>basename</span>(__file__).</span>removesuffix</span>('</span>.py</span>')}</span>:app</span>"
</span>    uvicorn.</span>run</span>(uvicorn_app, </span>host</span>="</span>0.0.0.0</span>", </span>port</span>=</span>8000</span>, </span>reload</span>=</span>True</span>)
</span></code></pre>
This code runs the FastAPI</code> service, so we can write basically as much complex API as we want. Mangum</code> is a wrapper for the Lambda API, it does all the request processing logic in Lambda internally, allowing FastAPI</code> to work normally. The code in the __name__ == "__main__"</code> section is needed to run the application during development — we can write our API locally as usual.</p>
The following library should be added to requirements.txt</code>:</p>
fastapi==0.103.2
</span>mangum==0.17.0
</span>uvicorn==0.23.2
</span></code></pre>
And finally the Dockerfile</code> to build an image for our application:</p>
FROM</span> public.ecr.aws/lambda/python:3.11
</span>ENV </span>PYTHONUNBUFFERED=1
</span>
</span>COPY</span> requirements.txt ./
</span>RUN </span>pip3 install -r requirements.txt
</span>
</span>COPY</span> app.py ./
</span>CMD [ "</span>app.handler</span>" ]
</span></code></pre>
Creating ECR</a></h2>
We have the application ready, now we need to run it in AWS Lambda. First of all, we need to create a resource for ECR image. All the work below continues in the main.tf</code> file.</p>
# --- ECR ---
</span>
</span>resource </span>"</span>aws_ecr_repository</span>" "</span>api</span>" {
</span>  </span>name                 </span>= "</span>lambda-api</span>"
</span>  </span>image_tag_mutability </span>= "</span>MUTABLE</span>"
</span>  </span>force_delete         </span>= </span>true
</span>
</span>  image_scanning_configuration {
</span>    </span>scan_on_push </span>= </span>true
</span>  }
</span>}
</span></code></pre>
Build & Push Docker image</a></h2>
The next step is to build the image directly in the Terraform recipe. For this purpose null_resource</code> will be used.</p>
# --- Build & push image ---
</span>
</span>locals</span> {
</span>  </span>repo_url </span>= aws_ecr_repository.api.repository_url
</span>}
</span>
</span>resource </span>"</span>null_resource</span>" "</span>image</span>" {
</span>  </span>triggers </span>= {
</span>    </span>hash </span>= md5(join("</span>-</span>", [for x in fileset("", "</span>./code/{*.py,*.txt,Dockerfile}</span>") : filemd5(x)]))
</span>  }
</span>
</span>  </span>provisioner </span>"</span>local-exec</span>" {
</span>    </span>command </span>= <<EOF
</span>      aws ecr get-login-password | docker login --username AWS --password-stdin ${</span>local</span>.repo_url}
</span>      docker build --platform linux/amd64 -t ${</span>local</span>.repo_url}:latest ./code
</span>      docker push ${</span>local</span>.repo_url}:latest
</span>    EOF
</span>  }
</span>}
</span>
</span>data "aws_ecr_image" "latest" {
</span>  repository_name = aws_ecr_repository.api.name
</span>  image_tag       = "latest"
</span>  depends_on      = [null_resource.image]
</span>}
</span></code></pre>
The local-exec part is a classic bash script that logs into ECR, builds and flushes the image. We just wrote it in a Terraform file directly.</p>
The more interesting part is the triggers. It is needed to rebuild the image every time the files in the code folder are changed. The files are selected by mask, so if you need other extensions you should add them to the list.</p>
And the last block is to get information about the last loaded image in ECR. We will need this next to update AWS Lambda.</p>
Creating Lambda resource</a></h2>
As is typical in AWS, we need permissions to perform various actions. Lambda is no exception and needs a separate role as well. Let’s create a new role with access to Lambda functions and the ability to write logs.</p>
# --- IAM Role ---
</span>
</span>resource </span>"</span>aws_iam_role</span>" "</span>lambda</span>" {
</span>  </span>name </span>= "</span>lambda</span>"
</span>
</span>  </span>assume_role_policy </span>= jsonencode({
</span>    </span>Version </span>= "</span>2012-10-17</span>"
</span>    </span>Statement </span>= [{
</span>      </span>Effect    </span>= "</span>Allow</span>"
</span>      </span>Action    </span>= "</span>sts:AssumeRole</span>"
</span>      </span>Principal </span>= { </span>Service </span>= "</span>lambda.amazonaws.com</span>" }
</span>    }]
</span>  })
</span>}
</span>
</span>resource </span>"</span>aws_iam_role_policy_attachment</span>" "</span>lambda_logs</span>" {
</span>  </span>role       </span>= aws_iam_role.lambda.name
</span>  </span>policy_arn </span>= "</span>arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole</span>"
</span>}
</span></code></pre>
Next, we create a resource for the Lambda function:</p>
# --- Lambda ---
</span>
</span>resource </span>"</span>aws_cloudwatch_log_group</span>" "</span>api</span>" {
</span>  </span>name              </span>= "</span>/aws/lambda/api</span>"
</span>  </span>retention_in_days </span>= </span>14
</span>}
</span>
</span>resource </span>"</span>aws_lambda_function</span>" "</span>api</span>" {
</span>  </span>function_name    </span>= "</span>api</span>"
</span>  </span>role             </span>= aws_iam_role.lambda.arn
</span>  </span>image_uri        </span>= "</span>${aws_ecr_repository.api.repository_url}:latest</span>"
</span>  </span>package_type     </span>= "</span>Image</span>"
</span>  </span>source_code_hash </span>= trimprefix(</span>data</span>.aws_ecr_image.latest.id, "</span>sha256:</span>")
</span>  </span>timeout          </span>= </span>10
</span>
</span>  environment {
</span>    </span>variables </span>= {}
</span>  }
</span>
</span>  </span>depends_on </span>= [
</span>    null_resource.image,
</span>    aws_iam_role_policy_attachment.lambda_logs,
</span>    aws_cloudwatch_log_group.api,
</span>  ]
</span>}
</span></code></pre>
Lambda public endpoint</a></h2>
Finally we need to create public endpoint. CORS block required to access API from browser. In this article all values is wildcard, but you can change it as your requirements, like: allow_origins = ["https://my-domain.com"]</code></p>
# --- Lambda Endpoint ---
</span>
</span>resource </span>"</span>aws_lambda_function_url</span>" "</span>api</span>" {
</span>  </span>function_name      </span>= aws_lambda_function.api.function_name
</span>  </span>authorization_type </span>= "</span>NONE</span>"
</span>
</span>  cors {
</span>    </span>allow_credentials </span>= </span>true
</span>    </span>allow_origins     </span>= ["</span>*</span>"]
</span>    </span>allow_methods     </span>= ["</span>*</span>"]
</span>    </span>allow_headers     </span>= ["</span>date</span>", "</span>keep-alive</span>"]
</span>    </span>expose_headers    </span>= ["</span>keep-alive</span>", "</span>date</span>"]
</span>    </span>max_age           </span>= </span>86400
</span>  }
</span>}
</span>
</span>output </span>"</span>api_url</span>" {
</span>  </span>value </span>= aws_lambda_function_url.api.function_url
</span>}
</span></code></pre>
Deploying</a></h2>
Initialise terraform and apply the configuration:</p>
terraform</span> init
</span>terraform</span> validate && </span>terraform</span> apply
</span></code></pre>
Once this is complete, we will see a link to AWS Lambda:</p>
Apply</span> complete! Resources: 7 added, 0 changed, 0 destroyed.
</span>
</span>Outputs:
</span>
</span>api_url</span> = "</span>https://xxx.lambda-url.us-east-1.on.aws/</span>"
</span></code></pre>
Let’s try:</p>
https </span>$(</span>terraform</span> output</span> -raw</span> api_url)
</span>HTTP/1.1</span> 200 OK
</span>Connection:</span> keep-alive
</span>Content-Length:</span> 25
</span>Content-Type:</span> application/json
</span>Date:</span> Thu, 12 Oct 2023 13:19:40 GMT
</span>X-Amzn-Trace-Id:</span> root=1-6527e150-3c0aed5e335681cb7e120cde;</span>sampled</span>=</span>0;lineage=1da281dc:0
</span>x-amzn-Remapped-content-length:</span> 25
</span>x-amzn-RequestId:</span> a560d83b-40c4-4c27-a381-0bb630547a4a
</span>
</span>"</span>Hello, from AWS Lambda!</span>"
</span></code></pre>
It’s works!</p>
Updating code and deploy new version</a></h2>
Since we have prepared all the necessary invalidations above, we simply update the app.py</code> file and re-run terraform apply. I’ll just add v2 to the welcome text. Check it:</p>
</p>
New application version available on same URL!</p>
Note: https is binary from httpie</a>.</p>

Next, depending on requirements, we can add AWS Lambda to Route53 (so that there is a link to our domain), configure restriction policy. But these are topics for another article. Good luck with your AWS journey!</p>


AWS ECS Cluster on EC2 with Terraform (2023)
2023-09-15T00:00:00+00:00
This short guide will describe how to create an AWS ECS Cluster on EC2 in 2023. Basically there are already articles and code samples on the Internet, but some products in AWS are becoming obsolete (like Launch Configuration) and Terraform is undergoing API changes. So the purpose of this article is to show how to run ECS Cluster on EC2 with Terraform today.</p>
This article will look at how to create a Terraform configuration to provide such resources:</p>

VPC with public subnet</li>
Internet Gateway to connect to the global Internet</li>
Security groups for EC2 Node & ECS Service</li>
Auto-scaling group for ECS cluster with Launch Templates</li>
Publish image to ECR</li>
ECS cluster with task and service definition</li>
Load Balancer to public access & scale ECS Service</li>
</ul>
Preparation</a></h2>
To get started, you need to create a new Terraform project. Let’s create new directory and main.tf</code> file.</p>
mkdir</span> ecs-ec2-demo; </span>cd</span> ecs-ec2-demo; </span>touch</span> main.tf
</span></code></pre>
In the Terraform file, it is required to declare the necessary dependencies and initialise the project.</p>
terraform</span> {
</span>  required_providers {
</span>    </span>aws </span>= { </span>source </span>= "</span>hashicorp/aws</span>", </span>version </span>= "</span>5.17.0</span>" }
</span>  }
</span>}
</span>
</span>provider </span>"</span>aws</span>" {
</span>  </span>profile </span>= "</span>default</span>"
</span>  </span>region  </span>= "</span>us-east-1</span>"
</span>}
</span></code></pre>
Creating a VPC</a></h2>
Let’s make a new VPC & two public subnets.</p>
# --- VPC ---
</span>
</span>data </span>"</span>aws_availability_zones</span>" "</span>available</span>" { </span>state </span>= "</span>available</span>" }
</span>
</span>locals</span> {
</span>  </span>azs_count </span>= </span>2
</span>  </span>azs_names </span>= </span>data</span>.aws_availability_zones.available.names
</span>}
</span>
</span>resource </span>"</span>aws_vpc</span>" "</span>main</span>" {
</span>  </span>cidr_block           </span>= "</span>10.10.0.0/16</span>"
</span>  </span>enable_dns_hostnames </span>= </span>true
</span>  </span>enable_dns_support   </span>= </span>true
</span>  </span>tags                 </span>= { </span>Name </span>= "</span>demo-vpc</span>" }
</span>}
</span>
</span>resource </span>"</span>aws_subnet</span>" "</span>public</span>" {
</span>  </span>count                   </span>= </span>local</span>.azs_count
</span>  </span>vpc_id                  </span>= aws_vpc.main.id
</span>  </span>availability_zone       </span>= </span>local</span>.azs_names[</span>count</span>.index]
</span>  </span>cidr_block              </span>= cidrsubnet(aws_vpc.main.cidr_block, </span>8</span>, </span>10 </span>+ </span>count</span>.index)
</span>  </span>map_public_ip_on_launch </span>= </span>true
</span>  </span>tags                    </span>= { </span>Name </span>= "</span>demo-public-${</span>local</span>.azs_names[</span>count</span>.index]}</span>" }
</span>}
</span></code></pre>
Then create Internet Gateway:</p>
# --- Internet Gateway ---
</span>
</span>resource </span>"</span>aws_internet_gateway</span>" "</span>main</span>" {
</span>  </span>vpc_id </span>= aws_vpc.main.id
</span>  </span>tags   </span>= { </span>Name </span>= "</span>demo-igw</span>" }
</span>}
</span>
</span>resource </span>"</span>aws_eip</span>" "</span>main</span>" {
</span>  </span>count      </span>= </span>local</span>.azs_count
</span>  </span>depends_on </span>= [aws_internet_gateway.main]
</span>  </span>tags       </span>= { </span>Name </span>= "</span>demo-eip-${</span>local</span>.azs_names[</span>count</span>.index]}</span>" }
</span>}
</span></code></pre>
And after it public route table:</p>
# --- Public Route Table ---
</span>
</span>resource </span>"</span>aws_route_table</span>" "</span>public</span>" {
</span>  </span>vpc_id </span>= aws_vpc.main.id
</span>  </span>tags   </span>= { </span>Name </span>= "</span>demo-rt-public</span>" }
</span>
</span>  route {
</span>    </span>cidr_block </span>= "</span>0.0.0.0/0</span>"
</span>    </span>gateway_id </span>= aws_internet_gateway.main.id
</span>  }
</span>}
</span>
</span>resource </span>"</span>aws_route_table_association</span>" "</span>public</span>" {
</span>  </span>count          </span>= </span>local</span>.azs_count
</span>  </span>subnet_id      </span>= aws_subnet.public[</span>count</span>.index].id
</span>  </span>route_table_id </span>= aws_route_table.public.id
</span>}
</span></code></pre>
Now it is possible to run terraform apply</code> and a new VPC with two public subnets will appear in AWS. Each subnet will placed in different Availability Zone (AZ) and have access to the Internet via IGW.</p>
Note: Load Balancer requires at least two subnets created in different Availability Zones (AZ).</em></p>
Creating a scalable ECS Cluster</a></h2>
The next step is to create a complete ECS Cluster.</p>
# --- ECS Cluster ---
</span>
</span>resource </span>"</span>aws_ecs_cluster</span>" "</span>main</span>" {
</span>  </span>name </span>= "</span>demo-cluster</span>"
</span>}
</span></code></pre>
The code above will create an empty cluster, but in order for the cluster to run containers it needs EC2 instances on which it will run these containers.</p>
AWS can automatically create these EC2 instances and scaling them based on load, but for this we need to define Launch Template & Autoscaling Group. To implement this we need to create such resources:</p>

IAM Role for ECS Node</li>
Security Group (SG) for ECS Node</li>
Launch Template (describes EC2 instance)</li>
Autoscaling Group (AG)</li>
Capacity Provider to connect ECS Cluster & Autoscaling Group</li>
</ul>
IAM Role & Security Group for ECS EC2 Node</a></h2>
Let’s start from creating IAM Role for ECS Node:</p>
# --- ECS Node Role ---
</span>
</span>data </span>"</span>aws_iam_policy_document</span>" "</span>ecs_node_doc</span>" {
</span>  statement {
</span>    </span>actions </span>= ["</span>sts:AssumeRole</span>"]
</span>    </span>effect  </span>= "</span>Allow</span>"
</span>
</span>    principals {
</span>      </span>type        </span>= "</span>Service</span>"
</span>      </span>identifiers </span>= ["</span>ec2.amazonaws.com</span>"]
</span>    }
</span>  }
</span>}
</span>
</span>resource </span>"</span>aws_iam_role</span>" "</span>ecs_node_role</span>" {
</span>  </span>name_prefix        </span>= "</span>demo-ecs-node-role</span>"
</span>  </span>assume_role_policy </span>= </span>data</span>.aws_iam_policy_document.ecs_node_doc.json
</span>}
</span>
</span>resource </span>"</span>aws_iam_role_policy_attachment</span>" "</span>ecs_node_role_policy</span>" {
</span>  </span>role       </span>= aws_iam_role.ecs_node_role.name
</span>  </span>policy_arn </span>= "</span>arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role</span>"
</span>}
</span>
</span>resource </span>"</span>aws_iam_instance_profile</span>" "</span>ecs_node</span>" {
</span>  </span>name_prefix </span>= "</span>demo-ecs-node-profile</span>"
</span>  </span>path        </span>= "</span>/ecs/instance/</span>"
</span>  </span>role        </span>= aws_iam_role.ecs_node_role.name
</span>}
</span></code></pre>
Then create simple Security Group for ECS Node which allow outgoing traffic (its required to pull image to start service later)</p>
# --- ECS Node SG ---
</span>
</span>resource </span>"</span>aws_security_group</span>" "</span>ecs_node_sg</span>" {
</span>  </span>name_prefix </span>= "</span>demo-ecs-node-sg-</span>"
</span>  </span>vpc_id      </span>= aws_vpc.main.id
</span>
</span>  egress {
</span>    </span>from_port   </span>= </span>0
</span>    </span>to_port     </span>= </span>65535
</span>    </span>protocol    </span>= "</span>tcp</span>"
</span>    </span>cidr_blocks </span>= ["</span>0.0.0.0/0</span>"]
</span>  }
</span>}
</span></code></pre>
Launch Template</a></h2>
Next step is to create Launch Template for EC2 Instance.</p>
# --- ECS Launch Template ---
</span>
</span>data </span>"</span>aws_ssm_parameter</span>" "</span>ecs_node_ami</span>" {
</span>  </span>name </span>= "</span>/aws/service/ecs/optimized-ami/amazon-linux-2/recommended/image_id</span>"
</span>}
</span>
</span>resource </span>"</span>aws_launch_template</span>" "</span>ecs_ec2</span>" {
</span>  </span>name_prefix            </span>= "</span>demo-ecs-ec2-</span>"
</span>  </span>image_id               </span>= </span>data</span>.aws_ssm_parameter.ecs_node_ami.value
</span>  </span>instance_type          </span>= "</span>t2.micro</span>"
</span>  </span>vpc_security_group_ids </span>= [aws_security_group.ecs_node_sg.id]
</span>
</span>  iam_instance_profile { </span>arn </span>= aws_iam_instance_profile.ecs_node.arn }
</span>  monitoring { </span>enabled </span>= </span>true</span> }
</span>
</span>  </span>user_data </span>= base64encode(<<-EOF
</span>      #!/bin/bash
</span>      echo ECS_CLUSTER=${aws_ecs_cluster.main.name} >> /etc/ecs/ecs.config;
</span>    EOF
</span>  )
</span>}
</span></code></pre>
A couple of comments on the code above:</p>

ecs_node_ami</code> is the recommended image for ECS (but it possible to use another image by passing different ami</code>)</li>
In user_data</code> you is required to pass ECS cluster name, so AWS can register EC2 instance as node of ECS cluster</li>
</ol>
Autoscaling Group</a></h2>
Next step is to create autoscaling group (ASG) and connect it with Launch Template:</p>
# --- ECS ASG ---
</span>
</span>resource </span>"</span>aws_autoscaling_group</span>" "</span>ecs</span>" {
</span>  </span>name_prefix               </span>= "</span>demo-ecs-asg-</span>"
</span>  </span>vpc_zone_identifier       </span>= aws_subnet.public[*].id
</span>  </span>min_size                  </span>= </span>2
</span>  </span>max_size                  </span>= </span>8
</span>  </span>health_check_grace_period </span>= </span>0
</span>  </span>health_check_type         </span>= "</span>EC2</span>"
</span>  </span>protect_from_scale_in     </span>= </span>false
</span>
</span>  launch_template {
</span>    </span>id      </span>= aws_launch_template.ecs_ec2.id
</span>    </span>version </span>= "</span>$Latest</span>"
</span>  }
</span>
</span>  tag {
</span>    </span>key                 </span>= "</span>Name</span>"
</span>    </span>value               </span>= "</span>demo-ecs-cluster</span>"
</span>    </span>propagate_at_launch </span>= </span>true
</span>  }
</span>
</span>  tag {
</span>    </span>key                 </span>= "</span>AmazonECSManaged</span>"
</span>    </span>value               </span>= ""
</span>    </span>propagate_at_launch </span>= </span>true
</span>  }
</span>}
</span></code></pre>
This ASG will create from 2 to 8 instances of EC2 using Launch Template. Name</code> tag is optional and used here just to simplify to recognise instances connected to cluster. AmazonECSManaged</code> tag is required by AWS.</p>
At this point we can run terraform apply</code> again and make sure the ASG is working. AWS should create two EC2 instances. The can be seen at AWS Console > EC2 > Instances</em>. If you manually Terminate any of these instances, AWS will automatically create a new instance to match the configuration of ASG (in our case minimum 2).</p>
</p>
Capacity Provider</a></h2>
The last step is to connect the ECS Cluster to the ASG group so that the cluster can use EC2 instances to deploy containers.</p>
# --- ECS Capacity Provider ---
</span>
</span>resource </span>"</span>aws_ecs_capacity_provider</span>" "</span>main</span>" {
</span>  </span>name </span>= "</span>demo-ecs-ec2</span>"
</span>
</span>  auto_scaling_group_provider {
</span>    </span>auto_scaling_group_arn         </span>= aws_autoscaling_group.ecs.arn
</span>    </span>managed_termination_protection </span>= "</span>DISABLED</span>"
</span>
</span>    managed_scaling {
</span>      </span>maximum_scaling_step_size </span>= </span>2
</span>      </span>minimum_scaling_step_size </span>= </span>1
</span>      </span>status                    </span>= "</span>ENABLED</span>"
</span>      </span>target_capacity           </span>= </span>100
</span>    }
</span>  }
</span>}
</span>
</span>resource </span>"</span>aws_ecs_cluster_capacity_providers</span>" "</span>main</span>" {
</span>  </span>cluster_name       </span>= aws_ecs_cluster.main.name
</span>  </span>capacity_providers </span>= [aws_ecs_capacity_provider.main.name]
</span>
</span>  default_capacity_provider_strategy {
</span>    </span>capacity_provider </span>= aws_ecs_capacity_provider.main.name
</span>    </span>base              </span>= </span>1
</span>    </span>weight            </span>= </span>100
</span>  }
</span>}
</span></code></pre>
Run terraform apply</code> again and after it we should see Capacity Provider & EC2 instances connect to ECS cluster on AWS Console > ECS > demo-cluster > Infrastructure</em> tab.</p>
</p>
Creating ECS Service</a></h2>
The last step is to run the service on the ECS and configure heath check and autoscaling for it. Plan of Action:</p>

Create Elastic Container Registry (ECR) & push image</li>
Create IAM Role for ECS Task</li>
Create ECS Task Definition</li>
Create Security Group for service</li>
Create ECS Service</li>
Create Load Balancer</li>
</ul>
Elastic Container Registry (ECR)</a></h2>
The example will use helloword-http</a> server, which we will run in our private ECR.</p>
# --- ECR ---
</span>
</span>resource </span>"</span>aws_ecr_repository</span>" "</span>app</span>" {
</span>  </span>name                 </span>= "</span>demo-app</span>"
</span>  </span>image_tag_mutability </span>= "</span>MUTABLE</span>"
</span>  </span>force_delete         </span>= </span>true
</span>
</span>  image_scanning_configuration {
</span>    </span>scan_on_push </span>= </span>true
</span>  }
</span>}
</span>
</span>output </span>"</span>demo_app_repo_url</span>" {
</span>  </span>value </span>= aws_ecr_repository.app.repository_url
</span>}
</span></code></pre>
Let’s run terraform apply</code> again. We should see output with repository URL in AWS. Now push helloword-http to ECR.</p>
# Get AWS repo url from Terraform outputs
</span>export </span>REPO</span>=$</span>(</span>terraform</span> output</span> --raw</span> demo_app_repo_url)
</span># Login to AWS ECR
</span>aws</span> ecr get-login-password | </span>docker</span> login</span> --username</span> AWS</span> --password-stdin </span>$</span>REPO
</span>
</span># Pull docker image & push to our ECR
</span>docker</span> pull</span> --platform</span> linux/amd64 strm/helloworld-http:latest
</span>docker</span> tag strm/helloworld-http:latest $</span>REPO</span>:latest
</span>docker</span> push $</span>REPO</span>:latest
</span></code></pre>
After this image should be available in AWS Console > AWS ECR > demo-app</em> and it can be used as source for ECS Task Definition.</p>
Note: In case you are using ECS Nodes on ARM processors, use --platform linux/arm64</code></em></p>
IAM Role for ECS Task</a></h2>
Next required step is to defined IAM roles for ECS task. Roles required to have access ECR, Cloud Watch, etc.</p>
# --- ECS Task Role ---
</span>
</span>data </span>"</span>aws_iam_policy_document</span>" "</span>ecs_task_doc</span>" {
</span>  statement {
</span>    </span>actions </span>= ["</span>sts:AssumeRole</span>"]
</span>    </span>effect  </span>= "</span>Allow</span>"
</span>
</span>    principals {
</span>      </span>type        </span>= "</span>Service</span>"
</span>      </span>identifiers </span>= ["</span>ecs-tasks.amazonaws.com</span>"]
</span>    }
</span>  }
</span>}
</span>
</span>resource </span>"</span>aws_iam_role</span>" "</span>ecs_task_role</span>" {
</span>  </span>name_prefix        </span>= "</span>demo-ecs-task-role</span>"
</span>  </span>assume_role_policy </span>= </span>data</span>.aws_iam_policy_document.ecs_task_doc.json
</span>}
</span>
</span>resource </span>"</span>aws_iam_role</span>" "</span>ecs_exec_role</span>" {
</span>  </span>name_prefix        </span>= "</span>demo-ecs-exec-role</span>"
</span>  </span>assume_role_policy </span>= </span>data</span>.aws_iam_policy_document.ecs_task_doc.json
</span>}
</span>
</span>resource </span>"</span>aws_iam_role_policy_attachment</span>" "</span>ecs_exec_role_policy</span>" {
</span>  </span>role       </span>= aws_iam_role.ecs_exec_role.name
</span>  </span>policy_arn </span>= "</span>arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy</span>"
</span>}
</span></code></pre>
Cloud Watch Logs</a></h2>
Next small step is to create Cloud Watch logs group to be able to see container logs.</p>
# --- Cloud Watch Logs ---
</span>
</span>resource </span>"</span>aws_cloudwatch_log_group</span>" "</span>ecs</span>" {
</span>  </span>name              </span>= "</span>/ecs/demo</span>"
</span>  </span>retention_in_days </span>= </span>14
</span>}
</span></code></pre>
ECS Task Definition</a></h2>
At this point, we simply describe from where and how to launch the docker container.</p>
Note: Task Definition is created per AWS account, not per ECS Cluster. So the family name must be unique.</em></p>
# --- ECS Task Definition ---
</span>
</span>resource </span>"</span>aws_ecs_task_definition</span>" "</span>app</span>" {
</span>  </span>family             </span>= "</span>demo-app</span>"
</span>  </span>task_role_arn      </span>= aws_iam_role.ecs_task_role.arn
</span>  </span>execution_role_arn </span>= aws_iam_role.ecs_exec_role.arn
</span>  </span>network_mode       </span>= "</span>awsvpc</span>"
</span>  </span>cpu                </span>= </span>256
</span>  </span>memory             </span>= </span>256
</span>
</span>  </span>container_definitions </span>= jsonencode([{
</span>    </span>name         </span>= "</span>app</span>",
</span>    </span>image        </span>= "</span>${aws_ecr_repository.app.repository_url}:latest</span>",
</span>    </span>essential    </span>= </span>true</span>,
</span>    </span>portMappings </span>= [{ </span>containerPort </span>= </span>80</span>, </span>hostPort </span>= </span>80</span> }],
</span>
</span>    </span>environment </span>= [
</span>      { </span>name </span>= "</span>EXAMPLE</span>", </span>value </span>= "</span>example</span>" }
</span>    ]
</span>
</span>    </span>logConfiguration </span>= {
</span>      </span>logDriver </span>= "</span>awslogs</span>",
</span>      </span>options </span>= {
</span>        "</span>awslogs-region</span>"        = "</span>us-east-1</span>",
</span>        "</span>awslogs-group</span>"         = aws_cloudwatch_log_group.ecs.name,
</span>        "</span>awslogs-stream-prefix</span>" = "</span>app</span>"
</span>      }
</span>    },
</span>  }])
</span>}
</span></code></pre>
ECS Service</a></h2>
In order for the ECS service to be available within the cluster and have access to the Internet, a new Security Group must be created. ecs_svc</code> allows all incoming and outgoing traffic. The service itself will be inside the subnet, without public ip, so it will not be accessible from the Internet. SG allows access to the service only for VPC network members.</p>
# --- ECS Service ---
</span>
</span>resource </span>"</span>aws_security_group</span>" "</span>ecs_task</span>" {
</span>  </span>name_prefix </span>= "</span>ecs-task-sg-</span>"
</span>  </span>description </span>= "</span>Allow all traffic within the VPC</span>"
</span>  </span>vpc_id      </span>= aws_vpc.main.id
</span>
</span>  ingress {
</span>    </span>from_port   </span>= </span>0
</span>    </span>to_port     </span>= </span>0
</span>    </span>protocol    </span>= "</span>-1</span>"
</span>    </span>cidr_blocks </span>= [aws_vpc.main.cidr_block]
</span>  }
</span>
</span>  egress {
</span>    </span>from_port   </span>= </span>0
</span>    </span>to_port     </span>= </span>0
</span>    </span>protocol    </span>= "</span>-1</span>"
</span>    </span>cidr_blocks </span>= ["</span>0.0.0.0/0</span>"]
</span>  }
</span>}
</span>
</span>resource </span>"</span>aws_ecs_service</span>" "</span>app</span>" {
</span>  </span>name            </span>= "</span>app</span>"
</span>  </span>cluster         </span>= aws_ecs_cluster.main.id
</span>  </span>task_definition </span>= aws_ecs_task_definition.app.arn
</span>  </span>desired_count   </span>= </span>2
</span>
</span>  network_configuration {
</span>    </span>security_groups </span>= [aws_security_group.ecs_task.id]
</span>    </span>subnets         </span>= aws_subnet.public[*].id
</span>  }
</span>
</span>  capacity_provider_strategy {
</span>    </span>capacity_provider </span>= aws_ecs_capacity_provider.main.name
</span>    </span>base              </span>= </span>1
</span>    </span>weight            </span>= </span>100
</span>  }
</span>
</span>  ordered_placement_strategy {
</span>    </span>type  </span>= "</span>spread</span>"
</span>    </span>field </span>= "</span>attribute:ecs.availability-zone</span>"
</span>  }
</span>
</span>  lifecycle {
</span>    </span>ignore_changes </span>= [desired_count]
</span>  }
</span>}
</span></code></pre>
The ordered_placement_strategy</code> will try to make sure that each service instance is equally distributed across Availability Zones. You can use a different placement strategy, read the AWS documentation about it.</p>
If we run terraform apply</code> at this step it will create & run new service. But because this service has no Public IP, we can't access to it from public Internet. But service accessible from internal VPC. So if we create a Bastion Host we can ping it.</p>
</p>
To make the service available from the public network, as well as to be able to scale the service depending on the load we need to create Load Balancer.</p>
Load Balancer (ALB)</a></h2>
ALB also needs a Security Group. We will allow any incoming traffic on ports 80 and 443. Next, the ALB will listen on port 80 and forward the traffic to our ECS Service. This is the simplest ALB setup sufficient for this article.</p>
# --- ALB ---
</span>
</span>resource </span>"</span>aws_security_group</span>" "</span>http</span>" {
</span>  </span>name_prefix </span>= "</span>http-sg-</span>"
</span>  </span>description </span>= "</span>Allow all HTTP/HTTPS traffic from public</span>"
</span>  </span>vpc_id      </span>= aws_vpc.main.id
</span>
</span>  dynamic "</span>ingress</span>" {
</span>    </span>for_each </span>= [</span>80</span>, </span>443</span>]
</span>    content {
</span>      </span>protocol    </span>= "</span>tcp</span>"
</span>      </span>from_port   </span>= ingress.value
</span>      </span>to_port     </span>= ingress.value
</span>      </span>cidr_blocks </span>= ["</span>0.0.0.0/0</span>"]
</span>    }
</span>  }
</span>
</span>  egress {
</span>    </span>protocol    </span>= "</span>-1</span>"
</span>    </span>from_port   </span>= </span>0
</span>    </span>to_port     </span>= </span>0
</span>    </span>cidr_blocks </span>= ["</span>0.0.0.0/0</span>"]
</span>  }
</span>}
</span>
</span>resource </span>"</span>aws_lb</span>" "</span>main</span>" {
</span>  </span>name               </span>= "</span>demo-alb</span>"
</span>  </span>load_balancer_type </span>= "</span>application</span>"
</span>  </span>subnets            </span>= aws_subnet.public[*].id
</span>  </span>security_groups    </span>= [aws_security_group.http.id]
</span>}
</span>
</span>resource </span>"</span>aws_lb_target_group</span>" "</span>app</span>" {
</span>  </span>name_prefix </span>= "</span>app-</span>"
</span>  </span>vpc_id      </span>= aws_vpc.main.id
</span>  </span>protocol    </span>= "</span>HTTP</span>"
</span>  </span>port        </span>= </span>80
</span>  </span>target_type </span>= "</span>ip</span>"
</span>
</span>  health_check {
</span>    </span>enabled             </span>= </span>true
</span>    </span>path                </span>= "</span>/</span>"
</span>    </span>port                </span>= </span>80
</span>    </span>matcher             </span>= </span>200
</span>    </span>interval            </span>= </span>10
</span>    </span>timeout             </span>= </span>5
</span>    </span>healthy_threshold   </span>= </span>2
</span>    </span>unhealthy_threshold </span>= </span>3
</span>  }
</span>}
</span>
</span>resource </span>"</span>aws_lb_listener</span>" "</span>http</span>" {
</span>  </span>load_balancer_arn </span>= aws_lb.main.id
</span>  </span>port              </span>= </span>80
</span>  </span>protocol          </span>= "</span>HTTP</span>"
</span>
</span>  default_action {
</span>    </span>type             </span>= "</span>forward</span>"
</span>    </span>target_group_arn </span>= aws_lb_target_group.app.id
</span>  }
</span>}
</span>
</span>output </span>"</span>alb_url</span>" {
</span>  </span>value </span>= aws_lb.main.dns_name
</span>}
</span></code></pre>
The output alb_url</code> displays the ALB URL provided by AWS. We can ping this url to prove that ALB is working. If we try to ping alb_url</code> right now it will respond with 503 code — that expected because ALB Target Group is empty right now.</p>
Connect ECS Service to ALB</a></h2>
At the final stage, we need to update the aws_ecs_service</code> configuration by adding a section with load_balancer</code>.</p>
# --- ECS Service ---
</span>
</span>resource </span>"</span>aws_ecs_service</span>" "</span>app</span>" {
</span>  </span># ...
</span>
</span>  </span>depends_on </span>= [aws_lb_target_group.app]
</span>
</span>  load_balancer {
</span>    </span>target_group_arn </span>= aws_lb_target_group.app.arn
</span>    </span>container_name   </span>= "</span>app</span>"
</span>    </span>container_port   </span>= </span>80
</span>  }
</span>}
</span></code></pre>
Finally run terraform apply</code>. ECS Service will be started and we can check it on ALB URL provided by AWS:</p>
curl </span>$(</span>terraform</span> output</span> --raw</span> alb_url) </span># Hello from ip-10-10-10-XXX
</span>curl </span>$(</span>terraform</span> output</span> --raw</span> alb_url) </span># Hello from ip-10-10-11-YYY
</span></code></pre>
If we run CURL multiple times, it should write "Hello from HOSTNAME", the hostname will be different between calls, indicating that ALB is working and redirecting requests to different copies of the ECS Service.</p>
Bonus: ECS Service Auto Scaling</a></h2>
# --- ECS Service Auto Scaling ---
</span>
</span>resource </span>"</span>aws_appautoscaling_target</span>" "</span>ecs_target</span>" {
</span>  </span>service_namespace  </span>= "</span>ecs</span>"
</span>  </span>scalable_dimension </span>= "</span>ecs:service:DesiredCount</span>"
</span>  </span>resource_id        </span>= "</span>service/${aws_ecs_cluster.main.name}/${aws_ecs_service.app.name}</span>"
</span>  </span>min_capacity       </span>= </span>2
</span>  </span>max_capacity       </span>= </span>5
</span>}
</span>
</span>resource </span>"</span>aws_appautoscaling_policy</span>" "</span>ecs_target_cpu</span>" {
</span>  </span>name               </span>= "</span>application-scaling-policy-cpu</span>"
</span>  </span>policy_type        </span>= "</span>TargetTrackingScaling</span>"
</span>  </span>service_namespace  </span>= aws_appautoscaling_target.ecs_target.service_namespace
</span>  </span>resource_id        </span>= aws_appautoscaling_target.ecs_target.resource_id
</span>  </span>scalable_dimension </span>= aws_appautoscaling_target.ecs_target.scalable_dimension
</span>
</span>  target_tracking_scaling_policy_configuration {
</span>    predefined_metric_specification {
</span>      </span>predefined_metric_type </span>= "</span>ECSServiceAverageCPUUtilization</span>"
</span>    }
</span>
</span>    </span>target_value       </span>= </span>80
</span>    </span>scale_in_cooldown  </span>= </span>300
</span>    </span>scale_out_cooldown </span>= </span>300
</span>  }
</span>}
</span>
</span>resource </span>"</span>aws_appautoscaling_policy</span>" "</span>ecs_target_memory</span>" {
</span>  </span>name               </span>= "</span>application-scaling-policy-memory</span>"
</span>  </span>policy_type        </span>= "</span>TargetTrackingScaling</span>"
</span>  </span>resource_id        </span>= aws_appautoscaling_target.ecs_target.resource_id
</span>  </span>scalable_dimension </span>= aws_appautoscaling_target.ecs_target.scalable_dimension
</span>  </span>service_namespace  </span>= aws_appautoscaling_target.ecs_target.service_namespace
</span>
</span>  target_tracking_scaling_policy_configuration {
</span>    predefined_metric_specification {
</span>      </span>predefined_metric_type </span>= "</span>ECSServiceAverageMemoryUtilization</span>"
</span>    }
</span>
</span>    </span>target_value       </span>= </span>80
</span>    </span>scale_in_cooldown  </span>= </span>300
</span>    </span>scale_out_cooldown </span>= </span>300
</span>  }
</span>}
</span></code></pre>

Here is the end of ECS cluster configuration. Next it is possible to add a domain name, SSL certificate, https termination, routing traffic between different ECS services — but all this is stuff for other articles.</p>


How to still scrape millions of tweets with twscrape
2023-07-02T00:00:00+00:00
Twitter is a great place to gather data and assess various trends. Many analytics teams have used this source for their models.</p>
In February 2023, Twitter set unrealistic prices for its API, giving away crumbs of data for big bucks. Some started using libraries such as snscrape, which used web public APIs. But in April 2023, Twitter closed that option as well — making search only for authorized accounts.</p>
But data can still be collected in much the same way as before using the authorised account approach.</p>
Introduction to twscrape</a></h2>
Released in May 2023, it is a tool for scraping data from tweets. It collects data such as user profiles, follower lists and follower lists, likes and retweets, as well as keyword searches.</p>
Getting Started with twscrape</a></h2>
Requirements: Python 3.10 or higher</p>
Installing twscrape</a></h3>
pip</span> install twscrape
</span></code></pre>
Or development version with latest features:</p>
pip</span> install git+https://github.com/vladkens/twscrape.git
</span></code></pre>
Adding accounts</a></h3>
twscrape</code> needs Twitter accounts to work. Each account has a rather small limit on the use of APIs, after which some time is no way to make requests through that account. twscrape</code> is designed to switch accounts when one of them is not available. In this way the data flow looks continuous to the user, although in fact the requests come from different accounts internally.</p>
Accounts can be added in two ways, via software API or CLI command. Let’s use the CLI command:</p>
# twscrape add_accounts <file_path> <line_format>
</span># line_format should have "username", "password", "email", "email_password" tokens
</span># tokens delimeter should be same as an file
</span>twscrape</span> add_accounts accounts.txt username:password:email:email_password
</span></code></pre>
Note: It is possible to register a new account or buy on special websites, e.g. here</a>.</em></p>
You then have to go through the login procedure to get the tokens to request the API. It’s not a quick process, but it’s needed once after adding new accounts. Then the token is stored in the SQLite database and reused for subsequent queries.</p>
twscrape</span> login_accounts
</span></code></pre>
Note: Not all accounts can pass authorisation because of the antifraud system. You can try logging into these accounts again later.</em></p>
Using twscrape</a></h2>
You can use twscrape</code> in two ways.</p>

Using an CLI (terminal) and receive JSON object</li>
Using Python API (usuful for custom data collection scripts)</li>
</ol>
Lets get some tweet details from CLI:</a></h3>
twscrape</span> tweet_details 1674894268912087040
</span></code></pre>
Result:</p>
{
</span>  "</span>id</span>"</span>:</span> 1674894268912087000,
</span>  "</span>id_str</span>"</span>: </span>"</span>1674894268912087040</span>",
</span>  "</span>url</span>"</span>: </span>"</span>https://twitter.com/elonmusk/status/1674894268912087040</span>",
</span>  "</span>date</span>"</span>: </span>"</span>2023-06-30 21:34:46+00:00</span>",
</span>  "</span>user</span>"</span>: </span>{
</span>    "</span>id</span>": 44196397,
</span>    "</span>id_str</span>": "</span>44196397</span>",
</span>    "</span>url</span>": "</span>https://twitter.com/elonmusk</span>",
</span>    "</span>username</span>": "</span>elonmusk</span>",
</span>    "</span>displayname</span>": "</span>Elon Musk</span>",
</span>    "</span>created</span>": "</span>2009-06-02 20:12:29+00:00</span>",
</span>    // ...
</span>    "</span>_type</span>": "</span>snscrape.modules.twitter.User</span>"
</span>  },
</span>  "</span>lang</span>"</span>: </span>"</span>en</span>",
</span>  "</span>rawContent</span>"</span>: </span>"</span>This platform hit another all-time high in user-seconds last week</span>"
</span>  </span>//</span> ...
</span>}
</span></code></pre>
It’s that simple. The data format is almost the same as it was in snscrape. So if you already have some scripts to process the data, you can continue to use them without too much trouble.</p>
Scrapping tweets from a text search query by Python API</a></h3>
Using the code below, we are scraping 5000 tweets between January 1, 2023, and May 31, 2023, with the keywords "elon musk". Then printing in console tweet id, tweet author and content.</p>
import </span>asyncio
</span>from </span>twscrape </span>import </span>API</span>, gather
</span>from </span>twscrape.logger </span>import </span>set_log_level
</span>
</span>async def </span>main</span>():
</span>    api = </span>API</span>()
</span>
</span>    q = "</span>elon musk since:2023-01-01 until:2023-05-31</span>"
</span>    </span>async for </span>tweet </span>in </span>api.</span>search</span>(q, </span>limit</span>=</span>5000</span>):
</span>        </span>print</span>(tweet.id, tweet.user.username, tweet.rawContent)
</span>
</span>
</span>if </span>__name__ == "</span>__main__</span>":
</span>    asyncio.</span>run</span>(</span>main</span>())
</span></code></pre>
A general execution time for the entire code could be anywhere between 5 mins — 10 mins, depending on the number of tweets fetched by your username or keyword query.</p>
Working with raw API reponses</a></h3>
If you don’t have enough data provided by Tweet & User objects or want to get more insights from the data, then there is an option to use raw Twitter responses. Each method has a _raw version that returns the original data.</p>
import </span>asyncio
</span>from </span>twscrape </span>import </span>API</span>, gather
</span>from </span>twscrape.logger </span>import </span>set_log_level
</span>
</span>async def </span>main</span>():
</span>    api = </span>API</span>()
</span>
</span>    q = "</span>elon musk since:2023-01-01 until:2023-05-31</span>"
</span>    </span>async for </span>rep </span>in </span>api.</span>search_raw</span>(q, </span>limit</span>=</span>5000</span>):
</span>        </span># rep is httpx.Response object
</span>        </span>print</span>(rep.status_code, rep.</span>json</span>())
</span>
</span>if </span>__name__ == "</span>__main__</span>":
</span>    asyncio.</span>run</span>(</span>main</span>())
</span></code></pre>
Or same from CLI:</p>
twscrape</span> search "</span>elon musk since:2023-01-01 until:2023-05-31</span>"</span> --raw
</span></code></pre>
List of available functions</a></h3>

search</code> — just regular search by keywords</li>
tweet_details</code> — information about specific tweet</li>
retweeters</code> — list of users who retweet specific tweet</li>
favoriters</code> — list of users who like specific tweet</li>
user_by_login</code> — get user profile by login</li>
user_by_id</code> — get user profile by id</li>
user_tweets</code> — list of tweets of specific user (max 3200 tweets)</li>
user_tweets_and_replies</code> — list of tweets and replies of specific user</li>
followers</code> — list of followers of specific user</li>
following</code> — list of users the user is subscribed to</li>
list_timeline</code> — get all tweets of list</li>
</ul>
Found a bug or need new feature? Fill free to open an issues</a></p>

More examples of use can be found on the project's Github page</a>.</p>


Better forms handling with react-use-form
2023-04-04T00:00:00+00:00
React-use-form is a powerful library that simplifies form handling in React applications. Its intuitive API and robust feature set make it an excellent choice for developers.</p>
One of its standout features is the ability to work with custom components while minimizing re-renders of the main component, which is undeniably cool. This is achieved through the use of the Controller component, although its syntax may not be the most convenient.</p>
In this article, I’ll demonstrate how to create a generic wrapper that makes it easy to use your own components with the full power of Controller.</p>

Imagine we have a cool input component:</p>
import </span>{ </span>zodResolver </span>} </span>from </span>"</span>@hookform/resolvers/zod</span>";
</span>import </span>{ </span>FC</span>, </span>useId </span>} </span>from </span>"</span>react</span>";
</span>import </span>{ </span>Controller</span>, </span>useForm </span>} </span>from </span>"</span>react-hook-form</span>";
</span>import </span>{ </span>z </span>} </span>from </span>"</span>zod</span>";
</span>
</span>type </span>SuperInputProps = {
</span>  </span>label</span>: string;
</span>  </span>value</span>: string;
</span>  </span>onChange</span>: (</span>value</span>: string) </span>=> </span>void;
</span>  </span>error</span>?: string;
</span>};
</span>
</span>const </span>SuperInput</span>: FC<SuperInputProps> = (</span>props</span>) </span>=> </span>{
</span>  </span>const </span>{ </span>label</span>, </span>value</span>, </span>onChange</span>, </span>error </span>} = </span>props</span>;
</span>  </span>const </span>id </span>= </span>useId</span>();
</span>
</span>  </span>return </span>(
</span>    <</span>div</span>>
</span>      <</span>label </span>htmlFor</span>=</span>{</span>id</span>}</span>></span>{</span>label</span>}</span></</span>label</span>>
</span>      <</span>input </span>id</span>=</span>{</span>id</span>} </span>value</span>=</span>{</span>value</span>} </span>onChange</span>=</span>{</span>(</span>e</span>) </span>=> </span>onChange</span>(</span>e</span>.target.value)</span>} </span>/>
</span>      </span>{</span>error </span>&& <</span>span</span>></span>{</span>error</span>}</span></</span>span</span>></span>}
</span>    </</span>div</span>>
</span>  );
</span>};
</span>
</span>const </span>schema </span>= </span>z</span>.</span>object</span>({
</span>  firstName: </span>z</span>.</span>string</span>().</span>min</span>(</span>1</span>, { message: "</span>Required</span>" }),
</span>  lastName: </span>z</span>.</span>string</span>().</span>min</span>(</span>1</span>, { message: "</span>Required</span>" }),
</span>});
</span>
</span>const </span>SuperForm</span>: FC = () </span>=> </span>{
</span>  </span>const </span>{ </span>control </span>} = </span>useForm</span><z.infer<typeof schema>>({
</span>    resolver: </span>zodResolver</span>(</span>schema</span>),
</span>  });
</span>
</span>  </span>return </span>(
</span>    <</span>form</span>>
</span>      <</span>Controller
</span>        </span>control</span>=</span>{</span>control</span>}
</span>        </span>name</span>="</span>firstName</span>"
</span>        </span>render</span>=</span>{</span>({ </span>field</span>: { </span>onChange</span>, </span>value </span>}, </span>fieldState</span>: { </span>error </span>} }) </span>=> </span>(
</span>          <</span>SuperInput </span>label</span>="</span>First name</span>" </span>value</span>=</span>{</span>value</span>} </span>onChange</span>=</span>{</span>onChange</span>} </span>error</span>=</span>{</span>error</span>?.</span>message</span>} </span>/>
</span>        )</span>}
</span>      />
</span>      <</span>Controller
</span>        </span>control</span>=</span>{</span>control</span>}
</span>        </span>name</span>="</span>lastName</span>"
</span>        </span>render</span>=</span>{</span>({ </span>field</span>: { </span>onChange</span>, </span>value </span>}, </span>fieldState</span>: { </span>error </span>} }) </span>=> </span>(
</span>          <</span>SuperInput </span>label</span>="</span>Last name</span>" </span>value</span>=</span>{</span>value</span>} </span>onChange</span>=</span>{</span>onChange</span>} </span>error</span>=</span>{</span>error</span>?.</span>message</span>} </span>/>
</span>        )</span>}
</span>      />
</span>    </</span>form</span>>
</span>  );
</span>};
</span></code></pre>
The part about the Controller looks verbose. If you have a lot of forms in your application, it’s tedious to write it every time. Let’s make it better!</p>

To do this, we first need to write some TypeScript helpers to get a list of fields in dot notation for a particular data type.</p>
import </span>{ </span>Primitive </span>} </span>from </span>"</span>react-hook-form</span>";
</span>
</span>type </span>NestedImpl<K </span>extends </span>string | number, V, T> = V </span>extends </span>T
</span>  ? K
</span>  : V </span>extends </span>Primitive | Array<infer V>
</span>  ? never
</span>  : `</span>${</span>K</span>}.${</span>NestedByType</span><</span>V</span>, </span>T</span>></span>}</span>`;
</span>
</span>export type </span>NestedByType<O, T> = {
</span>  [K in keyof O]-?: K </span>extends </span>string ? NestedImpl<K, O[K], T> : never;
</span>}[keyof O];
</span>
</span>// test code
</span>
</span>type </span>A = { </span>abc</span>: string; </span>xyz</span>: number; </span>nested1</span>: { </span>abc</span>: number; </span>xyz</span>: string } };
</span>type </span>B = A & { </span>arr</span>: number[] };
</span>type </span>a1 = NestedByType<A, number>; </span>// "xyz" | "nested1.abc"
</span>type </span>a2 = NestedByType<A, string>; </span>// "abc" | "nested1.xyz"
</span>type </span>a3 = NestedByType<A, boolean>; </span>// never
</span>type </span>a4 = NestedByType<B, number[]>; </span>// "arr"
</span>type </span>a5 = NestedByType<B, string>; </span>// "abc" | "nested1.xyz"
</span></code></pre>
Now we need to make a small utility to comfortably retrieve errors from the form state for specific fields.</p>
const </span>getCtrlError </span>= (</span>formState</span>: { </span>errors</span>: Record<string, any> }, </span>name</span>: string) </span>=> </span>{
</span>  </span>const </span>tokens </span>= </span>name</span>.</span>split</span>("</span>.</span>");
</span>
</span>  </span>let </span>value </span>= </span>formState</span>.</span>errors</span>;
</span>  </span>for </span>(</span>const </span>token </span>of </span>tokens</span>) {
</span>    </span>if </span>(</span>value</span>[</span>token</span>]) </span>value </span>= </span>value</span>[</span>token</span>];
</span>  }
</span>
</span>  </span>return </span>value </span>&& </span>value</span>.</span>message </span>? (</span>value</span>.</span>message </span>as </span>string) : </span>undefined</span>;
</span>};
</span></code></pre>
Finally, we can make our Controlled wrapper for custom components.</p>
import </span>{ </span>FC </span>} </span>from </span>"</span>react</span>";
</span>import </span>{ </span>Control</span>, </span>Controller</span>, </span>FieldValues </span>} </span>from </span>"</span>react-hook-form</span>";
</span>
</span>const </span>makeCtrl </span>= <T </span>extends </span>Pick<T, "</span>value</span>" | "</span>onChange</span>"> & { </span>error</span>?: string }>(
</span>  </span>Component</span>: FC<T>,
</span>  </span>options</span>: { </span>errorKey</span>?: string } = {}
</span>) </span>=> </span>{
</span>  </span>// react-hook-form has FieldPathByValue but it slow, so we use NestedByType
</span>  </span>const </span>Wrapped </span>= <D </span>extends </span>FieldValues, P </span>extends </span>NestedByType<D, T["</span>value</span>"]>>(
</span>    </span>props</span>: Omit<T, "</span>value</span>" | "</span>onChange</span>" | "</span>error</span>"> & { </span>control</span>: Control<D>; </span>name</span>: P }
</span>  ) </span>=> </span>{
</span>    </span>const </span>{ </span>control</span>, </span>name</span>, ...</span>rest </span>} = </span>props</span>;
</span>    </span>const </span>errorKey </span>= </span>options</span>.</span>errorKey </span>? `</span>${</span>name</span>}.${</span>options</span>.</span>errorKey</span>}</span>` : </span>name</span>.</span>toString</span>();
</span>
</span>    </span>return </span>(
</span>      <</span>Controller
</span>        </span>control</span>=</span>{</span>control</span>}
</span>        </span>// @ts-expect-error because of comment above
</span>        </span>name</span>=</span>{</span>name</span>}
</span>        </span>render</span>=</span>{</span>({ </span>field</span>: { </span>value</span>, </span>onChange </span>}, </span>formState </span>}) </span>=> </span>{
</span>          </span>const </span>error </span>= </span>getCtrlError</span>(</span>formState</span>, </span>errorKey</span>.</span>toString</span>());
</span>          </span>const </span>props </span>= { ...</span>rest</span>, </span>value</span>, </span>onChange</span>, </span>error </span>} </span>as </span>unknown </span>as </span>T;
</span>          </span>return </span><</span>Component </span>{</span>...</span>props</span>} </span>/>;
</span>        }</span>}
</span>      />
</span>    );
</span>  };
</span>
</span>  </span>return </span>Wrapped</span>;
</span>};
</span></code></pre>
I assume that the component for which we do wrapper has properties: value, onChange, error. These properties will be removed from the resulting component and automatically inserted into child component. But if you want to “remove” other fields, you can add them to generic type of makeCtlr function, e.g.: makeCtrl<SomeType, "field_to_remove_1" | "field_to_remove_2">(...)</code>.</p>

And now it can be used somehow like this:</p>
const </span>SuperInputCtrl </span>= </span>makeCtrl</span>(</span>SuperInput</span>); </span>// create new component
</span>
</span>const </span>SuperForm</span>: FC = () </span>=> </span>{
</span>  </span>const </span>{ </span>control </span>} = </span>useForm</span><z.infer<typeof schema>>({
</span>    resolver: </span>zodResolver</span>(</span>schema</span>),
</span>  });
</span>
</span>  </span>return </span>(
</span>    <</span>form</span>>
</span>      <</span>SuperInputCtrl </span>control</span>=</span>{</span>control</span>} </span>name</span>="</span>firstName</span>" </span>label</span>="</span>First name</span>" />
</span>      <</span>SuperInputCtrl </span>control</span>=</span>{</span>control</span>} </span>name</span>="</span>lastName</span>" </span>label</span>="</span>Last name</span>" />
</span>    </</span>form</span>>
</span>  );
</span>};
</span></code></pre>
Now it looks much shorter, but still safe (the wrapper only allows fields that match the value type in the input). The validation error is automatically filled in. Only the input is re-rendered during data entry, not the entire form.</p>


Better date parsing for REST API with TypeScript
2022-12-15T00:00:00+00:00
A little trick in TypeScript to make it easier to work with date strings in the API. Let’s say we have a model like this on the client:</p>
type </span>User = {
</span>  </span>id</span>: number;
</span>  </span>username</span>: string;
</span>  </span>createdAt</span>: Date;
</span>};
</span></code></pre>
And we transfer this data in textual protocol in JSON format, e.g. via REST API or Websockets. And we get data in a format similar to this:</p>
GET /api/users/current
</span>{
</span>  "id": "1234567890",
</span>  "username": "johndoe",
</span>  "createdAt": "2020-01-01T00:00:00.000Z"
</span>}
</span></code></pre>
After that, you have to write data converters from the protocol model to the data model used in the application.</p>
type </span>UserDto = {
</span>  </span>id</span>: number;
</span>  </span>username</span>: string;
</span>  </span>createdAt</span>: string;
</span>};
</span>
</span>const </span>decodeUser </span>= (</span>dto</span>: UserDto): User </span>=> </span>{
</span>  </span>return </span>{
</span>    id: </span>dto</span>.id,
</span>    username: </span>dto</span>.</span>username</span>,
</span>    createdAt: new Date(</span>dto</span>.</span>createdAt</span>),
</span>  };
</span>};
</span>
</span>const </span>getCurrentUser </span>= </span>async </span>(): Promise<User> </span>=> </span>{
</span>  </span>const </span>user </span>= </span>await </span>get</span><UserDto>("</span>/api/users/current</span>");
</span>  </span>return </span>decodeUser</span>(</span>user</span>);
</span>};
</span>
</span>const </span>u </span>= </span>await </span>getCurrentUser</span>();
</span>console</span>.</span>log</span>(typeof </span>u</span>.</span>createdAt</span>, </span>u</span>.</span>createdAt </span>instanceof Date); </span>// string, false
</span></code></pre>
Basically, if the API only has a few methods, it’s not a problem to write decoders from DTO models. But usually the API is much bigger. And why should we do it if we can do not? :)</p>
In general, if we think about JSON format, it represents all basic types except date. So if we avoid the moment with string to date conversion, we won’t need any decodes at all. Let’s fix it.</p>
import </span>{ </span>parseISO </span>} </span>from </span>"</span>date-fns/esm</span>";
</span>
</span>const </span>ISODateFormat </span>= /</span>^</span>\d</span>{4}</span>-</span>\d</span>{2}</span>-</span>\d</span>{2}</span>T</span>\d</span>{2}</span>:</span>\d</span>{2}</span>:</span>\d</span>{2}</span>(?:\.</span>\d</span>*</span>)</span>?</span>(?:</span>[-+]\d</span>{2}</span>:</span>?</span>\d</span>{2}|</span>Z)</span>?</span>$</span>/;
</span>
</span>const </span>isIsoDateString </span>= (</span>value</span>: unknown): </span>value </span>is string </span>=> </span>{
</span>  </span>return </span>typeof </span>value </span>=== "</span>string</span>" && </span>ISODateFormat</span>.</span>test</span>(</span>value</span>);
</span>};
</span>
</span>const </span>handleDates </span>= (</span>data</span>: unknown) </span>=> </span>{
</span>  </span>if </span>(</span>isIsoDateString</span>(</span>data</span>)) </span>return </span>parseISO</span>(</span>data</span>);
</span>  </span>if </span>(</span>data </span>=== </span>null </span>|| </span>data </span>=== </span>undefined </span>|| typeof </span>data </span>!== "</span>object</span>") </span>return </span>data</span>;
</span>
</span>  </span>for </span>(</span>const </span>[</span>key</span>, </span>val</span>] of </span>Object</span>.</span>entries</span>(</span>data</span>)) {
</span>    </span>// @ts-expect-error this is a hack to make the type checker happy
</span>    </span>if </span>(</span>isIsoDateString</span>(</span>val</span>)) </span>data</span>[</span>key</span>] = </span>parseISO</span>(</span>val</span>);
</span>    </span>else if </span>(typeof </span>val </span>=== "</span>object</span>") </span>handleDates</span>(</span>val</span>);
</span>  }
</span>
</span>  </span>return </span>data</span>;
</span>};
</span></code></pre>
And now we can update our http client to automatically turn date strings into a native date object.</p>
const </span>http </span>= <T>(</span>url</span>: string, </span>config</span>?: RequestInit) </span>=> </span>{
</span>  </span>return </span>fetch</span>(</span>url</span>, </span>config</span>).</span>then</span>((</span>x</span>) </span>=> </span>handleDates</span>(</span>x</span>.</span>json</span>()) </span>as </span>Promise<T>);
</span>};
</span>
</span>const </span>get </span>= <T>(</span>url</span>: string) </span>=> </span>{
</span>  </span>return </span>http</span><T>(</span>url</span>);
</span>};
</span>
</span>const </span>getCurrentUser </span>= (): Promise<User> </span>=> </span>{
</span>  </span>return </span>http</span><User>("</span>/api/users/current</span>");
</span>};
</span>
</span>const </span>u </span>= </span>await </span>getCurrentUser</span>();
</span>console</span>.</span>log</span>(typeof </span>u</span>.</span>createdAt</span>, </span>u</span>.</span>createdAt </span>instanceof Date); </span>// object, true
</span>
</span>// or if you use axios you can make it event better
</span>
</span>axios</span>.</span>interceptors</span>.</span>response</span>.</span>use</span>((</span>rep</span>) </span>=> </span>{
</span>  </span>handleDates</span>(</span>rep</span>.data);
</span>  </span>return </span>rep</span>;
</span>});
</span>
</span>const </span>getCurrentUser </span>= </span>async </span>(): Promise<User> </span>=> </span>{
</span>  </span>const </span>rep </span>= </span>await </span>axios</span>.</span>get</span><User>("</span>/api/users/current</span>");
</span>  </span>return </span>rep</span>.data;
</span>};
</span></code></pre>


Docker Compose templating using YAML merge feature
2022-12-13T00:00:00+00:00
Useful trick when you have many of the same services in the docker-compose.yml</code> file.</p>
Imagine that we have some data processing application which has a master service and several different workers. All applications are written in the same language and use a common base docker container. In general this is a fairly common configuration. The docker-compose.yml</code> file will look something like this:</p>
version</span>: "</span>3.8</span>"
</span>
</span>services</span>:
</span>  </span>master</span>:
</span>    </span>build</span>:
</span>      </span>context</span>: </span>.
</span>    </span>ports</span>:
</span>      - "</span>8080:8080</span>"
</span>    </span>volumes</span>:
</span>      - "</span>.:/app</span>"
</span>    </span>command</span>: </span>python run_master.py
</span>
</span>  </span>worker-foo</span>:
</span>    </span>build</span>:
</span>      </span>context</span>: </span>.
</span>    </span>depends_on</span>:
</span>      - </span>master
</span>    </span>volumes</span>:
</span>      - "</span>.:/app</span>"
</span>    </span>environment</span>:
</span>      - </span>MASTER_HOST=master:8080
</span>    </span>command</span>: </span>python run_foo_worker.py
</span>
</span>  </span>worker-bar</span>:
</span>    </span>build</span>:
</span>      </span>context</span>: </span>.
</span>    </span>depends_on</span>:
</span>      - </span>master
</span>    </span>volumes</span>:
</span>      - "</span>.:/app</span>"
</span>    </span>environment</span>:
</span>      - </span>MASTER_HOST=master:8080
</span>    </span>command</span>: </span>python run_bar_worker.py
</span></code></pre>
It looks verbose. A lot of code duplication. Let’s fix it with YAML Anchors!</p>
version</span>: "</span>3.8</span>"
</span>
</span>x-base</span>: </span>&</span>base
</span>  </span>build</span>:
</span>    </span>context</span>: </span>.
</span>  </span>volumes</span>:
</span>    - </span>.:/app
</span>
</span>x-worker</span>: </span>&</span>worker
</span>  </span><<</span>: </span>*</span>base
</span>  </span>depends_on</span>:
</span>    - </span>master
</span>  </span>environment</span>:
</span>    - </span>MASTER_HOST=master:8080
</span>
</span>services</span>:
</span>  </span>master</span>:
</span>    </span><<</span>: </span>*</span>base
</span>    </span>ports</span>:
</span>      - "</span>8080:8080</span>"
</span>    </span>command</span>: </span>python run_master.py
</span>
</span>  </span>worker-foo</span>:
</span>    </span><<</span>: </span>*</span>worker
</span>    </span>command</span>: </span>python run_foo_worker.py
</span>
</span>  </span>worker-bar</span>:
</span>    </span><<</span>: </span>*</span>worker
</span>    </span>command</span>: </span>python run_bar_worker.py
</span></code></pre>
It looks much better! Such a file is easier to maintain, expand and edit in the future.</p>
The x-field is a special property in the docker-compose.yml file. In the Docker compose documentation they are called extension fields</a>. You can put any valid YAML code in them.</p>
Merge problems</a></h2>
Note that YAML does merge at the top level and do not deep merge. So this code will not work as expected:</p>
x-base</span>: </span>&</span>base
</span>  </span>environment</span>:
</span>    - </span>SUPER_SECRET=42
</span>
</span>x-worker</span>: </span>&</span>worker
</span>  </span><<</span>: </span>*</span>base
</span>  </span>environment</span>:
</span>    - </span>MASTER_HOST=master:8080
</span></code></pre>
The worker</code> service will completely overwrite environment of base and will contain only MASTER_HOST</code>. Unfortunately, this array combining cannot be achieved directly in the docker-compose file. Therefore, in such cases, you can use extends</a>.</p>
Solution for single variable</a></h2>
However, if you want to make only one value common, you can put that value into a variable, and use that variable as an array element.</p>
x-super-secret</span>: </span>&</span>super-secret </span>SUPER_SECRET=42
</span>
</span>x-base</span>: </span>&</span>base
</span>  </span>environment</span>:
</span>    - </span>*</span>super-secret
</span>
</span>x-worker</span>: </span>&</span>worker
</span>  </span><<</span>: </span>*</span>base
</span>  </span>environment</span>:
</span>    - </span>*</span>super-secret
</span>    - </span>MASTER_HOST=master:8080
</span></code></pre>
This can be useful for volumes</code>, depends_on</code> and other array-like directives.</p>
Solution for single object</a></h2>
As for environment, you can put the shared variables in a separate object and combine them in the same way we did with services. 🙃</p>
x-env</span>: </span>&</span>env
</span>  </span>SUPER_SECRET</span>: </span>42
</span>
</span>x-base</span>: </span>&</span>base
</span>  </span>environment</span>:
</span>    </span><<</span>: </span>*</span>env
</span>
</span>x-worker</span>: </span>&</span>worker
</span>  </span><<</span>: </span>*</span>base
</span>  </span>environment</span>:
</span>    </span><<</span>: </span>*</span>env
</span>    </span>MASTER_HOST</span>: </span>master:8080
</span></code></pre>

That’s all for now. If this information was helpful to you, don’t forget to subscribe to receive notifications of new posts.</p>


asdf — a good tool to manage python, node, etc runtime
2022-12-10T00:00:00+00:00
When you develop or maintain many projects or use a microservice architecture, over time you have a zoo of different programming languages and their versions on your computer. Each service has its own dependencies. After a while it becomes hellishly difficult to work comfortably.</p>
Initially, I controlled runtimes versions through homebrew, switching them when needed. Later I switched to nvm for NodeJS, pyenv for Python. If you add to this tools automatic version selection</a> on cd to project folder, it becomes almost comfortable. But this solution in the terminal is visually slow.</p>
I was looking for alternatives and found an amazing tool with a strange name – asdf</a>. This is single cli-tool for managing multiple runtimes and their versions. Each runtime is added via plugins and there are already about 450 of them</a>.</p>
asdf itself knows how to install different versions, can switch runtime when directory changed and works fast.</p>
Installation</a></h2>
On MacOS you can install asdf with homebrew (for other OS follow official guide</a>):</p>
brew</span> install asdf
</span></code></pre>
After that we can set the necessary runtimes:</p>
# Install NodeJS
</span>asdf</span> plugin-add nodejs
</span>asdf</span> install nodejs latest:16
</span>asdf</span> install nodejs latest:14
</span>
</span># Install Python
</span>asdf</span> plugin-add python
</span>asdf</span> install python latest:3.10
</span>asdf</span> install python latest:3.11
</span>
</span># Also possible to get list of available versions for inslation
</span>asdf</span> list all nodejs
</span>asdf</span> list all python
</span>
</span># Or list of already installed versions
</span>asdf</span> list nodejs
</span>asdf</span> list python
</span>
</span># Get current selected runtimes
</span>asdf</span> current
</span>
</span># Set global runtime (will create a .tool-versions in root of user directory)
</span>asdf</span> global nodejs latest:16
</span>asdf</span> global python latest:3.10
</span>
</span># Or set local runtime (will create a .tool-versions file in current directory)
</span>asdf</span> local nodejs latest:14
</span>asdf</span> local python latest:3.10
</span></code></pre>
Version select on directory change</a></h2>
asdf</code> can also automatically select the correct version depending on the current directory and will do this on each cd</code>. It does this by going up in the hierarchy until it finds version of required runtime in the .tool-versions</code> file.</p>
This allows flexible control over multiple versions of runtimes in each folder. For example you have an old project where tooling is written in python 3.8, you want to start code a new microservice in python 3.11, and globally in the system you are using version 3.9. asdf</code> will handle it for you!</p>
To enable this auto version select feature add next line to your shell rc-file (for more options or other shells see official guide</a>):</p>
# For zsh (~/.zhsrc)
</span>. </span>$(</span>brew --prefix</span> asdf)/libexec/asdf.sh
</span>
</span># For fish-shell (~/.config/fish/config.fish)
</span>source </span>"</span>(brew --prefix asdf)</span>"/libexec/asdf.fish
</span></code></pre>
That’s all. Now just make a cd to the project folder and set the required runtime version. Now when you navigate to the project folder, asdf will automatically select python of this version. You can check selected versions with:</p>
# for example install global python version
</span>asdf</span> global python latest:3.11
</span>
</span># go to project folder
</span>cd </span>~</span>/some_project
</span>
</span># install reqired python version (will create or update .tool-versions file)
</span>asdf</span> local python latest:3.10
</span>
</span># see current active runtime versions (should be 3.10)
</span>asdf</span> current
</span>
</span># check python version (should be same)
</span>python --version
</span>
</span># go back to user folder
</span>cd </span>~
</span>
</span># check python version again (should be 3.11)
</span>asdf</span> current
</span>python --version
</span></code></pre>
Also you can commit the .tool-versions</code> file directly into GIT. So then all team members will also know what versions of runtimes is in use for that service.</p>
One more life story</a></h2>
Using asdf you can manage not only runtime language versions, but also cli tools such as aws-cli, terraform, kubectl.</p>
So if you have different versions of Kubernetes clusters, even the official documentation does not recommend using more than one version of difference.</p>
Using multiple versions via brew is not convenient (you have to switch versions manually all the time). But these cli tools can also be put in .tool-versions</code>.</p>
asdf</span> plugin-add kubectl https://github.com/asdf-community/asdf-kubectl.git
</span>
</span>asdf</span> install kubectl latest:1.20
</span>asdf</span> install kubectl latest:1.25
</span>
</span>asdf</span> global kubectl latest:1.25
</span>
</span>cd </span>~</span>/legacy_project
</span>asdf</span> local kubectl latest:1.20
</span></code></pre>

That’s all for now. If this information was helpful to you, don’t forget to subscribe to receive notifications of new posts.</p>


How to see env in Docker container?
2022-12-06T00:00:00+00:00
TL;DR</a></h2>
# get all
</span>cat</span> /proc/1/environ | </span>tr </span>'</span>\0</span>' '</span>\n</span>'
</span>
</span># get by name
</span>cat</span> /proc/1/environ | </span>tr </span>'</span>\0</span>' '</span>\n</span>' | </span>grep</span> ABC
</span></code></pre>
Details</a></h2>
When you develop services, sometimes you need to know what environment variables a process in a container is running with.</p>
For example if you use kubernetes and the environment variables come from several places or if you debug a production container and want to know what environment it is running with.</p>
Well, that's easy. Normally your process in the docker is running under PID=1</code>. You can check this by running the command:</p>
docker</span> exec <CONTAINER_ID> ps aux
</span># output:
</span>PID</span>   USER     TIME  COMMAND
</span>    </span>1</span> root      0:02 node src/main.js
</span></code></pre>
Now we can see the environment variables that this process uses:</p>
docker</span> exec <CONTAINER_ID> cat /proc/1/environ | </span>tr </span>'</span>\0</span>' '</span>\n</span>'
</span># output:
</span>NODE_VERSION</span>=</span>16.18.1
</span>HOSTNAME</span>=</span>544193567d05
</span>YARN_VERSION</span>=</span>1.22.19
</span>SHLVL</span>=</span>2
</span>HOME</span>=</span>/root
</span>PATH</span>=</span>/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
</span>PWD</span>=</span>/app
</span></code></pre>
This approach is also useful if you need to get environment variables of kubernetes pod:</p>
# connect to pod
</span>kubectl</span> exec</span> --stdin --tty</span> node-app-abc123 -- /bin/sh
</span>
</span># get process envs
</span>cat</span> /proc/1/environ | </span>tr </span>'</span>\0</span>' '</span>\n</span>'
</span># output: list of environment variables
</span></code></pre>


Keep Type Guards valid after refactoring in TypeScript
2022-12-03T00:00:00+00:00
TypeScript is a great language for writing applications with type safety checks. Extending or refactoring code in TypeScript is much easier than in plain JavaScript.</p>
TypeScript has a nice built-in functionality for interface narrowing — Type Guards. But they don’t always protect against errors during code extension / refactoring, especially if you have a large project.</p>
Problem description</a></h2>
For example, we have a tea store. We sell two kinds of tea: loose and bagged. We show all the goods in one list. Each type of product has humanised link, and in the name of the product we want to specify the number of grams or the number of tea bags in a pack.</p>
import </span>React </span>from </span>"</span>react</span>";
</span>
</span>type </span>Tea = { </span>id</span>: number; </span>name</span>: string; </span>price</span>: number };
</span>
</span>type </span>LooseTea = Tea & { </span>weight</span>: number };
</span>type </span>BaggedTea = Tea & { </span>size</span>: number };
</span>
</span>const </span>isLooseTea </span>= (</span>x</span>: Tea): </span>x </span>is LooseTea </span>=> </span>"</span>weight</span>" in </span>x</span>;
</span>const </span>isBaggedTea </span>= (</span>x</span>: Tea): </span>x </span>is BaggedTea </span>=> </span>"</span>size</span>" in </span>x</span>;
</span>
</span>const </span>getLink </span>= (</span>x</span>: Tea): string </span>=> </span>{
</span>  </span>if </span>(</span>isLooseTea</span>(</span>x</span>)) </span>return </span>`</span>/loose-tea/${</span>x</span>.id}</span>`;
</span>  </span>if </span>(</span>isBaggedTea</span>(</span>x</span>)) </span>return </span>`</span>/bagged-tea/${</span>x</span>.id}</span>`;
</span>  </span>throw </span>new Error("</span>Unknown tea</span>");
</span>};
</span>
</span>const </span>getTitle </span>= (</span>x</span>: Tea): string </span>=> </span>{
</span>  </span>if </span>(</span>isLooseTea</span>(</span>x</span>)) </span>return </span>`</span>${</span>x</span>.name} / ${</span>x</span>.</span>weight</span>}g</span>`;
</span>  </span>if </span>(</span>isBaggedTea</span>(</span>x</span>)) </span>return </span>`</span>${</span>x</span>.name} / ${</span>x</span>.size} teabags</span>`;
</span>  </span>throw </span>new Error("</span>Unknown tea</span>");
</span>};
</span>
</span>const </span>TeaItems</span>: React.FC<{ </span>items</span>: Tea[] }> = ({ </span>items </span>}) </span>=> </span>{
</span>  </span>return </span>(
</span>    <</span>ul</span>>
</span>      </span>{</span>items</span>.</span>map</span>((</span>x</span>) </span>=> </span>(
</span>        <</span>li </span>key</span>=</span>{</span>x</span>.id</span>}</span>>
</span>          <</span>a </span>href</span>=</span>{</span>getLink</span>(</span>x</span>)</span>}</span>></span>{</span>getTitle</span>(</span>x</span>)</span>}</span></</span>a</span>>
</span>        </</span>li</span>>
</span>      ))</span>}
</span>    </</span>ul</span>>
</span>  );
</span>};
</span></code></pre>
Later, we got large tea bags (for the teapot) and changed our data model:</p>
type </span>BaggedTea = Tea & { </span>bags</span>: number; </span>weightPerBag</span>: number };
</span></code></pre>
After that we get an error in getTitle</code> function and fix it:</p>
// before:
</span>const </span>getTitle </span>= (</span>x</span>: Tea): string </span>=> </span>{
</span>  </span>if </span>(</span>isLooseTea</span>(</span>x</span>)) </span>return </span>`</span>${</span>x</span>.name} / ${</span>x</span>.</span>weight</span>}g</span>`;
</span>  </span>// Property 'size' does not exist on type 'BaggedTea'. ts(2339)
</span>  </span>if </span>(</span>isBaggedTea</span>(</span>x</span>)) </span>return </span>`</span>${</span>x</span>.name} / ${</span>x</span>.size} teabags</span>`;
</span>  </span>throw </span>new Error("</span>Unknown tea</span>");
</span>};
</span>
</span>// after:
</span>const </span>getTitle </span>= (</span>x</span>: Tea): string </span>=> </span>{
</span>  </span>if </span>(</span>isLooseTea</span>(</span>x</span>)) </span>return </span>`</span>${</span>x</span>.name} / ${</span>x</span>.</span>weight</span>}g</span>`;
</span>  </span>if </span>(</span>isBaggedTea</span>(</span>x</span>)) </span>return </span>`</span>${</span>x</span>.name} / ${</span>x</span>.</span>bags</span>} teabags ~ ${</span>x</span>.</span>weightPerBag</span>}</span>`;
</span>  </span>throw </span>new Error("</span>Unknown tea</span>");
</span>};
</span></code></pre>
Everything seems to be fine and TypeScript doesn’t found any error. But in fact during the rendering process an "Unknown tea" error will occur, because the Type Guard in the getLink</code> function is no longer pass in isBaggedTea</code> guard.</p>
Problem solution</a></h2>
The problem with our Type Guard is that internally we check for the presence of a field in the object. This is a JavaScript operation and it is not typed in any way.</p>
Well, let’s handle that. I propose to use a factory function to create Type Guard functions based on the data-model fields.</p>
type </span>AnyObject = Record<string, any>;
</span>
</span>// https://stackoverflow.com/a/52991061
</span>type </span>RequiredKeys<T> = {
</span>  [K in keyof T]-?: AnyObject </span>extends </span>Pick<T, K> ? never : K;
</span>}[keyof T];
</span>
</span>const </span>createShapeGuard </span>= <T </span>extends </span>AnyObject>(...</span>keys</span>: RequiredKeys<T>[]) </span>=> </span>{
</span>  </span>return </span>(</span>obj</span>: unknown): </span>obj </span>is T </span>=> </span>{
</span>    </span>if </span>(typeof </span>obj </span>!== "</span>object</span>" || </span>obj </span>=== </span>null</span>) </span>return </span>false</span>;
</span>
</span>    </span>for </span>(</span>const </span>key </span>of </span>keys </span>as </span>string[]) {
</span>      </span>if </span>(!(</span>key </span>in </span>obj</span>)) </span>return </span>false</span>;
</span>    }
</span>
</span>    </span>return </span>true</span>;
</span>  };
</span>};
</span></code></pre>
Now let’s rewrite our Type Guards in a new way:</p>
const </span>isLooseTea </span>= </span>createShapeGuard</span><LooseTea>("</span>weight</span>");
</span>const </span>isBaggedTea </span>= </span>createShapeGuard</span><BaggedTea>("</span>bags</span>", "</span>weightPerBag</span>");
</span></code></pre>
Everything works fine now. If we change the data structure again in the future, TypeScript will check and tell us that there is now an error in our Type Guard:</p>
type </span>BaggedTea = Tea & { </span>bags</span>: number; </span>bagWeight</span>: number };
</span>
</span>// Argument of type '"weightPerBag"' is not assignable to parameter of type 'RequiredKeys<BaggedTea>'.ts(2345)
</span>const </span>isBaggedTea </span>= </span>createShapeGuard</span><BaggedTea>("</span>bags</span>", "</span>weightPerBag</span>");
</span></code></pre>

That’s all for now. If this information was helpful to you, don’t forget to subscribe to receive notifications of new posts.</p>


Sync all Git branches with remote
2022-12-02T00:00:00+00:00
Sometimes there is no good internet connection, but you can still work at your laptop (merge PR, etc). It’s very handy to have a local full local copy of the Git repository of a project on your computer for this purpose.</p>
Unfortunately, Git itself doesn’t have out-of-the-box functionality to sync your local repository with a central server (like Github).</p>
I searched for a solution on stackoverflow, but surprisingly there are only separate commands to do different parts of this task.</p>
So I made this little script in bash to be able to do this synchronisation.</p>
#!/usr/bin/env bash
</span>
</span># Debug: Print all commands with expanded variables
</span># set -x
</span>
</span># Get all remote refs & remove outdated local refs
</span>git</span> fetch</span> --all --prune
</span>
</span># Create local copy of remote branch & load latest changes
</span>for</span> rb </span>in </span>`</span>git</span> branch</span> --remotes </span>| </span>grep -v </span>'</span>HEAD</span>'`; </span>do
</span>  </span>lb</span>=$</span>{</span>rb</span>#</span>origin/}
</span>  </span>echo </span>"$</span>rb</span> -> </span>$</span>lb</span>"
</span>  </span>git</span> branch</span> --track </span>$</span>lb </span>$</span>rb </span>2</span>>/dev/null
</span>  </span>git</span> fetch</span> --update-head-ok</span> origin $</span>lb</span>:$</span>lb
</span>done
</span>
</span># Remove all merged local branches
</span>git</span> branch</span> --merged </span>| </span>xargs</span> echo | </span>grep -v </span>'</span>^*\|^main$\|^develop$|^stage$</span>' | </span>xargs</span> git branch</span> -d
</span></code></pre>
Note: On line 18 you can specify a list of branches that should always remain local.</em></p>
This script will also be useful if you often have to check other branches manually before merging them.</p>
Installation & Run:</p>
curl</span> https://gist.githubusercontent.com/vladkens/5261c703111fd1d0a870c3bf5e53c698/raw/698af0fdac327bcb3b7d8777a4593a4d66321bb6/git-sync.sh > git-sync.sh
</span>chmod</span> +x git-sync.sh
</span>
</span>./git-sync.sh
</span></code></pre>


Docker Desktop alternative for MacOS
2022-07-22T00:00:00+00:00
Starting in February 2022, Docker Desktop became a paid service for large teams. This may be a reason to look for alternatives to Docker Desktop.</p>
In my case one day Docker Desktop just stopped opening. The default solution of uninstalling and re-installing didn’t work. I didn’t have enough time to figure out the problem. So I looked for alternatives to running containers on macOS locally.</p>
One of the projects I looked at is Colima</a>. I found it minimalistic to install and easy to use. Colima supports M1 & Intel chips, port forwarding, volume mounts, docker-compose & kubernetes. Basically everything you need for local development.</p>
How to install Colima?</a></h2>
Install docker & docker-compose client (just clients to interact with Colima containers runtime):</p>
brew</span> install docker
</span>brew</span> install docker-compose
</span></code></pre>
Then install Colima:</p>
brew</span> install colima
</span></code></pre>
Usage</a></h2>
Start Colima is easy:</p>
colima</span> start
</span></code></pre>
That’s it. Now you can run your docker-compose files and everything works. To stop Colima:</p>
colima</span> stop
</span></code></pre>
Colima also support different profiles, kubernetes, etc. You can check start options in docs</a> or by:</p>
colima</span> start</span> --help
</span></code></pre>
Customising</a></h2>
By default Colima runs with 2 CPUs, 2GiB memory and 60GiB storage. This is not enough for me so these settings can be changed:</p>
# stop VM before apply changes
</span>colima</span> stop
</span># this will change configuration of exits VM
</span>colima</span> start</span> --cpu</span> 2</span> --memory</span> 4
</span># or by changing config file
</span>colima</span> start</span> --edit
</span></code></pre>
Possible problems</a></h2>
If you are using utilities such as dry</a> or dive</a>, they may not start with the error like “docker.sock not found”. This is easy to fix:</p>
sudo</span> ln</span> -sf ~</span>/.colima/docker.sock /var/run/docker.sock
</span></code></pre>

That’s all for now. I hope that article was useful for you. If yes, don’t forget to follow me to get new updates.</p>


How to reduce NodeJS Docker Image Size?
2022-01-16T00:00:00+00:00
Hi. Today I want to talk about how to reduce the size of the Docker Image with NodeJS inside.</p>
In my company project we use NodeJS as a backend of one of the services. I noticed that there is a big delay between when the project is built in CI and when the new version of the service actually starts working.</p>
I started researching this topic and noticed that the NodeJS Docker Image size takes up 1.3Gb of space. So on each deploy these data first uploaded to registry and then downloaded back to run in Kubernetes cluster.</p>
To check the size of your Docker need to run few commands:</p>
> docker </span>build -t</span> app .
</span>> docker </span>images </span>| </span>grep</span> app
</span>app</span>     latest    62dec8181ae0   30 seconds ago   1.28GB
</span></code></pre>
Well, let’s fix that problem.</p>
Smaller Docker Base Image</a></h2>
The most basic and simple solution would be to change the basic Docker Image to the alpine version. To do this, change in the first line of the Dockerfile FROM node:16</code> to FROM node:16-alpine</code>. This was already in my Dockerfile, so this optimization won’t help me. The original Docker file looks like:</p>
FROM node:16-alpine
</span>
</span>WORKDIR /app
</span>COPY . .
</span>
</span>RUN yarn install
</span>RUN yarn app:build
</span>
</span>EXPOSE 3000
</span>CMD node dist/app.js
</span></code></pre>
Multi-Stage Builds</a></h2>
The process of creating a Docker container can be divided into several steps, where different parts of the application are prepared individually. And then the necessary files from the intermediate stages can be copied into the final image.</p>
This makes sense, because during the build a lot of temporary files are created, there are dev dependencies in package.json and other things that are not really needed for the final application to work.</p>
I divided the build process of container into three steps. The first step builds the application (compiling the TypeScript, validation schemes, etc.). The second stage installs only those dependencies that are needed for the production of the application. Finally, the third stage copies the compiled application from the first stage and the dependencies from the second stage and runs the application.</p>
This approach requires a little more time to build the container, but it also reduces its size considerably. For example, the TypeScript dependency alone takes up over 80 mb of space. The final Dockerfile looks like:</p>
FROM node:16-alpine as dist
</span>WORKDIR /app
</span>COPY . .
</span>RUN yarn install
</span>RUN yarn app:build
</span>
</span>FROM node:16-alpine as deps
</span>WORKDIR /app
</span>COPY package.json yarn.lock ./
</span>RUN yarn install --production --frozen-lockfile
</span>
</span>FROM node:16-alpine
</span>WORKDIR /app
</span>COPY --from=deps /app/node_modules ./node_modules
</span>COPY --from=dist /app/package.json /app/.env ./
</span>COPY --from=dist /app/dist/src ./src
</span>
</span>EXPOSE 3090
</span># CMD du -sh ./node_modules/* | sort -nr | grep '\dM.*'
</span>CMD node src/app.js
</span></code></pre>
In line 19 I comment a sh command to print size of depencencies in node_modules</code>. So you can check what dependencies remained in the final build and possibly remove some of them, or replace them with the optimal version. For example, if you use AWS S3 in your application, you can use a dependency only for S3 and not for all AWS services.</p>
Final size of the Docker Image is 322Mb, which 4x times smaller than original image.</p>
> docker </span>images </span>| </span>grep</span> app
</span>app</span>     latest    b6f283f00bb2   12 seconds ago   322MB
</span></code></pre>
Conclusion</a></h2>
Using simple things you can reduce the size of the image of your application and speed up the time between building and running the application in the production.</p>

That’s all for now. I hope that article was useful for you. If yes, don’t forget to follow me to get new updates.</p>
vladkens.cc

My 2025 Summary

Optimizing RIPEMD-160 with SIMD

From iTerm To WezTerm

2024 In Review

Creating OG Images for Static Pages with Rust

Deploy FastAPI application with SQLite on Fly.io

Setting up WireGuard VPN at AWS with Terraform

Objective</a></h2> We need to provide a quick and easy solution to give access to internal AWS VPC resources for our team. In general, there are two ways to do this:</p> Bastion / Jump Host – EC2 instance with SSH access for each member</li>

Fast multi-arch Docker build for Rust projects

Problem definition</a></h2> So there are two problems with building multi-arch Docker images for Rust projects:</p> Layer invalidation and full rebuild on any change in Cargo.toml </code></li>

apigen-ts – Simple TypeScript API Client Generator

Table sorting and pagination with HTMX

How to get macOS power metrics with Rust?

macmon – Mac Usage Monitor in Terminal

REST API Deployment on AWS Lambda with Terraform (2023)

AWS ECS Cluster on EC2 with Terraform (2023)

How to still scrape millions of tweets with twscrape

Better forms handling with react-use-form

Better date parsing for REST API with TypeScript

Docker Compose templating using YAML merge feature

asdf — a good tool to manage python, node, etc runtime

How to see env in Docker container?

Keep Type Guards valid after refactoring in TypeScript

Sync all Git branches with remote

Docker Desktop alternative for MacOS

How to reduce NodeJS Docker Image Size?
