Hugo Martins

Abstractions can be expensive

Sat, 10 Aug 2024 00:00:00 +0000

Daniel Lemire wrote in Performance tip: avoid unnecessary copies, comparing Bun to Node.js, about a particular case where copying data becomes expensive:

It turns out that the copy was not happening as part of base64 decoding but in a completely separate function. There is an innocuous function in Node.js called StringBytes::Size which basically must provide an upper on the memory needed by the Buffer.from function.

As shown in the benchmarks, in the piece, Node.js becomes much slower than Bun when computing the same thing (e.g decoding base64). Even though both of these runtimes use the same library, an unsuspicious copy of data deep in the code of Node.js made it more expensive and slower.

I’ve encountered a similar situation in Go. A service was misbehaving and using too much memory. Profiling and investigating, I narrowed it down to the fact that it was using io.ReadAll to transform a response into a slice of bytes ([]byte). In itself, there is nothing wrong with this idea but there’s something particular about io.ReadAll (source):

func ReadAll(r Reader) ([]byte, error) {
	b := make([]byte, 0, 512)
	for {
		if len(b) == cap(b) {
			// Add more capacity (let append pick how much).
			b = append(b, 0)[:len(b)]
		}
(...)

Inside a for loop, ReadAll will validate that the buffer it created (512 bytes) has enough capacity to store the data. If it doesn’t, it duplicates the size of the existing buffer. This happens until the buffer has enough space to store all of the data. It inherently means we will use a lot more memory than the original size of the data. All of this might be unexpected.

If we don’t know the size of the data a priori, there’s not much we can do to reduce this (I believe) but, if we do know the size of the buffer, io.Copy might be a good alternative. It allows you to specify the size of the buffer, before writing data to it, avoiding the memory-intensive capacity increases.

This was true up until, at least, Go 1.20.4. In recent versions, the implementation has changed. But it doesn’t change the underlying fact that this abstraction could potentially be expensive memory-wise, or even contribute to a program performing poorly or misbehaving. As Lemire points out:

The story illustrates why our software is slower than it should be. We have layers of abstractions to fight against. Sometimes you win, sometimes you lose.

These layers are there for a reason, but they are not free.

To make matters worse… these abstraction layers often thicken over time… and the friction goes up.

It is not that the layers are unnecessary: they do good work. They allow us to avoid reimplementing the same things over and over again. They allow us to build on top of several individuals’ expertise and they support frameworks that allow us to be more productive…but they are not always free, there’s a cost associated with these abstractions. And it is important to keep an eye (and monitoring) open to the possibility that sometimes, only sometimes, they are more expensive than we are willing to accept.

On What Lexers Do

Tue, 21 Feb 2023 15:31:16 +0000

A couple of weeks ago, I’ve started reading Writing An Interpreter In Go. Reading this book means I’ll be building an interpreter for a language called monkey (you can follow my implementation). It is a topic that I’ve always been curious about but never actually took the plunge and decided to study it.

I’ve dabbled a little bit with it in university. But we never actually wrote any of the internal tools. We always used generators for the code and focused more on the concepts. Rest assured, I can’t remember most of it by now.

While reading this book, I reconnected with the basic concepts of how interpreters and compilers work. I’ve been taking notes and trying to expand my knowledge with other sources. I’ve always heard that to know something, you need to be able to explain it. I’m trying to do that here.

Lexers transform source code into symbols (or tokens). They execute what is called “lexical analysis”. From Writing an Interpreter in Go:

(…) from source code to tokens, is called “lexical analysis”, or “lexing” for short. It’s done by a lexer (…)

Wikipedia defines “lexical analysis” as “the process of converting a sequence of characters (…) into a sequence of lexical tokens (…)”. This aligns well with what we read in the book.

So, when we execute a lexer, we transform the source code (text) into a series of data structures (tokens). It is an integral component of writing interpreters (or compilers) as we need to transform source code into a representation that we understand and can process in later stages (e.g. parsing).

As an example, in Go, if we have the statement str := "string", we are in the presence of three tokens: an identifier token (str), an assignment token (:=) and a string token (string). Tokens are meaningful data structures, allowing us to approach source code as data structures, rather than dealing with text inside of interpreters.

Lexers also help us ignore elements of source code that aren’t relevant (e.g. whitespaces in some languages) and focus just on the elements that actually matter to us. After lexing, all irrelevant whitespace has been stripped off. For example, in C or Go, I wouldn’t expect whitespace to appear as tokens after lexing. In Python, where whitespace is relevant, I expect it to be taken into account in lexing.

One concept I was particularly interested in, as I read, was the fact that lexers need to look-ahead. This isn’t always done the same way, it depends on the approach used, but it is integral to lexing, to be able to resolve conflicts.

As an example, reading a “=“ sign might represent different things. It can be an equal sign (e.g. assignment) or it can be the start of a comparison (e.g “==“). Without looking ahead, it is impossible to resolve conflicts and accurately know what a given character represents in source code.

Lexical analysis is only but a small step, in the process of interpreting source code and its results, that will need to be fed into a parser, which will perform additional steps with the tokens.

It has been a good experience reading these initial sections of Writing An Interpreter In Go and finally feel I’m starting to understand the basics of these concepts.

Traffic: July to December 2022

Tue, 14 Feb 2023 00:00:00 +0000

I’ve kept this tradition of looking back at the traffic every six months since 2021 (here and here). This time, it also marks the end of the year, in terms of traffic, which makes this review two-fold: a six-month review along with a yearly review.

In total, from July 2022 to December 2022, there were 11868 visitors and 13517 pageviews (~73/day). Google is still the biggest source of traffic, followed by Bing and DuckDuckGo. Compared with the previous month, visitors were down 10%, while pageviews were down 12%.

This isn’t a surprise to me in any way. I have written only two essays during this period and I didn’t share any of them broadly, I was actually quite surprised when I noticed the numbers didn’t go down even more.

The top essays have remained the same, throughout this semester, with only a slight change in ranking:

Using require and check Functions in Kotlin surpassed both How to Access Host Resources in Minikube Pods? and Understanding Differences Between List and MutableList in Kotlin.

Throughout 2022, there were 24994 unique visitors and 28673 pageviews. Most of what was written in 2022 didn’t get much traction, except for Essential Fields in Kubernetes Manifests, which seems to have found its way to Top 10 somehow.

I didn’t share much (or anything) of what I wrote during the entire year, which is something I might want to change in 2023, but some of the things I wrote on the journey to CKAD seemed to have resonated with people, even though I didn’t share them.

For 2023, I don’t want to necessarily increase traffic but I’d be happy if, by writing and sharing more, traffic would necessarily grow organically. I’ll put my focus more on the writing than I did in 2022, and we’ll see the impact doing that will have.

On Unit Testing in Temporal

Wed, 28 Dec 2022 00:00:00 +0000

I’ve been using Temporal, “a developer-first, open source platform that ensures the successful execution of services and applications”, for troubleshooting purposes for a while now. We use it quite extensively at Datadog. Recently, I’ve started actually implementing something on top of Temporal and was amazed at the simplicity of its testing framework.

At first I was a bit confused, as I’m interested in Go and could only find documentation focusing on Java. Eventually, I found How to test Workflow Definitions in Go, a great introduction to the testing framework that Temporal offers with testsuite.

It all starts with a WorkflowTestSuite, in which we can initialize a TestWorkflowEnvironment. This TestWorkflowEnvironment is powerful and we immediately get access to a few important functions, such as AssertExpectations or ExecuteWorkflow. These allow you to do things as diverse as checking that all expected calls (to mocks) have been made and to execute a workflow inside a test environment. We can use RegisterActivity or RegisterWorkflow as well, to ensure our Activities and Workflows are available in the environment we’ll execute our tests in.

Unfortunately, as pointed out in Temporal’s documentation, “unless the Activity invocations are mocked(…) the test environment will execute the actual Activity code including any calls to outside services.” This is particularly harmful if the Activities being invoked execute code that makes calls to endpoints on a network because:

it makes the tests unable to execute without connecting to the network;
it makes for unreliable and inconsistent behaviour of the tests;
and it doesn’t allow us to test multiple failure modes.

For this reason, Temporal’s testing framework offers methods to mock Activities in a TestWorkflowEnvironment such as OnActivity. This works well when the Activities are implemented internally but I couldn’t get it to work when the Activities are implemented in a third-party package.

I read a bit of the temporalio/samples-go repository, which houses a collection of samples and accompanying tests, to understand what needs to be done. It is seems simple enough:

type UnitTestSuite struct {
    suite.Suite
    testsuite.WorkflowTestSuite
}

func (s *UnitTestSuite) TestSampleWorkflow() {
	// Initialize test environment for UnitTestSuite.
	env = s.NewTestWorkflowEnvironment()

	// Register needed Activities.
	env.RegisterActivity(externalpackage.ExternalActivity)

	// Mock Activities.
	env.OnActivity(externalpackage.ExternalActivity, mock.Anything, mock.Anything).Return(nil)

	// Execute Workflow and Assert.
	env.ExecuteWorkflow(SampleWorkflow)
	(...)
}

I was surprised it was this easy. I confess I was expecting a little bit more complexity. That might be due to my lack of knowledge about the internals of Temporal, but I have to say I was quite excited about how easily I was able to make progress.

Traffic: January to June 2022

Mon, 25 Jul 2022 00:00:00 +0000

Last year I started publishing short summaries about the traffic this website receives. You can see the initial reports for January to June 2021 and July to December 2021. This year I’m trying to keep doing just that. Here’s the traffic this website received during the initial semester of 2022.

In total, from January to June 2022, there were 13127 unique visitors and 15156 pageviews (84/day). Google was where most traffic came from, followed by Bing, DuckDuckGo and Yandex. Compared with the previous six months, pageviews per day were up 1.7x, which I attribute to the compounding effect of Google redirecting more traffic to the website since around October of last year.

I’ve written five essays throughout the initial half of 2022, with one being a traffic summary and four of those being focused on Kubernetes and CKAD. Throughout this initial semester of 2022, the top essays have remained exactly the same:

. I attribute this situation to three reasons:

Google has continuously been directing people to those articles, which will naturally make them stand out more.
I haven’t published anything in April, May or June, and have published five pieces in January, February and March. This means that there isn’t a lot of fresh essays ready for people to read, so they will naturally gravitate to older essays.
I haven’t made much of an effort to promote the website, which means that once the most recent essays are published they won’t get as much traffic in the longer term, at least not as much as the most established essays.

I’ve started this website as a place where I could explore the practice of writing, where I could experiment and, in some cases, take notes. I’m not discouraged by the halt in growth because it makes sense, given what I’ve produced recently.

The website keeps receiving some traffic, which I’m happy about, even though I haven’t been as consistent with a publishing schedule, which is really what I’m not so happy about and I will aim at improving that in the following six months.

Understanding 'kubectl explain'

Tue, 29 Mar 2022 00:00:00 +0000

kubectl explain can be used to get more information about which fields a given resource needs, as well as the meaning behind those fields. It can be used in mainly two ways: either for getting information about the fields of a resource (e.g. kubectl explain <resource>) or by getting information about a specific field of a resource (e.g. kubectl explain <resource>.<field/s>).

Together with api-versions and api-resources it can quickly provide a lot of insight into Kubernetes and what needs to be written on a manifest to properly create or edit objects.

Examples

Getting information about root fields of Pods:

$ kubectl explain pods
KIND:     Pod
VERSION:  v1

DESCRIPTION:
     Pod is a collection of containers that can run on a host. This resource is
     created by clients and scheduled onto hosts.

FIELDS:
   apiVersion   <string>
     APIVersion defines the versioned schema of this representation of an
     object. Servers should convert recognized schemas to the latest internal
     value, and may reject unrecognized values. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources

   kind <string>
     Kind is a string value representing the REST resource this object
     represents. Servers may infer this from the endpoint the client submits
     requests to. Cannot be updated. In CamelCase. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds

   metadata     <Object>
     Standard object's metadata. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata

   spec <Object>
     Specification of the desired behavior of the pod. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status

   status       <Object>
     Most recently observed status of the pod. This data may not be up to date.
     Populated by the system. Read-only. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status

Getting information about the spec of Pods:

kubectl explain pods.spec
KIND:     Pod
VERSION:  v1

RESOURCE: spec <Object>

DESCRIPTION:
     Specification of the desired behavior of the pod. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status

     PodSpec is a description of a pod.

FIELDS:
   activeDeadlineSeconds        <integer>
     Optional duration in seconds the pod may be active on the node relative to
     StartTime before the system will actively try to mark it failed and kill
     associated containers. Value must be a positive integer.

   affinity     <Object>
     If specified, the pod's scheduling constraints

   automountServiceAccountToken <boolean>
     AutomountServiceAccountToken indicates whether a service account token
     should be automatically mounted.

   containers   <[]Object> -required-
     List of containers belonging to the pod. Containers cannot currently be
     added or removed. There must be at least one container in a Pod. Cannot be
     updated.

   dnsConfig    <Object>
     Specifies the DNS parameters of a pod. Parameters specified here will be
     merged to the generated DNS configuration based on DNSPolicy.

   dnsPolicy    <string>
     Set DNS policy for the pod. Defaults to "ClusterFirst". Valid values are
     'ClusterFirstWithHostNet', 'ClusterFirst', 'Default' or 'None'. DNS
     parameters given in DNSConfig will be merged with the policy selected with
     DNSPolicy. To have DNS options set along with hostNetwork, you have to
     specify DNS policy explicitly to 'ClusterFirstWithHostNet'.

(...)

It explains what this field represents, as well as all the fields that can be configured inside this specific field, and their meanings. It can be a powerful tool to understand Kubernetes. A sort of man for Kubernetes itself.

It can be particularly interesting to use kubectl explain to inspect, when you forget, essential fields in Kubernetes manifests, or internal details.

What Can Be Done With 'kubectl run' Command?

Fri, 18 Mar 2022 00:00:00 +0000

kubectl run is a command that can “create and run a particular image in a pod”, which means that it will start a pod with a particular image. It can also create the pod with some particular details by using combinations of flags. We can see the code for it in pkg/cmd/run/run.go.

Interesting Flags

--image specifies the image to be executed.
--command allows to overwrite the command field in the container with whatever arguments are sent after --.
--env to set environment variables in the pods, in the form of key:value and one variable per --env flag.
--expose and --port to create a ClusterIP associated with the pod.
--labels which allows us to configure labels for the specific pod. Labels are accepted with only one flag, separated by a comma.
--dry-run for testing purposes.

Examples

From documentation:

kubectl run nginx --image=nginx
kubectl run hazelcast --image=hazelcast/hazelcast --env="DNS_DOMAIN=cluster" --env="POD_NAMESPACE=default"
kubectl run hazelcast --image=hazelcast/hazelcast --labels="app=hazelcast,env=prod"
kubectl run nginx --image=nginx --command -- <cmd> <arg1> ... <argN>

Use Case

While I don’t see much use in kubectl run for dealing with production grade systems, it can be a great resource for testing out different configurations of pods and creating a bunch of pods if there’s a need for it. It can also be a great introductory command for when first starting to deal with Kubernetes because it eases the interaction with Kubernetes without having to deal with YAML right away.

Due to its --output flag, it can also be a great way to use kubectl to create pods, in a stable way, from scripts because we can have stable and reproducible outputs from it. The fact that we can use a --dry-run flag is also a big plus.

Essential Fields in Kubernetes Manifests

Tue, 15 Feb 2022 00:00:00 +0000

I recently started a quest to complete CKAD in the next few months, by May 2022. As I’ve explained in that previous essay, “I have spent quite some €€€ enrolling in it and I feel that it can still teach me a lot of relevant concepts about Kubernetes that will be useful” on my day-to-day.

While studying for CKAD, through Kubernetes Certified Application Developer (CKAD) with Tests and KodeKloud, I’ve come to realize the importance of understanding the syntax of manifests in Kubernetes. Subconsciously, I obviously already knew this - the same way I know how important it is to dominate the syntax of a given programming language - but it is too easy to fall into a pattern of copy-pasting-and-changing, or simply filling in the gaps in already existing manifests.

Manifests in Kubernetes are the baseline of describing and defining resources, that we can then create and edit afterwards. Manifests represent the object specification describing “its desired state, as well as some basic information about the object (such as a name).” ¹ These manifests are most often described in .yaml files.

In essence, there are four essential fields in Kubernetes manifests that must be present in all manifests. These are: apiVersion, kind, metadata and spec. Each of these might have widely varying values populating them.

As an example, a starting point for a Kubernetes manifest would be:

apiVersion:
kind:
metadata:
spec:

apiVersion

apiVersion allows us to define what version of the API a given resource is going to be using. It can be simply v1, which means it will be part of the core API specified at /api/v1. It can also be <name>/<version>, for example batch/v1, specifying that at resource uses an API that is under /api/<name>. ²

We can find more about what APIs and versions exist on a given cluster by executing kubectl api-version:

$ kubectl api-versions
admissionregistration.k8s.io/v1
admissionregistration.k8s.io/v1beta1
apiextensions.k8s.io/v1
apiextensions.k8s.io/v1beta1
apiregistration.k8s.io/v1
apiregistration.k8s.io/v1beta1
apps/v1
authentication.k8s.io/v1
authentication.k8s.io/v1beta1
(...)

Results will differ from cluster to cluster, and between Kubernetes versions. We can have custom APIs, disabled APIs, or recent APIs could’ve been implemented in different Kubernetes versions.

ReplicationController and ReplicaSet are two popular objects that clarify this and differ in the apiVersion. ReplicaController is a component of the core API in v1 so we would write apiVersion: v1 in its spec. ReplicaSet, which evolved from ReplicaController, is a component of a more recent API served at apps/v1. This sort of versioning is incredibly powerful and flexible, allowing Kubernetes to evolve while keeping a lot of backwards compatibility.

kind

kind represents the kind of object that is specified via a manifest. Each kind of resource will be available on a particular API. This makes it essential that the specified kind and apiVersion match on a specific manifest.

We can inspect which kind we can use for objects by executing kubectl api-resources:

kubectl api-resources
NAME                              SHORTNAMES   APIVERSION                        NAMESPACED   KIND
bindings                                       v1                                true         Binding
componentstatuses                 cs           v1                                false        ComponentStatus
configmaps                        cm           v1                                true         ConfigMap
apiservices                                    apiregistration.k8s.io/v1         false        APIService
controllerrevisions                            apps/v1                           true         ControllerRevision
daemonsets                        ds           apps/v1                           true         DaemonSet
deployments                       deploy       apps/v1                           true         Deployment
replicasets                       rs           apps/v1                           true         ReplicaSet
statefulsets                      sts          apps/v1                           true         StatefulSet

(...)

With kubectl api-resources we can quickly see as well which apiVersion needs to be specified in order to specify a particular resource.

metadata

metadata describes information of an object that allows for the unique identification of that object. When creating a manifest, this field should have at least an associated name. Usually we will also see a field named labels.

spec

spec defines the desired state of the object in Kubernetes. It will vary widely between different resources and API versions, which means that it can be tricky to figure out - or memorize - all the needed fields.

I’ve found that there’s one instance of a resource that can be created without a spec field which is namespaces. If we create a namespace with only apiVersion, kind and metadata, creating the namespace with kubectl create, Kubernetes will accept that manifest but it will create the namespace internally with an appropriate spec. As an example:

apiVersion: v1
kind: Namespace
metadata:
  name: my-namespace

Running kubectl create results in:

$ kubectl create -f my-namespace.yaml
namespace/my-namespace created

Executing kubectl get will allows us to see the injected spec field:

$ kubectl ns my-namespace -o yaml
apiVersion: v1
kind: Namespace
metadata:
  name: my-namespace
  (...)
  selfLink: /api/v1/namespaces/test
  uid: f1b901a6-31d6-457a-aaaf-0cb6d600d52c
spec:
  finalizers:
  - kubernetes
status:
  phase: Active

All of this information can be confirmed in Kubernetes’ own documentation by reading Required Fields. Although this isn’t a deep exploration of manifests, having solid bases can be extremely important to understand what has been built on top of this. Reasoning about this structure also provides a glimpse at the baseline that provides so much flexibility to Kubernetes, allowing it to have 50+ components out-of-the-box and a lot of extensibility via custom resources.

Certified Kubernetes Application Developer (CKAD) in 2022

Sun, 06 Feb 2022 00:00:00 +0000

May 2021 in sunny Portugal. I virtually attended KubeCon + CloudNativeCon Europe 2021, writing a series of daily highlights about this experience. It focused on the talks I most enjoyed, what I learned and technologies I’d keep an eye on. I even took the time to write some tweets, even though I barely use Twitter.

At the time, I was using Kubernetes extensively. I was working as an SRE, which was really more a Platform Engineer, working on tooling in the Software Development Life Cycle ™️ domain, following the hype of Internal Platform Teams. Kubernetes was quickly becoming, if it wasn’t already, the de facto standard framework to build these sorts of platforms on. Kubernetes serves as the underlying “OS” of sorts, while a lot of applications and customization are baked on top of it.

Due to the type of work I had been doing, and because it is a domain that interests me and I want to keep working on it, I ended up making the decision to enroll in the Certified Kubernetes Application Developer (CKAD). I racked in an hefty discount from attending KubeCon + CloudNativeCon Europe 2021 and had an entire year to study for, register and pass CKAD. What could go wrong?!

It was all very promising and hopeful but I ended up not being able to study for CKAD and ended up not registering to take the exam after almost a year. To make matters worse, CKAD went through a series of program changes just four months after I enrolled. It now includes questions about Helm, more security-oriented aspects, deployment strategies and CRDs, among other changes.

Although I no longer feel the same motivation and excitement towards passing the exam, mostly because I’m not working as closely with Kubernetes anymore, I have spent quite some €€€ enrolling in it and I feel that it can still teach me a lot of relevant concepts about Kubernetes that will be useful, as well as help me improve my kubectl proficiency, which I still use everyday.

Because of the reasons above, I’m going to try and complete Certified Kubernetes Application Developer (CKAD) in 2022, more specifically, in the last few months I have until May. This means that I’ll have to dive into Kubernetes concepts and tooling, while studying, and write more about it which I’m hoping I’ll share in the form of small snippets and notes.

I don’t want it to become a reference guide to Kubernetes but I believe that sharing notes and small essays might help me get more motivated towards studying for CKAD. In this way, I believe, I’ll probably start a small series of notes on Kubernetes. I’m not yet sure on what this series will be in terms of structure, or its name, but I’m hopeful that it will give me the extra boost I need in order to successfully study and complete CKAD.

Traffic: July to December 2021

Tue, 04 Jan 2022 00:00:00 +0000

Earlier this year I wrote Traffic: January to June 2021 with a quick summary about the traffic this website received between January and June of 2021. As I stated when I wrote it, it “isn’t necessarily an immediate concern whether this website gets much traffic” but it is always interesting to analyze and I’m always curious about it. So, without further ado, here are a few notes about the traffic this website received between July and December 2021.

In total, from July to December, there were 8091 unique visitors and 9283 pageviews (50/day). Google was where most traffic came from, followed by DuckDuckGo, Bing and LinkedIn. Comparing with the previous six months, pageviews per day were up 2.3x which I attribute to Google appreciating some essays and redirecting much more traffic to them than before.

Top Essays:

All of the top essays were already the top essays in the previous 6 months, although How To Pass Multiple Parameters To Same Value in Query Strings jumped from fifth to first. This change seems to be motivated by Google appreciating this essay much more than the remaining ones.

Although, during these six months, I’ve written much less than in the previous six months (5 essays), it seems that the content written in the beginning of the year has lived long and enjoyed a good amount of traffic from Google.