An Ode to Maintainers

This week I read an incredible article about maintainers.This article is mostly focused towards infrastructure maintenance. It completely adapts itself to the current reality of software development.

The reality is that maintenance is not as shiny as innovation. But maintainers are, mostly, what keeps society, and code, working. If it weren’t for the software maintainers most of our software would already be broken, by now.

This excerpt, from the article, beautifully, explains what I feel about this:

The most unappreciated and undervalued forms of technological labour are also the most ordinary: those who repair and maintain technologies that already exist, that were ‘innovated’ long ago.

Hail the maintainers!

Bash and Floating-Point Arithmetic

This week I was working on HackerRank’s Arithmetic Operations. Turns out, to my surprise, that Bash does not support floating point operations. I had no idea.

Off I went, in my searching. I searched through a lot of StackOverflow answers. The best one I found is this one - a summary of possible solutions. This answer is objective and straight to the point, offering enlightenment into a lot of possible solutions to go around this problem. Nonetheless, it is dependent on the context which meant I needed to do some research.

I ended up using the solution involving bc, an arbitrary precision calculator language. This nifty tool allows you to do floating-point calculations straight from your terminal.

Usage of the tool is pretty simple. You just pipe an expression into it, and it outputs the result of processing said expression. As an example, you can do stuff like this:

$ echo "3 / 2" | bc
1

As you can see, if you just pipe a usual expressions, it won’t produce floating-point types. For that, you need to let bc know you want to see the actual decimals using scale=X. The above example, with this new information, would be:

$ echo "scale=3; 3 / 2" | bc
1.500

As you can see it now returns 3 decimal points - 1.500. Unfortunately, this started showing some odd behaviour. For some calculations, it would round the decimal points incorrectly. I started searching and found that there’s some undefined behaviour when dealing with decimal points equal to 5. From 1 to 4, it rounds down. From 6 to 9, it rounds up. It starts getting very tricky when the rounding has to be done on a decimal point of 5.

This means that if I had a value of 1.5675, bc might actually round that to 1.567. I couldn’t have that because, for my specific context, I needed 5 to be rounded up.

I found the solution to this problem by using printf along with bc. I make my calculations with bc and use printf for rounding. This has the possible issue of requiring you to decide on a fixed maximum decimal rounding but it works for me.

As an example:

$ echo "scale=2; 13 / 7" | bc
1.85

Scaling with scale=1, we get an undesired rounding:

$ echo "scale=1; 13 / 7" | bc
1.8

But when using printf:

$ printf "%0.1f\n" $(echo "scale=2; 13 / 7" | bc)
1.9

The more I use floating-points, the more I understand how complex they are. I’ll probably never get to the bottom of it all but at least I’ll never forget how bc and printf can, together, be a solution for this problem.

Automatic Deploys with Hugo, TravisCI & Github Pages

tl;dr

I automated the deployment of this website with Travis CI, Github, Hugo and Github Pages. The website gets build from each push to master and, if the build is successful, deployed to Github Pages for website hosting.

Introduction

A few months ago I wrote an article describing how interesting it can be to automate the deploy workflows - Automate Simple Deploy Workflows - Cool!. The easeness of deploying automatically was appealing but it still required some manual work. This is due to the way I wrote the deploy scripts.

During the last few days I’ve been thinking about better ways to automatically deploy this website, from a repository where its source code is hosted to another repository that serves as a Github Pages’ repository.

The initial article I wrote had one lazy script - deploy-sh - that did all the hard work of compiling the website and pushing it to the remote host. That wasn’t good enough for me.

When I started thinking about this, I set some goals:

  • To only manage one repository. One is enough.
  • To not have to insert any credentials when pushing to remote but still use them.
  • The build and deployment had to be done for each commit to master. This is the only way for the website to be updated.
  • It had to be triggered without me doing anything other than pushing the source code to the original repository.

To achieve this goals I used:

  • Hugo - still the best static website engine I have tried.
  • Travis CI - to manage the builds.
  • Github - where the git repos are hosted.
  • Github Pages - for website hosting.

Setting Up deploy.sh

The initial deploy.sh script was as simple as:

#!/bin/bash

echo "Building website..."

cd ..
hugo -t cocoa > /dev/null 2>&1
cd "public/" || exit

echo "Copying new website into other repo..."

cp -R ./* ../../caramelomartins.github.io/

echo "Push changes to remote..."

cd "../../caramelomartins.github.io/" || exit
git add -A > /dev/null 2>&1
git commit -m "Push new content" > /dev/null 2>&1
git push > /dev/null 2>&1

echo "Finished deployment..."

This was bad for several reasons. The website was built along with deploying it. It needed acess to an environment that was already configured git. It didn’t provide any information of what commit was deployed.

I decided to revamp this script, from scratch. First I decided to use an environment variable called GITHUB_API_KEY to represent a personal token that can be used to push the changes.

We begin by cloning the Github Pages repository:

echo "Cloning caramelomartins.github.io..."
git clone -q "https://caramelomartins:[email protected]/caramelomartins/caramelomartins.github.io.git"

After this is completed, we force the configuration of git on Travis CI. This allows me to control who pushes changes from the Travis CI settings’ pages.

echo "Setting up git configuration..."
git config --global user.email "$GITHUB_EMAIL"
git config --global user.name "$GITHUB_NAME"
git config --global push.default matching

GITHUB_EMAIL and GITHUB_NAME are also environment variables.

We proceed with replacing the contents. First by removing the old contents and then by copying the new content.

echo "Removing contents..."
rm -r caramelomartins.github.io/*

echo "Copying new contents..."
cp -r public/* caramelomartins.github.io/

This presented a challenge: CNAME was replaced in each build. To circunvent this, I added the CNAME file to the content of the website. Now, CNAME is copied along.

I wanted to know what commit I was deploying from so I added a simple line to this script: HASH=$(git rev-parse --short HEAD). This gives me the commit hash of the latest commit in HEAD - this represents the last commit to master since Travis CI is configured to only build on master commits.

I then push the new contents:

echo "Pushing new contents..."
cd caramelomartins.github.io || exit
git add ./*
git commit -q -m "Auto Deploy - $HASH" -m "Built from: https://github.com/caramelomartins/website/commit/$HASH." --allow-empty
git push -q "https://caramelomartins:[email protected]/caramelomartins/caramelomartins.github.io.git"
cd ..

This provides a beautifully minimalist commit message in every deployment:

Auto Deploy - f8154f7
Built from: caramelomartins/[email protected]

It tells me what commit was used for deployment, along with an URL to that commit.

After that is done, the script cleans up the assets from the Travis CI environment and exits. The script got 10 lines longer but it became much more effective. You can see the whole script here.

Configuring Github

After the script was done, I had to configure a personal access token. This can be done on the Settings of your Github profile. Under “Developer settings”, there’s a section called “Personal access tokens”.

A personal access token will allow you to authenticate with Github without having to share your credentials. It also allows you to set the exact permissions of your access token.

In this case I only wanted to push to my public repo. That’s exactly the only permission I gave to it: repo.public_repo.

Configuring Travis CI

With Travis CI, I just wanted to have it export 3 shell environment variables. The ones I talked initially. So I configured:

  • GITHUB_API_KEY
  • GITHUB_EMAIL
  • GITHUB_NAME

These environment variables which are configured to be hidden from the logs, allow me to push to another repository without disclosing my credentials on Github itself.

Conclusion

With this solution I reached my initial goals. I now have fully automated deployments from each commit to master.

Now that the basics of the deployments are done, adding testing or minimazing assets automatically should be fairly simple to do.

On Elevating Shell Scripts

Automatically obtaining superuser permissions always seemed like black magic. I have a vague idea of how it works and I am very aware of their potential interesting uses.

I have always seen those magic installers doing their black magic behind a (not so) beautiful GUI and wondered “how the hell can they install things in places my user can’t seem to reach?”.

Turns out that’s not that difficult and it can be achieved with relative transparency.

Recently, I have been tasked with writing some installation scripts which, in turn, had to access /usr/local/. As most will know, this is usually forbidden land for regular users. Most can only read and can’t even read anything.

So I set out to figure out how would I be able to install files in places users can’t access, without the help of a GUI.

My version of occurrence was the usual one: make them sudo the script. This solution would result in something like:

sudo script.sh

I find this obnoxious because you are not helping your users. They need to know before hand that they need to run this as superuser and actually run the command as a superuser. I discarded this idea as simple but not optimal.

I went on to Google-land to search for a better wait to do it. Right away I found a couple of answers ranging from “this is awful” to “you can do it this way”.

One of the best resources I found was this answer on unix.stackexchange.

Reading through that answer I am intriged by five things, in those snippets: $EUID, $0, [email protected], $? and $UID. These looked like interesting, useful Bash environment variables. I research a little bit more to find out what they are.

Again, I turned my attention to StackOverflow, specifically this answer, this one and Wikipedia. I not always use StackOverflow but sometimes it helps a ton. $EUID and $UID, represent two separate things. $UID gives the user identifier which identifies the real user ID of the calling process, in the system. We can use $EUID to give the effective user, in the system. Apparently, there is also the file system, saved and real (ruid). [1] The differences between all this types of users are still unclear to me but, what I know so far, it that a superuser will always have $EUID as 0. An important note is that this works only for bash.

The remaining parts of this are also bash variables:

  • $0 is a positional parameter that refers to the filename of the script in execution. So, it is effectively argument - 1.

  • [email protected] represents the parameters, “without interpretation or expansion”. It will output the parameters as they were written.

  • $? will get you the exit status of a command.

All of this can be further investigated on Advanced Bash-Scripting Guide’s Chapter 9.

The top voted answer was an interesting way to ask the system: “Are you running this as superuser?”. If the system reply that it wasn’t running it as super user, it would re-run the script, along with its parameters, preprending sudo. The full answer is very short but very powerful:

if [ $EUID != 0 ]; then
    sudo "$0" "[email protected]"
    exit $?
fi

So, anywhere on top of this solution, you let the user know that elevated permissions will be required. This provides an interesting command line interface, to elevate your permission level, without the user having to know that he must run it as a superuser.

Of course, all this has security implications that must be consider. Then again, running anything, as a superuser, without trusting what you are running wouldn’t be much of an option either.

[1] http://man7.org/linux/man-pages/man2/getuid.2.html

Managing Atom's Installed Packages with Ruby

I use the Atom editor a lot. In fact, be in Windows or Linux, it’s my standard text editor.

I love it’s flexibility and the fact that it’s completely hackable. I can do whatever I want with it. It supports a whole range of programming languages. It has great support from community.

With that being said, managing a Windows configuration and a Linux configuration, in separate devices, can be very confusing.

Last week, I had my first experience of messing up my configuration because of that. I started messing around with Atom’s packages folder and, soon enough, I had deleted it. Given that I had another configuration, I was able to salvage what I had just deleted - not without some panic involved.

I decided that I would no longer suffer such panic again, so I wrote a nifty little Ruby script to help me.

The script is very basic. Clocking at 39 LOC - with 10 being comments.

What the script does is very simple:

  • Fetch all packages in Atom, in JSON format, with apm list -j.
  • Iterate over those packages and filter only the user installed packages.
  • For each of the user installed packages, fetch its name, version, homepage and repository details.
  • Save that information into an array.
  • Write it to a file, in a predefined location.

As you can see by the easiness of the description, the script is basic. Although it is a basic script, it allowed me to get a taste for the Ruby language, which I have been yearning to learn for some months.

I must say that I’m amazed both at its power and syntax. I’ll be sure to keep working through it to see if I can hack it.

The code to the script is open sourced on my dotfiles repository.

Time & Choice

Time is a valid asset. It’s the most valuable asset we have in our lives. It doesn’t scale. It’s impossible to scale time.

We - as programmers, developers and tinkerers - have a multitude of interests that aren’t always compatible. This means reading and learning things at speeds that are not always advisable.

This is way time seems to always slip away. If more time is available to us, we’ll spend it until the last second.

Today I read a very interesting article about one of Linus Torvald’s talks. In the midst of talking, about software development, Linus said something that strikes with me:

“The innovation the industry talks about so much is bullshit. Anybody can innovate. Don’t do this big ‘think different’… screw that. It’s meaningless. Ninety-nine per cent of it is get the work done.”

I’m not sure why but this quote resonate a lot with me and what I’m thinking about at this moment.

The work of a software developer, or an engineer, is getting the work done. It’s sweating out the details. It is delivering the solution to a problem, with quality, in a timely efficient manner.

It will never be just about creating innovation. Sometimes it’s just about fixing a problem - and that’s hard enough.

With that in mind, we have to be very selective of what we want to do and why. There’s a lot of noise in the development community and we have to focus and be there to work out the details.

To build what really matters to us.

Trying to Understand Computation & Complexity

The theory of computation is a subject that has, since an early moment, caught my interest. I had a course during my Bachelor’s study called “Computational Theory” and I liked it a lot.

I always liked to understand, with detail, how things work. How better to understand computers than to understand the roots behind how they work?

In that interest, and since I never actually got around to doing so, I’m reading Introduction to the Theory of Computation by Michael Sipser. I’ve been reading it for about two months, trying to absorb everything in it. Hey, I’ve been doing other things too!

Looking at the theorems. Looking at proofs of theorems. Looking at corollaries. This has given me a new perspective on the mathematics behind computation.

I hope I’ll be able to follow this with other writings related to this topic.

Automate Simple Deploy Workflows - Cool!

I recently changed my blog from Jekyll to Hugo because it’s fast, has custom support theme, shares my name and I never got along with Jekyll, honestly.

With changing my configuration I had to make some changes to the workflow I used with my blog and writing. With that in mind I created a new repository - website - which holds the actual source code that Hugo takes to build the website. The website is then hosted with Github Pages with a custom theme at hugomartins.io.

With this new setup in mind I got tired really fast of deploying the compiled code from one repository to another and because I’m lazy I decided to automate the deployiment with a simple script called deploy-sh.

The script is very simple:

#!/bin/bash

echo "Building website..."

cd ..
hugo -t cocoa > /dev/null 2>&1
cd "public/" || exit

echo "Copying new website into other repo..."

cp -R ./* ../../caramelomartins.github.io/

echo "Push changes to remote..."

cd "../../caramelomartins.github.io/" || exit
git add -A > /dev/null 2>&1
git commit -m "Push new content" > /dev/null 2>&1
git push > /dev/null 2>&1

echo "Finished deployment..."

The only requirement basically is that the repository must be configured to use SSH otherwise the script won’t be able to push to the other repository. Also, both repositories must be inside the same parent folder.

The script simply goes to the source code directory and compiles the website. It then takes the compiled files and copies them over to the other repository. After that, it commits everything with a generic message which I’m hoping to improve and then pushes the data into the repository.

It’s a very simple script and I see lots of interesting opportunities of improvement - like registering the commit hash of the deployment in the commit message - but for now it does the work!

Generalism vs Speciality

Note: This is mostly a brain dump with no coherence whatsoever.

For most of my life I’ve been a generalist. I’ve been a jack of all trades. I know enough about a lot of different subjects that can help me fix things but I don’t have deep, or even below, the surface knowledge on most things.

There are some consequences to this:

  • Impostor syndrome. It happens a lot to me. When you feel that you are not worthy of being congratulated by your achievements because “it was just a little thing”.
  • Having trouble focusing one only one thing at a time. My mind just wanders too much.
  • Not being able to actually learn something new properly because you are only just being pulled from all places as the guy that knows a lot.

This is just draining but I really like to be a generalist. I like learning new things and I like knowing how to fix issues.

I don’t think there’s a right way to go about it, it is really just a matter of personality but I think both types of people are really important in the current state of affairs. We need people that are very specialized in something but once in a while, more frequently than not, we need people that are simply jacks of all trades and lot some things about a lot o subjects.

Both are equally interesting. Both are equally relevant.

With that said I personally thinks being a generalist fits my personality because I find interest in a lot of things and I like reading and absorbing information from a lot of different sources. I like to be able to connect different ideas from separate, distinct, areas and integrate them to build something new.

I think I’d like this to be the year in which I try to focus a little more on specific topics, just to feel like I actually can write code in a language or use a specific framework without feeling eternally like an impostor.

November 2016 - Digest

This month I have decided to take a step back and understand what I have been doing. I might be doing this from now on or simply this month but I hope I carry this one.

This is, I supposed, a really simple way of keeping myself in check and analysing what I have been doing constantly as well as keeping a record of things.

This past month was an interesting month from the point of view of starting things but I haven’t been able to complete some of them.

I’ll divide the month by the aspects in which it most focused.

Open Source

I have tried to enter, and become part of, the Open Source community for a while but this month I decided it was time to act and try, in fact, to contribute to projects.

awesome-linters

This month was an interesting month for awesome-linters. We have received about 15 PRs this month alone which, for this project, is absolutely huge. We have some awesome contributors who have given their time and resources to try and improve the curation we are doing there.

I think this might have had something to do with the fact that I have published the project on Hacker News by the beginning of November. I did this for two distinct reasons, one more selfish than the other: 1. I believe the project must have its proper audience and sharing it is the best way of finding said audience. In today’s world the customers don’t come to you, you have to find them. 2. I have come to understand that projects, as small as they are, deserve to be shared. There might always be someone out there that would like to use the project and no sharing it would be a crime against your own creative self.

I find that by sharing the project I have gained valuable contributors and the project has improved and evolved for better.

Also, in some good news, we are trying to get it into sindresorhus/awesome. More details can be found here.

cfdocs

Apart from awesome-linters I contributed a few documents to cfdocs.

As you can see if you enter the website, I’m now a part of the contributors list with 3 contributions and 35 points (whatever those points mean).

I have done this out of frustration that I couldn’t understand other people’s code but I still wanted to contribute so, after reading a bunch of articles, I decided that I could try and help out with some documentation.

Since I have been using cfdocs for some time now because of the necessity to learn ColdFusion for professional reasons, I thought it’d be nice to start with some project that I had used and helped me.

I hope to keep contributing with more documentation from time to time.

Conferences

This month I also attended my first tech conferences with mixed feelings between the two. I actually had both in the same week, one after another:

Small Projects

I found a small interesting project that I thought I would be able to complete but with my Masters’ Degree and professional work I wasn’t able but I think I’ll try next year again. The project was NaNoGenMo 2016 and the details of my, incomplete, participation can be found here

Reflections, Reflections…

All in all, this month was an interesting one from the perspective of starting the walk in the direction of what I like to do and facing some fears regarding participating in the tech community more actively.

I hope to continue this kind of work during the course of December.