dmerej's blog

Classes vs Types (or Yet Another Reason To Learn Rust)

Thu, 23 Jun 2022 10:11:47 +0000

Introduction #

Note: this is both a follow-up to the do classes suck or do classes suck articles, and yet another reason why you should learn Rust, so you should have read at least one of them in order to understand the context for this post.

Back to the spec #

As a reminder, here’s our spec:

When robots come off the factory floor, they have no name.
The first time a robot boots, a random name is generated.
Every once in a while, the robots are reset to their factory settings, and the next time they boot, they get a new random name.

Using Rust #

The goal of Rust is to allow to code to be 3 things at once: correct, fast, and expressive. This is no easy task and that’s why Rust is a bit complicated to learn (which is not such a bad thing in my opinion, but I digress).

Anyway, what’s interesting with the Rust programming language is that we can express the spec in such a way that invalid states lead to compilation errors and without using anything but structs and methods.

Here’s how. Since the specs talks about names that may or may not exist, we’re going to use two different types - one for the unnamed robots, and one for the robots that have a name. To be precise, those are called structs:

// We wrap the robot name  inside a struct
pub struct NamedRobot {
  name: String,
}

// We create a brand new type for "robots that don't have names"
pub struct UnnamedRobot;

// Note : interestingly, this struct costs *nothing* to allocate and is
// known as a zero-sized type (ZST) in Rust parlance.

Then we can express the transitions between allowed states as public methods on those types (inside a impl block) or as free functions

Note: that most of the bodies are omitted here, but you can find tho whole source code on github

// Note: this is a private implementation detail, so no
// `pub` here!
fn generate_random_name() -> String { /* ... */ }

pub fn new_robot() -> UnnamedRobot { /* ... */ }

impl UnnamedRobot {
    // Note: emit a compiler warning if users call start() without
    // using the return value
    #[must_use]
    pub fn start(self) -> NamedRobot {
      let name = generate_random_name();
      NamedRobot { name }
    }
}

impl NamedRobot {
    pub fn name(&self) -> &str { /* ... */ }

    pub fn stop(&self) { /* ... */ }

    pub fn start(&self) { /* ... */ }

    // Note: taking ownership of `self` here!
    pub fn reset(self) -> UnnamedRobot { /* ... */ }
}

Valid code compiles of course:

let robot = new_robot();
let robot = robot.start();
let name = robot.name();
println!("New robot with name: {name}");

And invalid code won’t compile. For instance:

let robot = new_robot();
let name = robot.name()
// Error: method name() not found for UnnamedRobot

You can reset a robot, restart it and get a new name:

let robot = new_robot();
let robot = robot.start();
let name1 = robot.name().to_string();

let robot = robot.reset();

let robot = robot.start();
let name2 = robot.name().to_string();
assert_ne!(name1, name2);

You may wondering why we need .to_string() here. Well, let me explain.

Owning and borrowing #

I mentioned earlier that reset takes ownership of the robot.

That’s because the method uses self as first parameter.

In contrast, the name method uses &self and borrows the robot.

This matters because in addition to having a rich type system as we saw above, the Rust compiler also enforce rules about ownership.

Here are those rules:

Each value in Rust has an owner.
There can only be one owner at a time.
At any given time, you can have either one mutable reference or any number of immutable references.

What this means is that when the robot is started, you can access the robot name any time you want, as long as you don’t try to modify it. But, once reset() is called, the value has been moved, and thus you can no longer access the robot name.

Let’s this in practice:

let robot = new_robot();
let robot = robot.start();
let name1 = robot.name().to_string();
let name2 = robot.name().to_string();
robot.reset();
let name3 = robot.name();

// Error: value `robot`  moved during called to reset reset()

If Rust had allowed this code to compile, we would have a name3 variable after the robot has been reset, which is not allowed by our spec!

Note that this also means we can’t have name1 and name2 be merely string references (&str).

That’s why we use the .to_string() method, which gives us as a mutable copy (a String) for the robot name.

Conclusion #

If you learn Rust, you’ll find that:

There’s more to Object-Oriented Programming than just classes
The borrow checker, when used well, can prevent a bunch of mistakes at compile time.

Side note: I used two different types to make a point. You will find a more idiomatic version of the code in the aptly named “idiomatic” branch on GitHub.

The again show again how the borrow checker rules prevent invalid states at compile time, this time because some methods (like start or reset) use &mut self and other just &self (like name). It also shows the “new type” pattern, in order to enforce robot name format at runtime.

Is Rust worth learning? Part 1: logs and secrets

Tue, 04 Jan 2022 10:11:47 +0000

A language that doesn’t affect the way you think about programming, is not worth knowing. – Alan J. Perlis - Yale University

To see if Rust is worth trying, then, let me tell you a story - based on real events.

Logs and secrets #

Let’s say you are writing a Java application that needs to make HTTP calls on an external API.

To do this, you have a client class that implements a call method.

public class Client {
  private void authenticate(String appSecret) {
    //
  }

  private void doRequest(String url) {
    //
  }

  public void call(String appSecret, String url) {
    authenticate(appSecret);
    doRequest(url);
  }
}

Note that you need do send an app secret along the url to make the call, hence the separate authenticate() method.

You also have a Config record to hold the configuration of the application:

public record Config(String appSecret, String url) {
}

Finally, you have a main class that reads the values from the environment, builds a Config and a Client instances and uses it to make calls:

public class App {
  public static void main(String[] args) {
    String appSecret = System.getenv("APP_SECRET");
    String url = System.getenv("API_URL");
    Config config = new Config(appSecret, url);

    Client client = new Client();
    client.call(config.appSecret(), config.url());
  }
}

Since logs are important, you also add a call to logger.info() when making the call:

public void call(String appSecret, String url) {
  authenticate(appSecret);
  logger.info("Making the call"); // <- new
  doRequest(url)
}

A vulnerability happens #

You push the code into production, and some time later you get assigned to the following bug in the issue tracker:

BUG: APP_SECRET found in the logs
SEVERITY : critical
PRIORITY : high

The logs sent by the application at startup contains the
APP_SECRET

...
INFO: config: Config[appSecret=s3cret, url=https://api.dev]
...
INFO: Making the call

So you take a look at the code base, and sure enough, you find the problem: someone from an other team added a log containing the contents of the Config class:

// SomewhereElse.java

logger.info("config:" + config.toString());

Sure, the bug is easy to fix. You can just remove the call to logger.info.

But after thinking about it more, you decide to also override the toString() method of the Config struct so that the app secret is always redacted:

public record Config(...) {
+  @Override
+  public String toString() {
+    return String.format("Config[appSecret:REDACTED, url:%s]", this.url);
+  }
}

There. You can mark the bug as “fixed” and move to something else, like take a look at this new programming language called Rust …

A few hours later, here’s what you learned.

Ownership #

First, Rust has this notion of ownership: every piece of data must have exactly one owner. By default, data is moved and can’t be used after the move.

Here’s an example:

struct StringOwner {
   inner: String, // <- note: fields are private by default
}

fn main() {
  let s = String::from("hello"); // Creates a new string

  let owner = StringOwner { inner: s }; // Move the string into
                                        // the 'StringOwner' struct

  println!("{}", s); // Compile error: cannot use the string after it's moved
}

Second, if you don’t want to move the value, you have to “borrow” it.

fn borrow_the_string(s: &String) { // < Note the '&'
   //
}

fn main() {
  let s = String::from("hello"); // Creates a new string

  borrow_the_string(&s); // Note the '&'

  println!("{}", s); // OK
}

This is a immutable (or shared) borrow: you won’t be able to modify the string.

You’ve also learned about mutable (or exclusive) borrows. They would allow you to modify the string, but they are not relevant for this story.

The trait system #

You also learned that classes don’t exist in Rust. Instead, Rust uses traits, and adds methods to structs using impl blocks:

/* in stuff.rs */

// This says that the Stuffer trait
// contains a method named do_stuff
pub trait Stuffer {
   fn do_stuff(&self);
}

struct Foo {
  // ...
}

// This says that Foo implements the Stuffer trait,
// and than compilation will fail if do_stuff() is not present
// or has not the proper signature
impl Stuffer for Foo {
  fn do_stuff(&self) {
    // Implementation here
  }
}

The trait must be in the scope where you are using it:

use stuff::Foo;

let foo = Foo::new():
foo.do_stuff(); // Error: Stuffer trait is not in scope

use stuff::Foo;
use stuff::Stuffer; // <- new

let foo = Foo::new():
foo.do_stuff(); // OK: Stuffer trait is in scope

A rewrite #

So, feeling adventurous, you decide to try and re-implement your application in Rust, just to feel like how the code would look like and if you can find a better way of handling the security issue you had to fix in Java.

You start with the Config class:

struct Config {
  url: String,
  app_secret: String,
}

Then you add the Client class:

struct Client {
  // ...
}

impl Client {
  fn authenticate(&self, app_secret: String) {
    // ...
  }

  fn do_request(&self, url: String) {
    // ...
  }

  fn call(app_secret: String, url: String) {
      self.authenticate(app_secret);
      info!("{}", "Making the call");
      self.do_request(url);
  }
}

And finally, the main() function:

fn main() {

    let app_secret = std::env::var("APP_SECRET").unwrap();
    let url = std::env::var("API_URL").unwrap();

    let config = Config { app_secret, url };

    let client = Client;
    client.call(config.app_secret, config.url);
}

“Well, that was easy” you think. “I wonder what people mean when they’re talking about ‘fighting the borrow checker’”.

Then you try to use client.call a second time:

fn main() {

    let client = Client;
    client.call(config.app_secret, config.url);
    // ...
    client.call(config.app_secret, config.url); // <- new
}

error[E0382]: use of moved value: `config.app_secret`
  --> src/main.rs:27:17
   |
23 |     client.call(config.app_secret, config.url);
   |                 ----------------- value moved here
...
27 |     client.call(config.app_secret, config.url);
   |                 ^^^^^^^^^^^^^^^^^ value used here after move

“Oh, right”, I need to “borrow” the app_secret and the url in the client if I want to be able to keep using the config struct.

So you change the code to be like this instead:

  impl Client {
-     fn call(&self, app_secret: String, url: String) {
+     fn call(&self, app_secret: &str, url: &str) {
          //
      }

  }

-    client.call(config.app_secret, config.url);
+    client.call(&config.app_secret, &config.url);

There. Now the client borrows the app secret and the url and the code compiles and runs fine.

Feeling confident, you try and reproduce the logging issue.

First, you add a #[derive(Debug)] annotation on top of the Config struct:

+ #[derive(Debug)]
  struct Config {

  }

And then you add a log displaying the contents of the config struct:

  fn main() {
    let config = Config { app_secret, url };
+   info!("config: {:?}", &config);
   }
}

This code works because Rust knows how to print debug representations of strings and booleans and the derive(Debug) annotation can automatically generates the code to print a debug representation of the Config struct.

Indeed, the code compile, and, sure enough, the secret shows up in the log:

[INFO] config: Config { app_secret: "s3cret", url: "https://api.dev" }
[INFO] Making the call to https://api.dev

OK, you’ve reproduced the problem. But can you find a better solution this time?

Time to think #

You’ve heard good things about Rust type system, so you wonder: “Would it be possible to solve the problem by adding a new type?”.

You start with a SecretString struct:

struct SecretString {
    inner: String,
}

impl SecretString {
    fn new(inner: String) -> Self {
        Self { inner }
    }
}

The constructor takes ownership of the string, which is a good thing, because it means the inner value can no longer be accessed once the SecretString struct is created:

let secret_value = std::env::var("APP_SECRET").unwrap();
let app_secret  = SecretString::new(secret_value):

dbg!(secret_value); // < does not compile!

Then you change the type of app_secret in Config from String to SecretString:

struct Config {
-    app_secret: String,
+    app_secret: SecretString,
}

Which means you have to get the inner value when authenticating the client:

impl Client {
-   fn call(&self, app_secret: &str, url: &str) {
-   self.authenticate(&app_secret);
+   fn call(&self, app_secret: &SecretString, url: &str) {
+      self.authenticate(app_secret.inner);
+   }

}

You are then faced with 2 compile errors:

First, the Debug macro no longer works:

error[E0277]: `SecretString` doesn't implement `std::fmt::Debug`
  --> src/main.rs:26:5
   |
24 | #[derive(Debug)]
   |          ----- in this derive macro expansion
25 | struct Config {
26 |     app_secret: SecretString,
   |     ^^^^^^^^^^^^^^^^^^^^^^^^ `SecretString` cannot be formatted using `{:?}`
   |
   = help: the trait `std::fmt::Debug` is not implemented for `SecretString`

To fix this, you implement the debug trait yourself:

use std::fmt::Debug;
// ^-- new - remember: traits needs to be in scope
// in order to use them


impl Debug for SecretString {
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
        write!(f, "REDACTED")
    }
}

Second, you can’t use the inner field directly in the call() method:

error[E0616]: field `inner` of struct `SecretString` is private
  --> src/client.rs:13:39
   |
13 |         let secret_value = app_secret.inner;
   |                                       ^^^^^ private field

So you add a public method to access the inner value:

impl SecretString {
    pub fn expose_value(&self) -> &str {
        &self.inner
    }
}

And you fix the client code:

pub fn call(&self, app_secret: &SecretString, url: &str) {
    let secret_value = app_secret.expose_value(); // <- new!
    self.authenticate(&secret_value);
}

Going further #

Then you think back about the rule about traits having to be in scope, and you decide to move the expose_value method into a trait:

pub trait ExposeSecret {
    fn expose_value(&self) -> &str;
}

impl ExposeSecret for SecretString {
// ^--  used to be `impl SecretString`

    fn expose_value(&self) -> &str {
        &self.inner
    }
}

And then the compiler says:

error[E0599]: no method named `expose_value` found for reference
`&SecretString` in the current scope
  --> src/client.rs:13:39
   |
13 |         let secret_value = app_secret.expose_value();
   |                                       ^^^^^^^^^^^^

   = help: items from traits can only be used if the trait is in scope
   = help: the following trait is implemented but not in scope; perhaps add a
           `use` for it:

use crate::secrets::ExposeSecret;

So you comply:

use crate::secrets::ExposeSecret; // <- new!
use crate::secrecy::SecretString;

There - the code compiles, and it’s now pretty hard to leak the app secret:

Indeed, if a value has been wrapped in the SecretString struct, anyone attempting to access it must:

call a method that sounds dangerous (expose_value())
import a trait that also sounds dangerous (secrets::ExposeSecret)

It’s also very easy to edit the code for safety: just list the usages of the ExposeSecret trait.

Conclusion #

I hope you found this post interesting: it shows how you can use some unique features of the Rust programming language to tackle an old issue in a novel way.

I mentioned at the beginning that the this post is based on true stories, so here are my two sources of inspiration:

The issue of a secret value exposed by mistake is based on an actual security vulnerability I reported to the Miniflux maintainers. You can see the details in the miniflux repository.
The SecretString struct is based on a real crate, called secrecy.

Thanks for taking this journey with me, and if you’re a reading this while trying to fix the log4j security vulnerability, you have my sincere sympathy :)

Cheers!

Merge first, ask questions later - Optimistic Merging in practice

Tue, 14 Sep 2021 14:06:46 +0000

Introduction - Pessimistic Merging #

If you’ve contributed to an open-source project on GitHub or GitLab, or written code for a company, you’re probably familiar with the concept of “Pessimistic Merging”. You write a patch (or a pull request or a merge request), but it is not merged (or applied) right away. Instead, you need to wait for human approval to be given and/or for continuous integration to pass.

I can think of two reasons why this strategy is often the one which is used:

It’s been used by really big projects (like the Linux kernel) for quite some time
It’s kind of the default way to collaborate on platforms like GitHub or GitLab

There are even tools like Gerrit which work by specifying a list of constraints before a patch can be merged ¹ - which is used among other things by Google’s Android Open Source Project or the Qt framework.

There’s also the fact that it allows to enforce some rules. Used in combination with source control software (like Git), Pessimistic Merging can ensure:

A “clean” git history (from the commit messages to their contents to the size of the patches)
A consistent coding style
A test suite that always pass on the main development branch
A code that is composed only of patches that have been reviewed and approved by an arbitrary number of people
… and so on

And indeed, there are some projects where this matters a lot - in a corporate environment, or in a “sensitive” project like a browser, a database engine or an operating system …

That being said, we rarely talk about the downsides of Pessimistic Merging - which is why you probably never thought that this concept needed a name.

Issues with Pessimistic Merging #

For this, I’ll let you read the article which prompted me to write this one: Why Optimistic Merging Works Better ².

The idea behind Optimistic Merging is very simple: you merge the patch first, and then deal with issues like broken builds, code style violation and other problems.

Sure, this means that the main development branch of your Git repository may now contain commits that do not fully work, code that does not always compile, or commits that only fix code styling errors.

But for a small or medium-sized project, that does nothing too security-sensitive, and for which maintainers and collaborators are hard to find - is it such a bad deal?

On this topic, see also Aleksey Kladov’s piece on Two Kinds of Code Review (which is the reason I learned about Optimistic Merging in the first place and wanted to try it out), where he explains how pull requests are reviewed and merged for the rust-analyzer project.

A note before I continue: they are plenty of cases where using Pessimistic Merging is the best strategy - here I’m just describing an alternate strategy you may find interesting.

With that out of the way, let’s now look how Optimistic Merging works for me in practice.

My workflow for Optimistic Merging #

Setting the stage #

I’m the maintainer of a project called “foo”, written in Python, and hosted in GitHub.

The project uses flake8 as a linter to catch small mistakes, the code is formatted with black, and it has has some tests (of course). It is also very easy to run the linters, the tests and the code formatter from the developer machine (this will matter later on).

Here comes a developer I never heard of (let’s call them Mallory) - and they open their first ever merge request to the GitHub repository.

Merging Mallory’s changes #

The first step is for me to get the proposed changes on my machine and to prepare a list of notes for later feedback - a simple text file is enough.

Since it’s a PR on GitHub, I can do something like this, where “some-branch” is the source ref of Mallory’s PR:

$ git remote add mallory git@github.com:mallory/foo
$ git fetch mallory
From github.com:mallory/foo
* [new branch] some-branch -> mallory/some-branch

Then I rebase some-branch on top of the main branch. This allows me to make sure the commits can be merged without conflicts.

During the rebase I look at each commit diff and I take notes.

Since everything is on my machine, it’s trivial to amend Mallory’s commits if a linter complains or to run black if I need to.

I can also choose to run the tests locally if I fear a commit may have caused a regression.

Finally, if there’s a small issue (like a misleading comment) I can fix it right away! There’s no need to go through the hassle of:

telling Mallory they did something wrong,
waiting for them to update the merge request,
re-starting the process from scratch.

When all of this is done, it’s time to push Mallory’s branch. I re-run the linters and the tests just to be sure, then I push the code and leave a message in the PR discussion telling Mallory the changes have been merged.

Looking at the interaction from Mallory’s point of view #

It’s very likely that Mallory was not paid to write the patch, (this is a medium-size open-source project, after all) but they still worked on it and thought their work was good enough to open a pull request.

They probably don’t know much about the project, and it’s likely they don’t know everything about Git, Python, flake8 or black. And it’s even possible they know nothing about testing!

Yet, the first reply they got for the maintainer is “Thanks! Your commits have been merged”. Boom - the collaboration is already working - what a great start!

Contrast with a typical response when using Pessimistic Merging, which would look like this:

Thank you for your contribution, but the CI does not pass (test_foo_bar4 is failing and there are some lint errors). Also, please merge or rebase your work on top of the main branch so that the merge is fast-forward.

… followed by a bunch of nitpicks about the merge request.

After the merge #

Anyway, now it’s time to put my notes to good use.

If there are some trivial changes I had to do, I can just tell Mallory:

It was a good idea to refactor the do_stuff() method inside the Foo class, but you did not adapt the doc string, which I’ve done in commit …

Again, we’re working together here, and I’m willing to bet Mallory is more likely to watch their docstrings next time they write a new patch :)

If there was some non-trivial changes (like a missed opportunity for a refactoring), I can just create an issue with a task list for instance.

I can also start a pull request and have Mallory review it.

Conclusion #

To be frank, I’ve just started implementing this strategy for my open-source projects, so it’s a bit early to draw any conclusions. That being said, so far it has been going great, and I wanted to share the idea of Optimistic Merging right way.

I’ll let you know how it goes in the future. Until then, happy coding!

Fun fact: to do this, Gerrit embeds an implementation of the Prolog Language ↩︎
If you have trouble with opening this link, make sure you are using the http:// URL, the https:// version is unfortunately broken. ↩︎

Two awesome Rust warnings

Tue, 02 Feb 2021 13:35:21 +0000

Introduction #

If you’ve ever used a compiler (or any other tool) that can produce warnings about your code, you may be used to messages that are hard to understand, confusing or just plain wrong.

But today I want to show you that good compiler warnings do exist and how they can make your code more correct.

The incorrect server API #

The first example comes from code I wrote to test the Rust implementation of the Tanker client SDK. You don’t need to know precisely how Tanker works to follow this example - let’s just say the client needs a “Tanker permanent identity” from an identity server to start a session.

In my case, the identity server was already written in Go using the Gin-Gonic framework.

Here’s what the implementation of the sign_in route looked like:

// This struct is used by various Gin handlers
type UserResponse struct {
	UserID              string `json:"user_id"`
	Email               string `json:"email"`
	PermanentIdentity   string `json:"tanker_permanent_identity"`
}


// handler for the /sign_in route
func (this *Server) signIn(c *gin.Context) {
	var body SignInBody
	jsonErr := c.BindJSON(&body)
	if jsonErr != nil {
		c.JSON(400, gin.H{"error": jsonErr.Error()})
		return
	}

	email := body.Email
	password := body.Password

	// Not shown: check email and password, fetch user from the db

	userResponse := UserResponse{
		Email:               user.Email,
		UserID:              user.UserID,
		PermanentIdentity:   user.PermanentIdentity,
	}

	c.JSON(200, userResponse)
}

To parse the JSON response from the server in the Rust client, I used serde, like this:

#[derive(Deserialize)]
struct UserResponse {
    user_id: String,
    email: String,
    tanker_permanent_identity: String,
}

impl Client {
    pub async fn sign_in(&mut self) -> Result<()> {
        let email = ...;
        let password = ...;

        let mut data = HashMap::new();
        data.insert(String::from("email"), email);
        data.insert(String::from("password"), password);

        let response = self.client.post("/sign_in", data).await?;
        let user_response = response.json::<UserResponse>().await?;

        self.start_tanker(&user_response).await?;
    }

    async fn start_tanker(&self, user_response: &UserResponse) -> Result<()> {
        let private_identity = &user_response.tanker_permanent_identity;
        let email = &user_response.email;
        let status = self.tanker.start(&private_identity).await?;
        // Not shown: verify identity by email when required

        Ok(())
   }
}

When I first compiled the code, I got this warning:

Warning: field is never read: `user_id`
  --> src/notepad.rs:24:5
   |
24 |     user_id: String,
   |     ^^^^^^^^^^^^^^^
   |
   = note: `#[warn(dead_code)]` on by default

This made sense: as you can see, to start a Tanker session we only need the email and tanker_permanent_identity fields of the JSON response returned by the server. In particular, we don’t need the user ID at all.

What’s interesting with this example is that at Tanker, we already wrote six other clients for the server in question (using Python, Go, Java, JavaScript, Objective-C and Ruby) and that was the first time we got an indication that our JSON API was wasting bandwidth by sending information that was actually not needed!

The useless mutation #

The second example comes from a personal project - a week ago I started implementing an interpreter in Rust, following the instructions from the Writing an Interpreter in Go, by Throsten Ball.

As I worked on the the lexer, I wrote the following code:

use std::iter::Peekable;
use std::str::CharIndices;

#[derive(Debug)]
pub(crate) struct Lexer<'a> {
    code: &'a str,
    iter: Peekable<CharIndices<'a>>,
}

impl<'a> Lexer<'a> {
    pub fn new(code: &'a str) -> Self {
        Self {
            iter: code.char_indices().peekable(),
            code,
        }
    }

    // ...

    /// Try to read an identifier starting at start_pos
    fn read_identifier(&mut self, start_pos: usize) -> &'a str {
        // end_pos is either `None` if we reached the end of the input code,
        // or Some(index) if we reached a character that is not part of
        // of a valid identifier

        let mut end_pos = Some(start_pos);
        loop {
            // Is the next character valid?
            let peek = self.iter.peek();
            match peek {
                Some((next_pos, next_c)) if Self::is_valid_ident(*next_c) => {
                    // Yes: advance next_pos
                    end_pos = Some(*next_pos);
                    self.iter.next();
                }
                Some((next_pos, _)) => {
                    // No: advance next_pos and exit the loop
                    end_pos = Some(*next_pos);
                    break;
                }
                None => {
                    // There's no next character : we've reached the end
                    // of the input code
                    end_pos = None;
                    break;
                }
            }
        }

        match end_pos {
            // end_pos is None: return all the code from start_pos to the end
            None => &self.code[start_pos..],
            // end_pos is not None: return the code from start_pos to end_pos
            Some(index) => &self.code[start_pos..index],
        }
    }
}

Here’s what the compiler had to say:

warning: value assigned to `end_pos` is never read
   --> src/lexer.rs:131:13
    |
131 |         let mut end_pos = Some(start_pos);
    |             ^^^^^^^^^^^
    |
    = note: `#[warn(unused_assignments)]` on by default
    = help: maybe it is overwritten before being read?

warning: value assigned to `end_pos` is never read
   --> src/lexer.rs:136:21
    |
136 |                     end_pos = Some(*next_pos);
    |                     ^^^^^^^
    |
    = help: maybe it is overwritten before being read?

The second warning was the easiest to fix: we keep setting end_pos to Some(*next_pos) but never read it until the loop exits. So we can just remove the line that sets end_pos before calling self.iter.next():

    fn read_identifier(&mut self, start_pos: usize) -> &'a str {
        let mut end_pos = Some(start_pos);

        loop {
              match peek {
-                 Some((next_pos, next_c)) if Self::is_valid_ident(*next_c) => {
-                     end_pos = Some(*next_pos);
+                 Some((_, next_c)) if Self::is_valid_ident(*next_c) => {
                      self.iter.next();
                  }
                  Some((next_pos, _)) => {
                      end_pos = Some(*next_pos);
                      break;
                  }
                  None => {
                      end_pos = None;
                      break;
                  }
              }
        }

But the compiler was still unhappy:

warning: value assigned to `end_pos` is never read
   --> src/lexer.rs:131:13
    |
131 |         let mut end_pos = Some(start_pos);
    |             ^^^^^^^^^^^
    |
    = note: `#[warn(unused_assignments)]` on by default
    = help: maybe it is overwritten before being read?

This took me a while to realize, but the compiler is telling me that I can declare the end_pos binding but not initialize it right away:

    fn read_identifier(&mut self, start_pos: usize) -> &'a str {
-       let mut end_pos = Some(start_pos);
+       let end_pos;
        loop {
              match peek {
                  Some((_, next_c)) if Self::is_valid_ident(*next_c) => {
                      end_pos = Some(*next_pos);
                      self.iter.next();
                  }
                  Some((next_pos, _)) => {
                      end_pos = Some(*next_pos);
                      break;
                  }
                  None => {
                      end_pos = None;
                      break;
                  }
              }
        }

I love this : the end_pos binding is no longer mut, and can only be assign a value in two cases: either in the None arm of the match peek block, or to end_pos when the peek character is not valid.

It’s much less likely for the code to be buggy!

Conclusion #

I hope those two examples gave you an idea of what it’s like to write code using a language that has such a focus on correctness and how nice it is to have useful warnings.

For now, Rust is the only language I’ve used which gave me this feeling of “the compiler has my back”, but I’m sure they are others - please tell me bellow if you know one.

And if this gave you a motivation to try and learn Rust, go for it!

dmerej.info: now on the Gemini space

Thu, 19 Nov 2020 16:10:22 +0000

Hi there,

Just a quick note to let you know my personal website is now available on the Gemini space too.

Get a Gemini client and see you at gemini://dmerej.info

But Why?

1. Because I can #

Because I can! - classic geeky answer I know.

I’ve been hosting my own server on the Web for quite some time, and it’s much, much easier to set up a Gemini server. So, feel free to give it a go yourself.

2. Because of what Gemini stands for #

The second reason is that I like what Gemini stands for.

As per the FAQ:

Gemini may be of interest to people who are:

Opposed to the web’s ubiquitous user tracking

Tired of obnoxious adverts, auto-playing videos and other misfeatures

Interested in low-power computing and/or low-speed networks

I’m interested in the first two for philosophical and political reasons. I care a bit less about the last one, but it’s a nice touch (think of the environmental reasons).

3. To produce more content #

To cite the FAQ:

Gemini already has a surprising number of client and server implementations in existence […], but the real shortage right now is not of software but of content […] The greatest contribution you can make to the project is to be a part of this process.

So, enjoy about 100 articles spanning 4 years!

"It's a waste of time!"

Sat, 30 May 2020 10:18:46 +0000

I have a confession to make: every time I heard the phrase “It’s a waste of time!”, I get a little angry.

I’ve heard this answer countless times, often after suggestions like:

“Let’s write tests before writing production code.”
“Let’s write documentation before writing code.”
“Let’s refactor before implementing the new feature.”
“Let’s discuss the API once more before shipping the next release.”

There are several reasons why I don’t like this answer.

Why? It’s all about knowledge.

Time spent vs. time saved #

First, it’s easy to see the time spent.

Yes, we spent one week writing and reviewing the new documentation. Yes, we spent three days refactoring the code, and we did not implement any feature. Yes, we had two one-hour meetings before getting a consensus on the new API and shipping the next release.

But it’s tough to see the time saved.

Who knows what horrible bug we would have to fix in production if we skipped writing those tests? Who knows how much breaking changes we would have made because the API was not good enough? Who knows how much time we would have spent implementing the next big feature if we skipped the refactoring?

Learning about the problem #

Second, you may have noticed that in all the situations I mentioned above, it’s all about gaining more knowledge about the problem before jumping to the implementation.

In other words, by saying: “It’s a waste of time,” you assume you know enough about the problem to start working on the implementation right away.

Trying to convince others #

Most of the time, I can’t convince people who tell me that I’m suggesting is a waste of time.

Instead, they skip what I think was a critical preparatory step, and then I watch them spend a ton of time rectifying the situation afterward - it’s frustrating.

Fortunately, after they’ve repeated the same mistake enough times, they tend to listen to me at last. But I’d very much like not to have to go through this painful process every time.

I’m not sure what to do about it, though. Maybe it’s just human nature?

To make matters worse, sometimes writing the code is a good way to gain this precious knowledge in a way that neither tests nor documentation nor brainstorms could have done.

Questions for you #

What do you think about all this? How do you react when confronted with the “It’ a waste of time!” argument?

Feel free to share your ideas below, and see you next time!

Tips for better Python tests

Sat, 25 Apr 2020 14:20:47 +0000

Today, I’d like to share with you a list of small tips to help you write better tests when using Python. Note that the tips often improve both the readability of the test implementation, and of the failure messages (which is pretty important too).

Use pytest #

Pytest does a great job in generating very good error messages from your assertions, without the need for anything other than the assert statement.

Take this assertion for instance:

assert foo(x) == bar(y)

Here’s what the failure will look like when using pytest if the assertion fails:

    def test_foobar():
        x = 2
>       assert foo(x) == bar(x)
E       assert 4 == 5
E        +  where 4 = foo(2)
E        +  and   5 = bar(2)

test.py:11: AssertionError

Note how much information you get about what went wrong, like the entire body of the function up to the assertion that failed, or when the values that were compared came from.

It gets even better - take this other assertion:

assert get_message() == "this is what I expected"

And note the nice, detailed diff between the two strings that gets printed:

    def test_get_message():
>       assert get_message() == "this is what I expected"
E       AssertionError: assert 'this is what I expected' == 'this is what I got'
E         - this is what I expected
E         ?                ^^^^^ --
E         + this is what I got
E         ?                ^^^

A more complex example #

For the rest of this post, let’s assume you are testing a sync_folders() function that can synchronize a remote folder with a local one.

Here’s one of the tests you wrote:

def test_can_update_local_file(remote, local):
    local_file  = local / "a.txt"
    local_file.write_text("old contents")
    new_contents = "new contents"
    remote.add_file("a.txt", contents=new_contents)

    sync_folders(remote, local)

    actual_contents = local_file.read_text()
    assert actual_contents == new_contents

Quick tip: I usually split my tests into three arrange /act / assert parts and I visualize them using vertical space.

Add context to assertions #

Did you know you can add a string at the end of the assert statement?

def test_sync(remote, local):
    ...

    actual_contents = local_file.read_text()
-   assert actual_contents == new_contents"
+   assert actual_contents == new_contents, "a.txt should have been updated"

This way, instead of having to read this:

E       AssertionError: assert 'old contents\n' == 'new contents'
E         - old contents
E         + new contents

you get this message, which contains a clue about where the diff actually comes from:

E       AssertionError: a.txt should have been updated
E       assert 'old contents\n' == 'new contents'
E         - old contents
E         + new contents

Reduce noise #

You can also improve the signal over noise ratio by using pytest.fail instead of the assert statement:

-  assert actual_contents == new_contents", "a.txt should have been updated"
+  if actual_contents != new_contents:
+      pytest.fail("a.txt should have been updated")

        if actual_contents != new_contents:
>           pytest.fail("a.txt should have been updated")
E           Failed: a.txt should have been updated```

Be careful when using this technique, because it may hinder the debugging of failing tests.

Use custom assertion helpers #

Finally, don’t hesitate to factorize code about assertions, for instance in a test/helpers.py file:

def assert_was_updated(path, contents):
    ...

def assert_was_created(path):
    ...

# and so on

Use docstrings to describe tests scenarios #

This is my favorite tip ever: if you are testing something complex, add a human-readable description of the test inside the docstring.

Still using our test_can_update_local_file example:

def test_can_update_local_file():
    """ Scenario:
    * Create a file named a.txt in the local folder with
      "old" contents
    * Add a new version of the `a.txt` file in the remote folder
    * Synchronize the remote and local folders
    * Check that `a.txt` has been updated
    """
    ....

There are two advantages to this approach:

It can be helpful to have a human-readable description of what the test is supposed to be doing when reading the implementation of the code - like any docstring
Since pytest displays the entire block of the function block that caused the assertion to fail, you get a reminder of what the test was about when reading the failure message.

Reflections on the last tip #

You can stop reading there if you want, but I thought it would be interesting to know how I end up using docstrings in my test code - especially since for a long time, I was convinced that docstrings were useless if the test implementation was clear enough¹!

What changed my mind? In two words: code review. let me elaborate.

Getting feedback #

I’ve had the chance to get my Python test code reviewed by some teammates who did not know pytest very well but were used to frameworks like Mocha or Cucumber. They help me realize this simple truth: using only function names and implementation (in other words, code) to express all the subtlety of what the tests are about cannot be enough - kind of obvious when you say it like that, right?

But in this case code review can only see you what needs to be improved, but not always how.

So I did what I had to: I took a closer look at those other frameworks.

Getting to know other frameworks #

Here’s an implementation of our test using Mocha:

describe('sync', function() {
  it('syncs a remote file', function() {
    remote.addFile('a.txt', { contents: newContents });

    syncFolders(remote, local);

    const actualContents = local.join('a.txt').readText();
    assert.equal(actualContents, newContents);
  });
});

And here’s an implementation using Cucumber:

# in synchronization.feature - the syntax is called Gherkin
Feature: Synchronization

Scenario: file updated remotely
  Given there is a local file 'a.txt' containing "old_contents"
  Given there is a remote file 'a.txt' containing "new_contents"
  When I synchronize the folders
  Then the local file a.txt contains "new_contents"

# in synchronization.rb
Given(/there is a local file '{word}' containing "{string}"/ do |path, contents|
  open(path, 'w') do |f|
      f.puts contents
  end
end

When(/I synchronize the folder/) do
  sync_folders(@local, @remote)
end

# ...

Quite different styles, right?

Finding a middle ground #

And there you have it: I came up with using docstrings with pytest because it was a nice middle ground between those two approaches.

With Mocha, you write short descriptions in it() and describe() and there’s no such thing as a “test name”.
With Cucumber, you write long and detailed text using the English natural languages.

I prefer the docstring solution to the ones above because:

docstrings can be as long as you want (you rarely see descriptions larger than one line in Mocha tests)
contrary to Cucumber, you are not forced to use any formatting or syntax in your description, and they live right next to the accompanying code

Conclusion #

So what did we learn?

Being reviewed helps to keep your code readable - but you already knew that, right?
Exploring new languages and frameworks gives you insights on how to improve your code
If you are in a team that contains people who use different languages and frameworks than yours and you get them to review your code, it will lead to even better code because you’ll be combining the two effects from above!

In other terms, consider increasing the diversity of your teams, and don’t hesitate to explore new things 😎.

I also believed good code did not need comments - fortunately, I read this article. ↩︎

symlinks and .so files on linux - what you need to know

Sat, 18 Apr 2020 14:03:19 +0000

The issue #

I’ve seen this happen countless times.

First, a Linux user asks for help about an error looking like this:

Error when running `bar`: `libfoo.so.5: no such file or directory`

Then, someone who’s had the same issue suggests creating a symlink from libfoo.so.6 to libfoo.so.5.

And then I come across the dialog some time later and I start crying.

Why do I cry - it looks like the problem is solved, right?

the bar program seems to run just fine;
the change was made with just one command (sudo ln) and can be undone easily (just remove the symlink);
there was already a symlink from libfoo.so to libfoo.so.6 anyway!

Well, despite the appearances, the problem is not solved, and doing this is a terrible idea in general - like planting a ticking time bomb in the street you live or shooting yourself in the foot because you’ve got a plantar wart.

But to understand why we need to talk about the language C, shared libraries, soname bumps, Linux distributions, and package management.

And because I love telling stories, you, dear reader, will be the hero in this one.

Solving the Ultimate Question #

Let’s assume a group of people you don’t really know (let’s called them The Experts), wrote a piece of C code than can get the answer to the Ultimate Question of Life, the Universe, and Everything.

For obvious reasons, they want to keep the source code private, so here’s what they did:

They wrote a header file containing the get_answer declaration named answer.h ¹:

// in answer.h
#pragma once
#ifdef __cplusplus
extern "C" {
#endif

char* get_answer();

#ifdef __cplusplus
}
#endif

They created a shared library called libanswer.so from their source file (answer.c in this case):

$ gcc -shared libanswer.so answer.c

That way, everyone who needs to get the answer to the Ultimate Question of Life, the Universe, and Everything can buy the libanswer.so compiled library and the answer.h header and call the get_answer() function - let’s see how.

Using the library from The Experts #

You’ve bought the libanswer.so and answer.h files from The Experts and have put them next to a file you’ve wrote named print-answer.c:

// in print-answer.c
#include <stdio.h>
#include <answer.h>

int main() {
  char* answer = get_answer();
  printf("The answer is %s\n", answer);
  return 0;
}

You were told that gcc can compile C code and link against shared libraries if you put them on the command line, so you try and run this:

$ gcc libanswer.so print-answer.c -o print-answer

But it does not work, and you get the following error message:

print-answer.c:2:10: fatal error: answer.h: No such file or directory

Wait a minute - the answer.h file is right there - what does that mean, “No such file or directory”?.

After a bit of research, you discover that gcc uses a list of paths called the “include path” where it looks for headers. You check on your machine, and sure enough, /usr/include/ is one of the elements of this list, and stdio.h is in /usr/include/stdio.h, which explains why gcc did not complain about the first include.

To fix the compilation error, you add the -I . option to the command line so that current directory is added to the list of include paths:

$ gcc -I . libanswer.so print-answer.c -o print-answer

And then it compiles.

Now you try and run the print-answer program, but you get a new error message:

$ ./print-answer
 error while loading shared libraries: libanswer.so
 cannot open shared object file: No such file or directory

It’s the same error message as in the introduction and the operating system is lying to us. The file libanswer.so is right there! What’s happening there?

After more investigation, you figure it out: when you compiled the print-answer executable, there was a small piece of binary inside it that recorded the name of the .so file it was linked against. You can check it by running the readelf command and display the dynamic section of your program:

$ readelf -d ./print-answer
Dynamic section at offset 0x2de8 contains 27 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libanswer.so]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 ...

In a way, the print-answer program “knows” that it needs libanswer.so to run. Crucially, it does not know nor cares about where libanswer.so really is. ²

Then, when you run ./print-answer, the operating system sees the name of the shared library in the dynamic section and tries to locate it. Like gcc, it finds libc.so.6 by itself (in /usr/lib/libc.so.6 for instance) - but it’s unable to find the libanswer.so shared library in the current directory.

Fortunately, you can fix that by using a special environment variable called LD_LIBRARY_PATH:

$ LD_LIBRARY_PATH=. ./print-answer
The answer is 42

This time, the operating system looks for libanswer.so in the current directory, finds it, and when required, invokes the code for the get_answer() function from the shared library.

That gets you thinking - you did not have to do any of this for print-answer to find the libc.so library and the stdio.h header.

What if it was possible to compile libanswer.so once and for all?
And what if there was a way to compile the print-answer.c source file without having to copy/paste the header and the shared library, and remember all the various gcc options?

Becoming a packager #

Let’s assume The Experts realized that their business model was not going to work and decided to publish their source files for free instead. Yay open source!

Here’s the contents of their answer.c file:

#include <answer.h>
#include <stdio.h>
#include <string.h>

char * get_answer() {
  // Disappointing, I know
  int r = 6 * 7;
  char buf[3];
  snprintf(buf, 3, "%d", r);
  return strdup(buf);
}

Since you are an Arch Linux user, you decide to lookup documentation about the pacman package manager and how generate and publish Arch Linux packages ³.

After a while, you manage to write a working PGKBUILD for libanswer:

pkgname=libanswer
pkgver=1.0
pkgrel=1
pkgdesc="Answer the Ultimate Question"
arch=('x86_64')
...

build() {
  gcc -I . -shared answer.c -o libanswer.so
}

package ()
{
  mkdir -p $pkgdir/usr/{include,lib}
  install answer.h $pkgdir/usr/include/
  install libanswer.so $pkgdir/usr/lib
}

Then you build the package with makepkg

$ makepkg
==> Making package: libanswer 1.0-1
...
==> Finished making: libanswer 1.0-1

Everything goes well, and you now have a file named libanswer-1.0-1-x86_64.pkg.tar.xz next to your PGKBUILD.

You install the package with pacman:

$ sudo pacman -U libanswer-1.0-1-x86_64.pkg.tar.xz

Then you find out that you can compile and run print-answer.c from anywhere in the system using the following commands:

$ gcc print-answer.c -lanswer -o ./print-answer
$ ./print-answer
The answer is 42

This time you need only the name of the library (the part without the lib prefix and the .so suffix) after the -l option. You do not have to worry about include paths or use the LD_LIBRARY_PATH environment variable - neat!

What you’ve accomplished is called packaging the answer library, and it’s what package maintainers do - well done.

Impressed by your packaging skills⁴, the Arch Linux maintainers allow you to publish your package in official repositories, which means everyone using Arch Linux is now able to install the libanswer package in just one command!

A simple change #

A few days later, The Experts release a new version of their library (1.1), containing a nice optimization:

char* get_answer() {
  return strdup("42");
}

Since you are the maintainer of the libanswer package, you quickly update the PKGBUILD and publish a new release.

-pkgver=1.0
+pkgver=1.1
 pkgrel=1
 pkgdesc="Answers the Ultimate Question"
 arch=('x86_64')
 source=(...)

You build and install the package on your machine and check it still works:

$ makepkg
$ sudo pacman -U libanswer-1.1-1-x86_64.pkg.tar.xz

And then you publish the new version of the libanswer package.

That’s where Linux distributions really shine (and the whole reason we use shared libraries in the first place). Once the new package is published, any Arch Linux user who installs it will get the latest version of libanswer.so in their system, and all the programs that were linked against it will use the latest version - this is especially important if the new version contains a security bug fix for instance.

Another change #

A week later, The Experts realize they don’t really need to return a string from the get_answer() function, and that a simple int would suffice.

So they modify both their header and source files:

- char* get_answer();
+ int get_answer();

#include <answer.h>

int get_answer() {
  return 42;
}

And they publish a release note:

# The answer project

## Version 2.0

* Change the return type of the `get_answer()` function from `char*` to `int`.

## Version 1.1

* This release implements performance optimizations.

## Version 1.0

* First public release!

Upon hearing the good news, you download the latest sources of the project, and you modify your print-answer.c source file to use the latest version of the library:

#include <stdio.h>
#include <answer.h>

int main() {
  int answer = get_answer();
  printf("The answer is %d\n", answer);
}

You compile everything and check the code still works:

$ gcc -I . -shared answer.c -o libanswer.so
$ gcc -I . libanswer.so print-answer.c -o ./print-answer
$ LD_LIBRARY_PATH=. ./print-answer
The answer is 42

Time to publish the v2!

-pkgver=1.0
+pkgver=2.0
 pkgrel=1
 pkgdesc="Answers the Ultimate Question"
 arch=('x86_64')
 source=(...)

$ makepkg
$ sudo pacman -U libanswer-2.0-1-x86_64.pkg.tar.xz

Easy as pie - satisfied, you publish the new package and go to bed.

When shit hits the fan #

The next morning you receive the following e-mail:

Subject : latest libanswer package update broke display-answer-pp program

Hello,

I'm using `display-answer-pp` version 0.4, and when I updated `libanswer` to
the version 2.0, I got the following error:

./display-answer-pp
zsh: segmentation fault (core dumped)  ./display-answer-pp

Downgrading the `libanswer` package fixes the problem. Please advise.

Signed: Bob

Welcome to the joys of packaging!

You’ve never heard of the display-answer-pp package - what is going on?

After a bit of research, you find out that someone wrote a display-answer-pp program using your libanswer package and published it on the official Arch Linux repositories a few days ago.

Here’s what the code for display-answer-pp looks like - it’s a single C++ file:

#include <answer.h>
#include <iostream>

int main() {
    auto answer = get_answer();
    std::cout << "The answer is: " << answer << std::endl;
    return 0;
}

So what happened?

Well, the problem is that you forgot to coordinate with your fellow packagers!

You see, when display-answer-pp was being packaged, the get_answer() function was returning a char*. When you published answer version 2.0, the code for the function get_answer() started returning an int instead.

So when Bob updated libanswer after having installed display-answer-pp, and tried to re-run the program, all hell break loose, because the compiled C++ code expected a pointer to a string and got an int instead.

In other terms, the application binary interface (or ABI for short) of the libanswer library broke.

You break it, you fix it #

Unfortunately, there’s only one way to fix an ABI breakage: you need to recompile everything that was linked against the old version of the library.

That’s one of the main issues package maintainers have to solve. They need two very different features when it comes to libraries updates:

If the new version of the library is ABI-compatible with the previous one, end-users should be able to get it by simply updating the one package that contains it.
If not, they need to make sure none of the programs that depend on the library break when the update is made.

The Arch Way #

Here’s how Arch maintainers solve this problem. In our example, they would have published libanswer v2 in a special repository named “staging”. Then, they would have rebuild every package that depends on libanswer (so both print-answer and display-answer-pp) and pushed those to the staging repository.

Finally, after a period of testing, they would have moved libanswer, print-answer and display-answer-pp in the official repositories in one swift update. They would have used a to do list like this one to coordinate the packaging tasks.

This means that if you try and update libanswer without upgrading every other package that depends on it, you risk breaking your installation - and that’s why partial updates are not supported on Arch Linux.

The Debian Way #

Debian maintainers use another strategy. When they package a library, they include its version number in the name of the package. What’s more, they have a separate development package that contains the files required for compiling programs that use the library. They also use a compilation trick called the soname option.

Let’s see how this works.

First, they tell gcc to use the correct soname option when linking the library ⁵:

$ gcc -I . answer.c -shared -Wl,-soname=libanswer.so.1 -o libanswer.so.1

They use the outcome of the build to generate two packages:

libanswer-dev, that contains a symlink libanswer.so -> libanswer.so.1 and the answer.h header
and libanswer1, that contains only the libanswer.so.1 file

Then, they build display-answer-pp for the first time, using the libanswer-dev package:

$ g++ -lanswer display-answer.cpp -o ./display-answer-pp

Because libanswer.so was built with the -soname=libanswer.so.1 option, the display-answer-pp binary now knows that it needs libanswer.so.1 at runtime:

$ readelf -d display-answer-pp
Tag                  Type               Name/value
0x0000000000000001 (NEEDED)             Shared library: [libanswer.so.1]
...

When the v2 version of libanswer is published by The Experts, they rebuild the shared library with an appropriate soname:

$ gcc -I . answer.c -shared -Wl,-soname=libanswer.so.2 -o libanswer.so.2

They use the outcome of the build to update the libanswer-dev package and to create a brand new libanswer2 package.

At this point:

libanswer-dev contains a symlink libanswer.so -> libanswer.so.2 and the updated answer.h header
libanswer2 contains only the libanswer.so.2 file

Then, they build display-answer-pp for the second time, using the updated libanswer-dev package:

$ g++ -lanswer display-answer.cpp -o ./display-answer

This time, display-answer-pp knows that it needs libanswer.so.2 at runtime:

$ readelf -d display-answer-pp
Tag                  Type               Name/value
0x0000000000000001 (NEEDED)             Shared library: [libanswer.so.2]
...

Let’s sum up:

Both libanswer1 and libanswer2 packages can co-exist in the same system, since they contain different filenames
The libanswer-dev package does not need to be installed for display-answer-pp to run since the program contain the versioned soname in its dynamic section
If you want to rebuild a program when a new version of the library is out, you just install the latest libanswer-dev package - the command used for compilation does not have to change at all!

Clever, right?

Because of this technique, the update from libanswer v1 to libanswer v2 is often called a soname bump - the update from v1 to v1.1 is just a regular update.

Quite often, a backward incompatible update correspond to the first digit of the soname to be updated. ⁶

Conclusion #

Now you know - different versions of a library have various sonames, and the symlinks are carefully crafted by distribution maintainers.

So, when you create a symlink yourself, you are taking a huge risk, especially when creating links between libraries that have a different leading digit in their soname!

As we saw, using the incorrect library at runtime can cause crashes, and if you break an essential binary (like bash for instance), you may no longer be able to log in, which means your only choice may be to re-install your whole system from scratch. This is not theoretical by the way: it happened to me years ago, and I still remember it to this day.

So, what to do if you get this error?

Error when running `bar`: `libfoo.so.5: no such file or directory`

If bar comes from a package on your distribution, update all packages and then open a bug report if the problem persists
If not, try and re-compile bar - possibly after updating the libfoo-dev package.

But please, please do not create a symlink from libfoo.so.6 to libfoo.so.5 or suggest this solution to someone else. Together, let’s put an end to this bad, bad, advice. Thank you!

If you’re puzzled by the __cplusplus stuff don’t worry - it’s just so that the answer library is usable from C++ too. ↩︎
By the way, it needs libc.so because the compiled code for printf lives there. ↩︎
You’re using Arch Linux because you’re cool and you want to learn - good for you! ↩︎
Reminder: it’s a story I made up! ↩︎
Actually, this is often done automatically by whatever build system the library is using ↩︎
Sadly it’s not always the case: lua and libpng are notable exceptions. ↩︎

Classes Rock

Sat, 16 Nov 2019 19:26:27 +0000

This is the second article in a 2-part series of blog posts that examine the complicated relationships between the real world and object-oriented code. If you haven’t already, you should read the first part: classes suck first.

Let’s recap: we used a Robot class to implement the client specifications. It worked well, until we tried to model the start and stop methods with a mutable state. We concluded that blindly using classes (with methods and mutable attributes) to represent real-worlds objects was a mistake.

Taking a second look at the code #

Let’s look at the code again (minus the stop method), along with the spec:

When robots come off the factory floor, they have no name:

def __init__(self):
    self._name = None

The first time you boot them up, a random name is generated:

def start(self):
    if self._name is None:
        self._name = generate_name()

When a robot is reset, its name gets wiped:

def reset(self):
    self._name = None

So we were modeling real-world objects with our class after all: the mistake we made was trying to model the robot. But using code to model the specification worked!

If you’re not convinced, consider this: the stop() method is empty because the specification said nothing about what happens when the robot is stopped.

If the specification said something like the robot cannot be reset when it is started, then using a stopped attribute may have been a good idea.

Encapsulation #

So far we talked only about methods and attributes. There’s another aspect of programming with classes I’d like to examine next: the concept of encapsulation, or how to separate “private” implementation and “public” interface.

Basically, there are some details of a class that matter to the people who wrote the class but that users of said class should not know about.

What does this have to do with the real world? Everything! Let me show you.

A single line telling a whole story #

In Python, to signify to users of a class that an attribute is not part of the “public” interface, you prefix it with an underscore, like this:

class Robot:
    def __init__(self):
        self._name = None

By the way, there are already two pieces of information here:

robot names are empty at first
the name likely changes over time (otherwise it would probably have been passed as a parameter in the constructor)

But that’s again about the specification. What does it mean to have _name as a private attribute?

Working in teams #

Let’s imagine we are writing software inside a big company with several teams. Reading the specification again, it’s clear that the client is extremely worried about robot names: each and every line of the specification talks about it! If there is a bug affecting the robot name, the client is going to get pissed.

And that’s where using encapsulation makes sense: by “hiding” the name attribute from the users of the Robot class, we are actually covering our bases. If someone from another team writes robot._name = "something-else", the client will get mad (the robot name no longer has the correct format), but we can tell him it’s not our fault: the other team should not have broken the encapsulation without asking us first!

Conclusion #

So, which is it? Do classes rock or suck at modeling the real world?

Well, it all depends on what you mean by “the real world”. My advice to you is:

clarify specifications as much as possible
write code that closely resembles the specification
use encapsulation when necessary to protect yourself against misuses of your code

When used correctly, classes are a great way to accomplish all of the above, but they are also easy to misuse.

Finally, let’s go back to our original question: “is modeling the real world in code worth it?”.

Well, if it allows you to implement the specifications correctly and makes things easier for you and your teammates, then yes, it’s definitely worth it.

It’s hard, but trying and solve this challenge is what I love about programming.

Cheers!

Update 2021-09-16: see also this fantastic article by By Mathias Verraes and Rebecca Wirfs-Brock: called Design and Reality.

Classes Suck

Sat, 09 Nov 2019 19:06:44 +0000

Introduction #

A long time ago I received a lecture about the usage of databases and software design. More specifically, how to translate relationships like “one-to-one”, “many-to-many”, or “one-to-many” inside databases schemas.

Ten years later, I still remember one thing the teacher said at the beginning of the course:

When you are designing your database schema, your enemy is the real world. The real world is always more complicated than you anticipated.

This is not new, of course, but it raises some interesting questions: how can we model the real world in our code? Is it worth it? Is not all software made of abstract objects? Are classes and instances the correct way to do it?

A two-part exploration #

In these two parts series, we’ll try and answer the last question.

We will use an exercise I’ve stolen from the exercism.io website, and the two parts will argue opposing points of view.

Let’s start with part one: classes suck. Bear in mind there we’ll be a part two called classes rock, so don’t leave angry comments below just yet :)

One last thing: this is a story based on real events: the code I’m about to show below was written this way by students I’ve just taught about instances and classes in Python.

The Robot factory exercise #

Here are the specifications of the exercise:

Your are a developer inside a robot factory. Your task is to manage robot settings. Here are the rules:

When robots come off the factory floor, they have no name.

The first time you boot them up, a random name is generated in the format of two uppercase letters followed by three digits, such as RX837 or BC811.

Every once in a while we need to reset a robot to its factory settings, which means that their name gets wiped. The next time you ask, it will respond with a new random name.

And here’s the code we’re given a starting point:

class Robot:
    # Put your code here:
    pass

def test_name_is_not_set_at_first():
    robot = Robot()
    assert robot.name() is None


def test_started_robots_have_a_name():
    robot = Robot()
    robot.start()
    actual_name = robot.name()
    assert re.match(r"^[A-Z]{2}\d{3}$", actual_name)


def test_name_does_not_change_when_rebooted():
    robot = Robot()
    robot.start()
    name1 = robot.name()

    robot.stop()
    robot.start()
    name2 = robot.name()
    assert name1 == name2


def test_name_changes_after_a_reset():
    robot = Robot()
    robot.start()
    name1 = robot.name()

    robot.stop()
    robot.reset()
    robot.start()
    name2 = robot.name()
    assert name1 != name2

Let’s try to fix tests one by one, starting with the first one.

Making the first tests pass #

All we need is a name() method that returns None:

class Robot:
    def name(self):
        return None

For the next test, we need a start method that will cause the name to change. Let’s store the name in a private attribute _name and adapt the rest of the code:

def generate_name():
    # implementation omitted for brevity

class Robot:
    def __init__(self):
        self._name = None

    def name(self):
        return self._name

    def start(self):
        self._name = generate_name()

To fix the next test, we need to implement stop:

def generate_name():
    ...

class Robot:
    def __init__(self):
        self._name = None

    def name(self):
        return self._name

    def start(self):
        self._name = generate_name()

    def stop(self):
        pass

And then we get:

test_name_does_not_change_when_rebooted:
>       assert name1 == name2
E       AssertionError: assert 'KJ721' == 'JO813'
E         - KJ721
E         + JO813

Making the test about reboot pass - first attempt #

Clearly, we have a bug in our implementation. The robot’s name must not change when it’s rebooted.

If we follow the “code objects must reflect real-world objects” rule, we may be tempted to say that robots have a state and that this state can be used to know when the name must be regenerated.

And we can choose to represent the state of the robot with a stopped attribute inside the Robot class:

class Robot:
    def __init__(self):
        self._name = None
        # Better make sure robots are stopped when
        # they come out of the factory floor :)
        self.stopped =  True

    def name(self):
        return self._name

    def start(self):
        self._name = generate_name()
        self.stopped = False

    def stop(self):
        self.stopped = True

And that’s where classes have failed us. This stopped attribute does not help at all making the test pass.

We tried and model the real world in our class but it was useless. And then we realize that there’s no so such thing as a “boolean” in the real world. In the real world, robots have LEDs that are turned on or turned off!

Making the test about reboot pass - second try #

Let’s revert the change that introduced the stopped attribute, and look at the implementation again. It becomes clear that to make the test pass, we just need an if in the start method:

class Robot:
    def __init__(self):
        self._name = None

    def name(self):
        return self._name

    def start(self):
        if self._name is None:
            self._name = generate_name()

    def stop(self):
        pass

Making the last test pass is easy: we only need to reset the ._name attribute when the robot is reset:

class Robot:
    def __init__(self):
        self._name = None

    def name(self):
        return self._name

    def start(self):
        if self._name is None:
            self._name = generate_name()

    def stop(self):
        pass

   def reset(self):
        self._name = None

Now all test pass and we’re done.

(Temporary) conclusion #

Looking at the code, it looks like our object has nothing to do with the real world. The closest connection with the real world is the ._name attribute which is probably just a bunch of zeros and ones inside the memory chip of the robot.

Let’s face it, using classes to represent the real world is a myth, probably spread by overzealous Java developers, or clueless researchers locked inside their ivory towers.

Or is it? Let’s find out in part two!.

I am a rusty frog

Sat, 27 Jul 2019 09:46:51 +0000

Do you know the fable of the frog and the boiling water? It goes like this:

If you drop a frog in a pot of boiling water, it will of course frantically try to clamber out. But if you place it gently in a pot of tepid water and turn the heat on low, it will float there quite placidly. As the water gradually heats up, the frog will sink into a tranquil stupor, exactly like one of us in a hot bath, and before long, with a smile on its face, it will unresistingly allow itself to be boiled to death.

At least that’s how Daniel Quinn’s tells it in The Story of B.

It is a powerful metaphor, although if you read the Wikipedia article, it turns out frogs do not actually behave that way.

But I think there’s still some truth it in: if you experience some increasing pain gradually enough, you may not realizing how much you are actually suffering until it’s too late. At least it’s what happened to me.

Let’s start at the beginning, before we start talking about Rust, refactoring, performance bugs and the weaknesses of the human mind.

Part 1: The setup #

A long time ago, I realized I spent far too much time typing cd to navigate between project directories, and that the default ctrl-r binding for zsh was a bit awkward to use for me.

Then I realized I could solve both of those problems by writing two very similar commands: cwd-history and cmd-history.

Let’s talk about the former. It’s a command-line tool that takes two actions, add and list. Here’s how it works:

cmd-history add takes one parameter and adds it to a “database” in ~/.local/share/bwataout/commands-history. The database is just a file with one command per line.
cmd-history list simply dumps the contents of the database to stdout.
Then, there’s a zsh hook to run right after a command has been read and is about to be executed. In the hook, cmd-history add is called with the text of the last command.
Finally, zsh is configured so that when ctrl-r is pressed, the output of cmd-history list is piped through fzf and then the selected command is run.

I use a similar program called cwd-history that does exactly the same thing but for the working directories. It runs in another zsh hook called chpwd, and is triggered by typing the command z¹.

There are a few specifics like making sure the entries are not duplicated or ensuring that every symlink in the added path is resolved, but we’ll get to that later.

Part 2: RIR #

The two programs were written in Python and shared a ton of functionality but since I am a lazy programmer I did not bother with refactoring the code.

But something was bugging me. There was a small noticeable delay whenever I was using them, particularly on old hardware. So I decided to try and rewrite them in Rust. And surely when I measured the performance, there was a huge difference in the startup time: Rust compiles directly to machine code, and there was no need to wait for the Python interpreter to start. I was happy.

Part 3: the switch to Kakoune #

A few months ago, I switched from Neovim to Kakoune for my main text editor and was faced with a problem.

Kakoune does not have any persisting state by design, and I was missing a particular vim feature that allowed me to quickly select any recently opened file (aka. MRU files).

But I already knew how to solve that! I wrote yet another tool called mru-files for reading and writing the MRU files in another database, and write a bit of Kakoune script:

# Call `mru-files add` with the buffer name when opening a new buffer
hook global BufOpenFile .* %{ nop %sh{ mru-files add  "${kak_hook_param}" } }

# Call `mru files list` when pressing <leader key>o:
map global user o
  -docstring 'open old files'
  ': evaluate-commands %sh{ mru-files list --kakoune }<ret>'

Note the --kakoune option when calling mru-files list: we need to call the menu command so that we can present the list of MRU files to the user and open the file when it’s selected.

It looks like this:

menu
  "/foo.txt" "edit -existing 'foo.txt:'"
  "/bar.txt" "edit -existing 'bar.txt'"
  ...

While I was it, I also added a --kakoune argument to cwd-history list so that I could switch to previously visited directories directly from Kakoune:

 map global cd o
   -docstring 'open old working directory'
   ': evaluate-commands %sh{ cwd-history list --kakoune }<ret>'

Part 4: the refactoring #

So now I was faced with three very similar code written in Rust. It was time for a refactoring ².

Here’s what I did:

First, I added a trait called EntryCollection that contained methods named add and list

pub trait EntriesCollection {
    fn add(&mut self, entry: &str);
    fn list(&self) -> &Vec<String>;
}

The trait was implemented by three structs: Commands, WorkingDirs and MruFiles.

pub struct Commands {
    entries: Vec<String>,
}

impl EntriesCollection for Commands {
  // ...
  fn add(&mut self, entry: &str) {
      // Deduplicate the entry and store it in self.entries
      // ...
  }

  fn list(&self) -> Vec<String> {
      // return self.entries
  }
}

pub struct WorkingDirs{
    entries: Vec<String>,
}

impl EntriesCollection for WorkingDirs {
   // ...
   fn add(&mut self, entry: &str) {
         // Convert the entry to a path, check  if it exists,
         // then canonicalize it and insert it in self.entries
   }
}

(and similar code for MruFiles)

I wrote a Storage struct to interact with the database:

impl Storage {
    pub fn new(
        mut entries_collection: Box<EntriesCollection>,
        path: &std::path::PathBuf,
    ) -> Storage {
        let db_path = path.join(entries_collection.name());
        let entries = read_db(&db_path);
        for entry in entries {
            entries_collection.add(&entry);
        }
        Storage {
            db_path,
            entries_collection,
        }
    }

    pub fn list(&self) -> &Vec<String> {
        &self.entries_collection.list()
    }

    pub fn add(&mut self, entry: &str) {
        &mut self.entries_collection.add(&entry);
        write_db(&self.db_path, &self.list())
    }

I wrote StorageManager struct to instantiate the Storage struct with the correct EntriesCollection implementation depending on a StorageType enum:


pub enum StorageType {
    CwdHistory,
    CommandsHistory,
    FilesHistory,
}

pub struct StorageManager {
   // ...
}

impl StorageManager {
    pub fn new(storage_type: StorageType) -> Self {
            let entries: Box<EntriesCollection> = match storage_type {
                StorageType::CwdHistory => Box::new(WorkingDirs::new()),
                StorageType::CommandsHistory => Box::new(Commands::new()),
                StorageType::FilesHistory => Box::new(MruFiles::new()),
            };


        };

        let storage = Storage::new(entries, &app_dir);
        StorageManager { storage }
    }

    pub fn list(&self) {
        // Delegates to self.storage.list()
    }

    pub fn add(&mut self, entry: &str) {
        // Delegates to self.storage.add()
    }

}

And finally, I wrote three main functions that called the StorageManager constructor, passing it the enum variant.

// In src/bin/cmd-history.rs
use bwataout::StorageType;
fn main() {
    // ...
    bwataout::run_storage_manager(StorageType::CommandsHistory)
}

// In src/bin/cwd-history.rs
use bwataout::StorageType;
fn main() {
    // ...
    bwataout::run_storage_manager(StorageType::CwdHistory)
}

// In src/bin/mru-files.rs.rs
use bwataout::StorageType;
fn main() {
    // ...
    bwataout::run_storage_manager(StorageType::FilesHistory)
}

And that was my first mistake: I forgot to measure performance after the refactoring. I was sure the code was correct. After all, if it compiles, it works, right?

And sure enough, the code did work. I could open MRU files and old project directories from Kakoune, and I was quite pleased with myself.

Of course, by now you should have guessed there’s a horrible performance bug in the code above. Did you spot it? If you did, congrats! I certainly did not at the time.

Part 5: The peer programming session #

The refactoring took place 6 months ago. I am using those tools at work, which means I was using those commands for 6 months, 5 days a week. The database files contained thousands of entries. The horrible bug was still there, consuming CPU cycles without a care in the world, and I did not notice anything.

Fortunately, I did a peer programming session with one of my colleagues and something weird happened. He got bored watching my zsh prompt getting stuck when I was using the z and ctrl-r shortcuts.

I then measured the performance of my tools again and I found that the cmd-history list command took 1.7 seconds to run. That’s one thousand and seven hundred milliseconds. It’s a very long time. No wonder my colleague got bored! What’s amazing though is that I did not notice anything wrong until he told me about it.

Part 6: The fix #

Now it’s time to squash the bug. The culprit lies here, in the Storage constructor:

impl Storage {
    pub fn new(
        mut entries_collection: Box<EntriesCollection>,
        path: &std::path::PathBuf,
    ) -> Storage {
        let db_path = path.join(entries_collection.name());
        let entries = read_db(&db_path);
        for entry in entries {
            entries_collection.add(&entry);
        }
        Storage {
            db_path,
            entries_collection,
        }
    }
}

We’re reading each line of the database file, and passing it to EntriesCollection.add(). This means we keep calling the add() method over and over. It does not do much, but it still has to go through all the entries when performing deduplication. This is a classic case of the Shlemiel algorithm and it explains the abysmal performance of the tool as soon as the database gets big enough.

I believe I wrote the code that way because I thought it would be nice to somehow “validate” the entries when reading the database. That way, if the algorithm in the add method changed, the database will be “migrated” automatically (for instance, all non-existing paths still present in the db will automatically disappear). Another classical mistake named YAGNI: it’s doubtful I’ll ever need to “migrate” the database, and when I need to, I’ll probably just have to write a tiny throw-away script to do it.

Anyway, now that we’ve decided the “auto migrating” feature can go away, we can solve our performance issue by adding an add_all method to the trait, and replacing the for loop in the Storage constructor:

 pub trait EntriesCollection {
     fn add(&mut self, entry: &str);
+    fn add_all(&mut self, entries: Vec<String>);
     fn list(&self) -> &Vec<String>;
}

impl Storage {
    pub fn new(
        mut entries_collection: Box<EntriesCollection>,
        path: &std::path::PathBuf,
    ) -> Storage {
        let db_path = path.join(entries_collection.name());
        let entries = read_db(&db_path);
-       for entry in entries {
-           entries_collection.add(&entry);
-       }
+       entries_collection.add_all(entries);
        Storage {
            db_path,
            entries_collection,
        }
    }
}

In turn, this means we need to implement the add_all() method on all structs:

impl EntriesCollection for Commands {
    // ...
    fn add_all(&mut self, entries: Vec<String>) {
        self.entries = entries;
    }
}

impl EntriesCollection for WorkingDirs {
    // ...
    fn add_all(&mut self, entries: Vec<String>) {
        self.entries = entries;
    }
}

impl EntriesCollection for MruFiles {
    // ...
    fn add_all(&mut self, entries: Vec<String>) {
        self.entries = entries;
    }
}

A small digression #

Surely you’ve noticed the implementation of add_all() is exactly the same in the three structs. Why did not we add blanket implementation of the add_all() method in the trait?

Well, because we need access to the entries field of the struct, and there’s no way to do that in Rust. Since I’m in love with Rust, I can rationalize this in several ways:

First, the code duplication is not that a problem since there’s only one line of code.
Second, the duplication is incidental. We could imagine using a Set instead of a Vec to store the entries, and the trait does not need to know about this. Ditto if we decide to rename the field.
Third, I can tell myself that this lacking feature is a good thing³ because I’ve often been bitten in other languages when sharing class members through inheritance (both in C++ and Python).

Take those how you want: I can tell myself those are good arguments, but you don’t have to believe me :)

The moral of the story #

Here are my takeaways:

When performance matters, don’t forget to measure it constantly, especially after refactorings (or even after each commit if you can afford to)
The boiling frog metaphor is actually a good one, and you may be inside some boiling water without realizing it.
And finally, listen to what people say to you when they tell you about something bad in your life. I actually felt something was off with my machine those last 6 months, but it took an other person to help me take a step back and understand what was wrong.

Take care and see you another time!

That’s because I was using a tool named z before I decided to re-implement it myself. ↩︎
If you are wondering why, go read The Rule of Three on the Coding Horror blog. ↩︎
By the way, not all Rust users think this way. You can find a RFC about using fields in traits on GitHub. ↩︎

Requests: What You Need To Build Useful Apps

Wed, 24 Jul 2019 11:01:01 +0000

Today’s article has been written by guest Abdur-Rahmaan Janhangeer. He is Organising Member of the Python Mauritius User Group, Arabic Coordinator for the docs and maintainer of pythonmembers.club and was kind enough to submit this article to be posted there.

Requests is a popular Python library. It has particularly been praised for its intuitive programming API. If you are new to Python, this should be a top priority in your learning path. Today we are going to give you some useful blocks to immediately get started! Have fun!

Just How Popular Is Requests? #

Requests gets 1M+ downloads per day, yes per day! That’s really crazy. It’s the lifetime download of an average package! It has also been used among others to help humanity see the first image of a blackhole. Sphinx, Celery and Tornado also use it!

How Do Server Requests Work? #

Requests get it’s name from requests made to servers. Servers always listen for requests to resources. You tell it: give me a packet of sugar, it looks for it, if found, it’ll tell: “ok! found it!”" else, it will tell: “oh oh not found”. In real life, your browser asks for images, webpages etc. Here are some types of requests:

GET
POST
PUT
DELETE

The most used type is the GET request. While opening a normal webpage, your browser makes a GET request.

Comparing Normal Code To Requests Code #

Here is what you’ll do to get a byte representation of a webpage (which gives the page source).

import urllib.request
content = urllib.request.urlopen("https://dmerej.info").read()
print(content)

with requests (it’s pip install requests), you do

import requests as r
req = r.get("https://dmerej.info")
print(req.content)

req has many attributes to try. Let’s explore some!

Requests Exploration #

Try the following in a shell:


>>> import requests as r
>>> req = r.get("https://dmerej.info")
>>> req.text
>>> req.status_code # 200 means ok!
>>> req.reason
>>> req.text
>>> req.content

Start Cooking: Using Beautiful Soup #

Here’s a BeautifulSoup snippet usage with requests:

from bs4 import BeautifulSoup
import requests as r

req = r.get("https://dmerej.info")
soup = BeautifulSoup(req.content, 'html.parser')
print(soup.text)

As a side note, if you have a url like this:

http://modifyme.org?name=dimitri

you can make a get request like this:

req = r.get('http://modifyme.org', params={'name':'dimitri'})

and you will get the same thing.

Getting JSON data #

JSON data are similar to Python dictionaries. Here is a snippet to try out:

import requests as r

req = r.get("https://randomuser.me/api/")
print(req.json())

It’s an api that generates fake user data. req.json() returns a Python dictionary ready to use.

Download File #

Here’s a function that downloads a file where you want, given the url:

import requests

def download_file(file_url, path):
    r = requests.get(file_url, allow_redirects=True)
    open(path, 'wb').write(r.content)

Since r.content returns a byte object, we write the file in wb (write binary) mode, thus getting our file. It works for any file url. Try it!

Upload File #

POST requests are used to send data to the server, a typical post requests look like this:

r.post('http://www.example.com/verify_login', data={'name':'moi', 'password':'1234'})

Let’s say while disseting a form you get a form using POST to upload a file which looks like this:

<form action="/upload" method = "POST"
enctype="multipart/form-data">
<input type="file" name="file_to_upload" />
<input type="submit"/>
</form>

We can write a script like this to upload a file, an image in this case:

import requests as r

files = {'file': open('<path_to>/image.png', 'rb')}
req = r.post("http://www.example.com/upload", files=files)
print(req.text)

The above snippet is useful if you have to upload to your own custom backend. But, in the real world, sites add some input values with different name attributes to detect automatic submissions and you might have to modify to

r.post("http://www.example.com/upload", files=files, data={'<name>':'<value>',})

Caveats To Look For #

It should be born in mind that requests does not execute the JavaScript on a web page. If you are doing webscraping, there are other tools like Selenium.

Requests in an incredible tool to add to your arsenal, make good use of it!

dmerej.info: now also in French

Sun, 05 May 2019 16:16:35 +0000

Hello there,

Just a quick note to let you know I’m now also producing contents in French, at https://dmerej.info/blog/fr

If you do speak French, you can read more about it in the introductory post.

Cheers!

The virtue of incremental development

Sun, 21 Apr 2019 12:10:49 +0000

Here’s today challenge: can you write a command-line tool that allows converting to and from various measurements units?

For instance, you could input “3 miles in meters” and get “4828.03”.

I submitted this challenge to my Python students last weekend, asking them to write the code from scratch. ¹

1 hour later, something miraculous happened that I never would have expect.

But let me tell you the full story.

Getting started #

I told the students that they could start by writing some “exploratory code”.

“Just hard-code anything you have to and keep everything in the main() function “, I said.

After a few discussions, we agreed to only write code that converted kilometers to miles, and that we’ll read the values from the command line.

Here’s what we came up with:

import sys


def main():
    kilometers = float(sys.argv)
    miles = kilometers / 1.609
    print(f"{.2f}", miles)

if __name__ == "__main__":
    main()

I then pointed out that the code was not generic. Indeed, “kilometers”, “miles” and “1.609” are hard-coded there.

Naming a new function #

The students understood there was a three-parameters function waiting to be written. So we went to the drawing board and after a while we decided to have a function called convert(value, unit_in, unit_out).

Note that we did not make any assumption about the body of the function. We just wanted to see how main() could become more generic, and we were still allowed to hard-code parts of the code:

def convert(value, unit_in, unit_out):
    coefficient = 1 / 1.609
    result = value * coefficient
    return result


def main():
    value = float(sys.argv[1])
    unit_in = sys.argv[2]
    unit_out = sys.argv[3]

    result = convert(value, unit_in, unit_out)
    print(f"{.2f}", result)

Some notes:

The main() function is now completely generic, and we probably won’t need to change it.
The signature of the convert function almost dictated the command-line syntax:

def convert(value, unit_in, unit_out):
    ...

# Usage: convert.py value unit_in unit_out
$ python3 convert.py 2 meters miles

Computing the coefficient #

Now it was time to get rid of the hard-coded coefficient. This time finding a function name was easier:

def get_coefficient(unit_in, unit_out):
   ...

Then we tried to figure out how to implement it. We knew we would be needing a dictionary, but the structure of it was unknown.

“Back to the drawing board”, I said. “Let’s write down what the dictionary should look like”.

Here’s our first attempt:

units =  {
   "km": { "miles": 1/1.609, "meters": 1/1000, ....},
   "yards": { "miles": 1/1760, "meters": ..., "km": ...}
   ...
}

“This won’t do”, I said. “Look at what happens if we add a new measurement unit, such as feet”.

We’ll have to:

add a new ‘feet’ key to the units dictionary,
compute all the coefficient to convert from feet to all the other units,
and add a feet key to all the other dictionaries

There has to be a better way!

After a short brainstorming session, we decided to limit ourselves to distance measurements, and to always convert to SI units first.

So we draw the new structure of the units dictionary:

# Coefficients convert from "meters"
distances = {
    "km": 1/1000,
    "yards": 1.094,
    "miles": 1/1609,
}

And then we thought about the algorithm. We found three possibilities:

If we want to convert from meters, we just have to look up the coefficient in the dictionary
If we want to convert to meters, we can look up the coefficient in the dictionary and return its inverse
Otherwise, we combine the two procedures above and return the product of the two coefficients.

“This is looking good”, I said. “Let’s try to implement the algorithm but just for the first case and see what happens”.

Testing the algorithm #

I showed my students how they could use Python’s interpreter to check the get_coefficient() function was working properly.

We quickly managed to get the first case working:

def get_coefficient(unit_in, unit_out):
    # FIX ME: only works with distances for now
    # Coefficients to convert from "meters"
    distances = {
        "km": 1/1000,
        "yards": 1.094,
        "miles": 1/1609,
    }
    if unit_in == "m":
        return distances[unit_out]

>>> import conversion
>>> conversion.get_coefficient("m", "km")
0.001
>>> conversion.get_coefficient("m", "yards")
1.094

“Cool, this works”, I said. “Let’s see what happens when the input value is not in meters:”

def get_coefficient(unit_in, unit_out):
    # FIX ME: only works with distances for now
    # Coefficients to convert from "meters"
    distances = {
        "km": 1/1000,
        "yards": 1.094,
        "miles": 1/1609,
    }
    if unit_in == "m":
        return distances[unit_out]
    else:
        reciprocal_coefficient = 1 / distances[unit_in]
        return reciprocal_coefficient * distances[unit_out]

>>> import conversion
>>> conversion.get_coefficient("miles", "yards")
1760

“Look how readable the code is”, I said. “We have a value that’s called reciprocal_coefficient and we get it by calling 1 over something else. Isn’t this nice?”.

The miracle #

I then pointed out that the ’else’ after the return was unnecessary.

def get_coefficient(unit_in, unit_out):
    # FIX ME: only works with distances for now
    # Coefficients to convert from "meters"
    distances = {
        "km": 1/1000,
        "yards": 1.094,
        "miles": 1/1609,
    }
    if unit_in == "m":
        return distances[unit_out]
    reciprocal_coefficient = 1 / distances[unit_in]
    return reciprocal_coefficient * distances[unit_out]

And then it happened. “Hey”, one of the students said, “what if we added meters in the distances dictionary with 1 as value? We could get rid of the first if too!”.

“Let’s do it”, I said:

def get_coefficient(unit_in, unit_out):
    # FIX ME: only works with distances for now
    distances = {
        "m": 1,
        "km": 1/1000,
        "yards": 1.094,
        "miles": 1/1609,
    }
    reciprocal_coefficient = 1 / distances[unit_in]
    return reciprocal_coefficient * distances[unit_out]

>>> import conversion
>>> conversion.get_coefficient("m", "m")
1
>>> conversion.get_coefficient("km", "m")
1000
>>> conversion.get_coefficient("m", "yards")
1760

And of course, this works. When meters is either unit_in or unit_out, all operations will involve multiplying or dividing by 1.

That was a really nice surprise for several reasons:

One, when I thought about the problem alone, before starting the workshop, I was pretty sure I would need a much more complex data structure.
Two, one of the students just refused to believe the code would work, even after having seen it in action in the interpreter ;)
Three, we killed one comment :)

Lessons learned #

We found a beautiful algorithm and a nice data structure, not by trying to solve everything at once, but by slowly building up more and more generic code, getting rid of hard-coded values one after the other, and by carefully thinking about naming.

I hope you find this approach useful, and I highly suggest you try using it next time you implement a new feature.

Cheers!

I’m using mob programming during my Python classes. It works really well. ↩︎

Crafting command line outputs

Mon, 25 Mar 2019 13:55:26 +0000

I’ve used many command line tools. It always makes me feel happy when I come across a tool that has a nice, usable and friendly output. I always try to make those I create or maintain as good as possible in this regard.

Along the way, I’ve learned a few tricks I’d like to share with you.

Start with the contents #

When designing command line output, start with the contents, not the style. In other words, think about what to print before thinking about how to print it - this is helpful in a lot of other situations.

Error messages #

The absolute minimum your command line tool needs are good error messages. Pay attention to those - good error messages can go a long way in helping your users and may even save you from having to answer a bunch of bug reports! ¹

A good error message should:

Start with a clear summary describing what went wrong
Contain as many details as relevant (but not too much)
And possibly a suggestion on how to fix the issues.

Bad example:

Could not open cfg file. ENOENT 2

Good example:

Error while reading config file (~/.config/foo.cfg)
Error was: No such file or directory
Please make sure the file exists and try again

Standard out and standard error #

Speaking of error messages, note that your can choose to write text to two different channels (often called stdout and stderr for short). Use stderr for error messages and error messages only. People sometimes need to hide “normal” messages from your tool, but they’ll need to know about those errors!

Colors #

Things to know about colors:

Use red for errors (that’s about the only convention I know of which is somewhat followed by everyone).
You can get a lot of meaning out of simple, ASCII-art decoration. Don’t necessarily reach from emojis immediately 😜.
Try to use colors in a consistent way. Having helper methods like print_error(), print_message () can help.
On Linux and macOS, coloring is achieved by emitting certain non-printable ASCII characters (sometimes referred to as ANSI escape codes). This is fine when your program runs in a terminal, but not when its output is redirected to a file, for instance.
People usually expect color activation to be controlled with a tri-state: “always”, “never”, or “auto”. The first two are self-explanatory, but “auto” needs some explaining.
When “auto” is set, your program should decide whether to use colors by itself. You can do so by calling isatty(stdout) or something equivalent.
On Windows, coloring is achieved by using the win32 API, but the same ideas apply.
There’s probably a library near you that implements all of this. Even if it does not seem much, there’s no point in re-inventing the wheel here.
Finally, try to follow the CLICOLORS standard.

Progress #

If your comman-line tool does some lengthy work, you should output something to your users so that they know your program is not stuck.

The best possible progress indicator contains:

An ETA
A progress bar
A speed indicator

wget does that really well. Your favorite language probably has one or two ready-to-use libraries for this.

If you can’t achieve the full progress bar, as a lightweight alternative you can just display a counter: (1/10, 2/10, etc.). If you do so, please make sure the user knows what is being counted !

Remove noise #

We already saw that people maybe running your tool without a terminal attached. In this case, you should skip displaying progress bars and the like.

Also, try to remove things that are only useful when debugging (or don’t display them by default).

This includes the time taken to perform a given action (unless it’s useful to the end-user, like if you are writing a test runner for instance). Also, don’t forget that users can and will prefix their command line with time to get precise results if they need to.

A good technique to remove noise is to completely erase the last line before writing a new one.

Here’s how to do it in Python:

size = shutil.get_terminal_size()   # get the current size of the terminal
print(first_message, end="\r")
do_something()
print(" " * size.columns, end="\r")  # fill up the line with blanks
                                     # so that lines don't overlap
print(second_message)

Of course, this only works if the user does not need to know about the whole suite of messages!

Tests #

End-to-end tests are a great way to tweak the output without having to do a bunch of setup by hand, and check the error messages look good.

Parting words #

Well, that’s all I’ve got today. Please keep those tips in mind when creating your own command line tool, and if you find a program that does not adhere to these rules, feel free to send patches. Until next time!

Assuming they take the time to read the output. But the better they are, the more they’re likely to get read :) ↩︎

My Blogging Flow - Part 2 - Publishing

Thu, 07 Feb 2019 12:14:20 +0000

Welcome to part 2 of “How this blog gets written”. If you haven’t already, feel free to read part 1 first :)

The Idea #

Everything starts with an idea. I have a small git repository on my dedicated machine which contains a bunch of markdown files, one per article. (This repository is private by the way).

The contents of these files can vary from just a few lines to an almost complete article. Most of the time they only contain a basic outline though.

I update the ideas repository as soon as ideas come. It’s important to write things down as soon as possible. Forgetting what you wanted to say or how you wanted to say it happens far more quickly than you would think.

The first draft #

Next, it’s time to start writing the full article. I have a small Python script which:

finds out the next article number and fill up the front matter for me (because I’m lazy)
runs git add on the generated Markdown file so I don’t delete it by mistake by running git clean (because mistakes happen)
opens the Markdown file in my text editor

Now it’s time to draft the article. I’ve configured my editor so that:

spell checking is activated
syntax highlighting is disabled (I’ve explained why in a previous blog post).

In an other terminal (usually hidden behind a tab), I have hugo serve running, and finally, next to the text editor, a web browser, so that I can see the changes happening in real time.

Proofreading #

And then it’s time to proofread, proofread, proofread …

I usually wait for a good night’s sleep before re-reading an article.

If the article is really big and complex, I sometimes open a pull request on GitHub and ask friends or colleagues to review it.

I do my proofread both in my text editor and in the browser, to increase the chance I’ll see spelling errors, awkward formulations, transitions problems and the like.

Publishing #

I have another Python script to automate publication. Here’s what it does:

make an automatic commit in case I forgot (the script just skips this part if the repository is clean)
run hugo --buildDrafts=false to build the HTML files
run rsync to upload them to my server
run git push to publish changes to GitHub

Note that the publishing of a new post triggers a new version of the blog’ feed to be written.

This script can be run after any corrections to an already published article.

When a brand new article has just been published, I run yet an other Python script to:

Parse the RSS feed and find out the URL and title of the latest article
Write an automatic tweet like this one (the additional hashtags at the end are given on the command line)
Write a toot on mastodon with similar content
Send an email to the people who subscribed to the newsletter. I manually maintain a list of subscribers inside a configuration file for now, since there’s no subscription form on the blog.

dev.to #

Then it’s time to wait until dev.to picks up the latest version of the RSS feed, parses the HTML from the feed, converts it back to markdown and add it into my dashboard.

(Yup, you can configure dev.to to do this!)

This works quite well given the weird round-trip through HTML. Still, I have to re-read the markdown created by the dev.to importer just in case. This gives me a final opportunity to proofread the article one last time.

Feedback #

Thanks to isso I get notified when new comments are posted. I also of course get notifications from dev.to.

When I’m lucky, @thepracticaldev tweets a link to my article, and I get new followers and more views. This is nice :)

That’s all folks! If you have any other questions regarding how the blog works, don’t be afraid to ask in the comments below!

2020-07-20 update: I’ve published a gist on GitHub containing the relevant part of the Python code mentioned above.

My Blogging Flow - Part 1 - Under the hood

Wed, 06 Feb 2019 11:46:32 +0000

Several people have asked me how my blog works and how it gets written, so here’s a blog post about how I blog (so meta).

Let’s start at the beginning.

The history of dmerej.info/blog #

Buying a DNS domain #

The first step was to buy a domain name. I chose gandi.net because I only heard good things about their registration service. I like the “retro” feel of the .info TLD, and the dmerej prefix is the one I use to shorten my name. Wether it’s a good thing or not, it’s too late to change it now, so there.

Getting my own machine #

I always wanted to control all aspects regarding the hosting of my blog, so I started by looking for a dedicated machine.

If you’re not familiar with the concept, having a dedicated machine means that you pay for the hardware and ssh access to a server owned by soneone else. In exchange, you get to install and configure everything by hand. If it sounds like a loosy deal to you, remember that I value control over convenience :)

Anyway, I bought a server at a company called dedibox. (For the sake of completeness, I must say I’m now hosted by the nice people at Digital Ocean).

But enough speaking about hosting services, let’s talk about blog engines!

Static or dynamic #

There are two types of blog engines: static and dynamic. It has to do with what happens when a visitor requests the article’s URL.

In a dynamic blog engine, the contents of each article is typically stored in a database, and the HTML is generated by the engine for each visitor. This means the server has to run some code for each visit.

On the other hand, a “static” engine typically generates a bunch of HTML files from sources written in a markup language (often Markdown) beforehand. Then all the server has to do is to present the correct file when a visitor wants to view a page.

Both of them have pros and cons:

Dynamic blog engines can handle comments, compute statistics (like view count), and so on. But because they require a database and some programming language support, they can get hard to deploy. Finding a way to do backups properly can get tricky, too.
Static blogs cannot handle comments or anything like that, but they are very easy to deploy: you can simply use existing web servers (like nginx or apache) directly. Also, backup is easy - you just need to backup the source files, and a git repository is more than enough.

Finding a blog engine #

I must say I spent way too much time finding the engine I would use for my blog.

My guts told me I wanted a static engine, so I started fiddling with many static blog generators. Alas, they are tons of static engines out there, and I almost gave up on finding the one I would use - there were simply too much choice.

That’s when I realized I was not tackling the problem correctly: I was worried on how to publish contents, but said content was yet to be written!

So I took a look at dynamic blog engines (less choice there), found Dotclear almost by chance, and started to write.

Finally, my articles were online and publicly readable by anyone on the world. What a joy!

Advice to my dear readers: if you want to publish something online, don’t make my mistake: make sure you have some contents before worrying about its publication.

The Dotclear period #

Configuring php for Dotclear was a bit tedious, but once it was done I must admit it was quite pleasant to use. The admin view is rich and featureful, and the editing form is ergonomic. Comments are supported out of the box, and they are easy to moderate.

But I still I had a problem: I was receiving many notifications about comments but almost all of them were spam :/

This was depressing, so I shut down the comment service and started looking for an alternative.

The Hugo period #

I went back looking for static engines, and found Hugo on the StaticGen engine comparison website.

I immediately liked it:

There are many beautiful themes available. (Dotclear themes are nice too, but they look a tad outmoded to me).
Hugo is easy to install (just one binary)
Its documentation is complete and easy to follow
I no longer had to setup php on my server. Just a few lines of nginx configuration was enough.
Plus it is ridiculously fast. It can build more than one hundred pages in less than one second.

To work, Hugo need a theme. A Hugo theme is made of various templates, which control how the HTML gets produced, and some static files (like CSS or images) that needs to be copied along the generated files.

Hugo does not come with a default theme, so after a few tries, I settled on blackburn.

Writing my own shortcode #

The nice thing about Hugo is that you can customize the look and feel of your blog as much as you like. I still remember fondly the day I managed to implement the “movie script look” for the short scenes I wrote in Heard and Seen at FOSDEM 2017.

Here’s how it works.

First, I have custom scene shortcode inside the layouts/shortcode directory of the theme:

<div class="scene">
<h1>
  {{ .Get "title" }}
</h1>
<p>
{{ .Inner | markdownify }}
</p>
</div>

This allows me to write things like this in the Markdown source ¹:

{{〈 scene title="Resurrecting Dinosaurs" 〉}}
SPEAKER: I did *not* expect to have a win32 architecture slide here at FOSDEM
{{〈 /scene 〉}}

When Hugo sees the scene directive it will generate HTML code containing:

The <div> and its special scene class
The title inside the h1 tag
And the text inside the scene directive, interpreted as Markdown too:

<div class="scene">
<h1>
  Resurecting Dinosaurs
</h1>
<p>
SPEAKER: I did <em>not</em> expect to have a win32 architecture slide
here at FOSDEM.
</p>
</div>

Finally, I have a bit of CSS to render the div properly:

.scene {
  font-family: monospace;
  line-height: 1.2em;

  h1 {
    line-height: 1.5em;
    font-size: 1.2em;
    font-style: bold;
    text-align: center;
  }
}

And that’s all there is to it :)

Switching to a new theme #

After a while, I was getting tired with blackburn. Its CSS files were not easy to edit, it also had big dependencies like font-awesome or Yahoo’s Pure CSS.

So I switched to minio, which is the theme I still use today. It uses Sass instead of pure CSS, and its sources are neatly organized. Plus, it comes with a nicely configured toolchain, which allows to get feedback on changes made to the theme instantly.

Source organization #

Each blog post has its own markdon file, numbered chronologically:

0001-hello-world.md
0002-i-use-vim-and-so-should-you.md
0003-finding-a-good-project-name.md
...

I like this scheme because posts are naturally sorted by time, and because I know immediately know how much articles I’ve written. (More than 100 today!)

Since I don’t really want the internal number to be seen outside, all the articles have a slug in the “front matter”:

# In 0042-foo.md
---
slug: "foo"
date: "2016-03-31T13:00:19+00:00"
draft: false
title: "..."
---

This means the 0042-foo.md file will get written as foo/index.html.

I also told nginx that blog/post/foo correspond to blog/post/foo/index.html:

location /blog {
  try_files $uri $uri/index.html =404;
}

This is called pretty URLs in Hugo’s documentation.

Comments #

The last piece of the puzzle is the comments system.

I use isso, a self-hosted comment service.

First, I’ve patched the Hugo template to add a tiny bit of JavaScript at the bottom of every article:

<!-- in layouts/entry.html -->
<div>
  <script data-isso="//dmerej.info/isso/"
            src="//dmerej.info/isso/js/embed.min.js"
  >
  </script>
  <section id="isso-thread">
  </section>
</div>

Then I made sure the isso service was reachable at https://dmerej.info/isso, with help from nginx:

# In /etc/nginx/conf.d/blog.conf
location /isso {
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Script-Name /isso;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_pass http://localhost:1234;
}

and systemd:

# In isso.service
[Unit]
Description=isso comments service
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
ExecStart=/srv/isso/.local/bin/isso -c /srv/isso/isso.conf run
User=isso

[Install]
WantedBy=multi-user.target

Some notes:

The comment form does not show if the visitor has disabled JavaScript. I find that a bit sad, but on the other hand it keeps robots from posting spam (at least so far).
All the comments are stored in a SQLite database. I have a systemd timer to back it up every day.
People can opt-in to leave their e-mail in the form. Isso does nothing with it but store them in its database. I sometimes use them to send them private messages.

So that’s how the blog works under the hood! Stay tuned for part 2, where I’ll explain how new articles get written and published.

I’m using unicode character RIGHT ANGLE BRACKET to prevent Hugo from expanding the shortcode in this article … ↩︎

Killing unwrap()

Wed, 30 Jan 2019 19:11:26 +0000

Wait, what? Who do you want to kill? #

In Rust, to indicate errors or absence of a value we use types named Result and Option respectively.

If we need the value of an Result or Option, we can write code like this:

let maybe_bar = get_bar();
// bar is now an Option<String>
if maybe_bar.is_some() {
  let bar = bar.unwrap();
  // now bar is a String
} else {
  // handle the absence of bar
}

This works well but there’s a problem: if after some refactoring the if statement is not called, the entire program will crash with: called Option::unwrap on a None value.

This is fine if unwrap() is called in a test, but in production code it’s best to prevent panics altogether.

So that’s the why. Let’s see the how.

Example 1 - Handling options #

Let’s go back to our first example: we’ll assume there is a bar::return_opt() function coming from an external crate and returning an Option<Bar>, and that we are calling it in my_func, a function also returning an option:

fn my_func() -> Option<Foo> {
  let opt = bar::return_opt();
  if opt.is_none() {
    return None;
  }
  let value = opt.unwrap();
  ...
  // doing something with `value` here
}

So how do we get rid of the unwrap() here? Simple, with the question mark operator:

fn my_func() -> Option<Foo> {
  let value = bar::return_opt()?;
  // Done: the question mark will cause the function to
  // return None automatically if bar::return_opt() is None
  ...

  // We can use `value` here directly!
}

If my_func() does not return an Option or a Result you cannot use this technique, but a match may be used to keep the “early return” pattern:

fn my_func() -> SomeType {
  let value = match bar::return_opt() {
      None => return SomeType::default()
      Some(v) => v
  };
  // do something with v
}

Note the how the match expression and the let statement are combined. Yay Rust!

Example 2 - Handling Result #

Let’s see the bad code first:

fn my_func() -> Result<Foo, MyError> {
  let res = bar::return_res();
  if res.is_err() {
    return Err(MyError::new(res.unwrap_err());
  }
  let value = res.unwrap();
  ...
}

Here the bar::return_res() function returns a Result<BarError, Bar> (where BarError and Bar are defined in an external crate). The MyError type is in the current crate.

I don’t know about you, but I really hate the 4th line: return Err(MyError::new(res.unwrap_err()); What a mouthful!

Let’s see some ways to rewrite it.

Using From #

One solution is to use the question mark operator anyway:

fn my_func() -> Result<Foo, Error> {
  let value = bar::return_res()?;
}

The code won’t compile of course, but the compiler will tell you what to do, and you’ll just need to implement the From trait:

impl From<BarError> for MyError {
    fn from(error: BarError) -> MyError {
        Error::new(&format!("bar error: {}", error))
    }
}

This works fine unless you need to add some context (for instance, you may have an IOError but not the name of the file that caused it).

Using map_err #

Here’s map_err in action:

fn my_func() -> Result<Foo, Error> {
  let res = bar::return_res():
  let some_context = ....;
  let value = res.map_err(|e| MyError::new(e, some_context))?;
}

We can still use the question mark operator, the ugly Err(MyError::new(...)) is gone, and we can provide some additional context in our custom Error type. Epic win!

Example 3 - Converting from Option #

This time we are calling a function that returns an Option and we want a Result.

Again, let’s start with the “bad” version:

fn my_func() -> Result<Foo, MyError> {
  let res = bar::return_opt();
  if res.is_none() {
    return Err(MyError::new(....));
  }
  let res = res.unwrap();

  ...
}

The solution is to use ok_or_else, a bit like how we used the unwrap_err before:

fn my_func() -> Result<Foo, MyError> {
  let value = bar::return_opt().ok_or_else(|| (MyError::new(...))?;
  ...
}

Example 4 - Converting to Option #

This time we want to discard the error and simply return None.

Bad example (even with the pattern matching):

fn my_func() -> Option<SomeType> {
  let result = bar::return_res();
  let v = match (result) {
     Err(_) => return None,
     Ok(v) => v
  };
  Some(v)
}

Better example:

fn my_func() -> Option<SomeType> {
  let v = bar::return_res().ok()?;
  Some(v)
}

Example 5 - Assertions #

Sometime you may want to catch errors that are a consequence of faulty logic within the code.

For instance:

let mystring = format!("{}: {}", spam, eggs);
// ... some code here
let index = mystring.find(':').unwrap();

We’ve built an immutable string with format() and we put a colon in the format string. There’s no way for the string to not contain a colon in the last line, and so we know that find will return something.

I reckon we should still kill the unwrap() here and make the error message clearer with expect():

let index = mystring.find(':').expect("my_string should contain a colon");

In tests and main #

Note: I’d like to thank Jeikabu whose comment on dev.to triggered the addition of this section:

Let’s take another example. Here is the code under test:

struct Foo { ... };

fn setup_foo() -> Result<Foo, Error> {
    ...
}

fn frob_foo(foo: Foo) -> Result<(), Error> {
    ...
}

Traditionally, you had to write tests for setup_foo() and frob_foo() this way:

#[test]
fn test_foo {
  let foo = setup_foo().unwrap();
  frob_foo(foo).unwrap();
}

But since recent versions of Rust you can write the same test this way:

#[test]
fn test_foo -> Result<(), MyError> {
  let foo = setup_foo()?;
  frob_foo(foo)
}

Another big win in legibility, don’t you agree?

By the way, the same technique can be used with the main() function (the entry point for Rust executables):

Old version:

fn main() {
    let result = setup_foo().unwrap();
    ...
}

New version:

fn main() -> Result<(), MyError> {
    let foo = setup_foo()?;
    ...
}

Closing thoughts #

When using Option or Result types in your own code, take some time to read (and re-read) the documentation. Rust standard library gives you many ways to solve the task at hand, and sometimes you’ll find a function that does exactly what you need, leading to shorter, cleaner and more idiomatic Rust code.

If you come up with better solutions or other examples, please let me know! Until then, happy Rusting :)

Letting the compiler tell you what to do - an example using Rust

Sat, 12 Jan 2019 12:26:27 +0000

If you’ve ever written code in a compiled language (C, C++, Java, …), you are probably used to compiler error messages, and you may think there are only here to prevent you from making mistakes.

Well, sometimes you can also use compiler error messages to design and implement new features. Let me show you with a simple command-line program written in Rust.

An example #

Here’s the code we have written so far:

use structopt::StructOpt;

#[derive(Debug, StructOpt)]
struct Opt {
    #[structopt(long = "--dry-run")]
    dry_run: bool,
}

fn main() {
    let opt = Opt::from_args();

    let dry_run = opt.dry_run;
    println!("dry run: {}", dry_run);
}

We implemented a --dry-run option using the structopt crate.

Now we want to add a --color option that can have the following values: never, always, and auto.

But structopt (nor clap, which it is based on) does not have the concept of “choice”, like argparse or docopt.

So we pretend it does and we write:

enum ColorWhen {
    Always,
    Never,
    Auto,
}

#[derive(Debug, StructOpt)]
struct Opt {
    #[structopt(long = "--dry-run")]
    dry_run: bool,

    #[structopt(
        long = "--color",
        help = "Whether to enable colorful output."
    )]
    color_when: ColorWhen,
}

fn main() {
  let opt = Opt::from_args();
  let dry_run = opt.dry_run;
  let color_when = opt.color_when;

  println!("dry run: {}", dry_run);
  println!("color: {}", color_when);
}

Note: this is sometimes called “programming by wishful thinking” and can be used in various situations.

Anyway, we try and compile this code and are faced with a bunch of compiler errors.

And that’s where the magic starts. We are going to make this work without opening structopt’s documentation and by only following the advice given by the compiler, one error after the other. Ready? Let’s go!

Error 1 #

color_when: ColorWhen,
   |     ^^^^^^^^^^^^^^^^^^^^^ `ColorWhen` cannot be formatted using `{:?}`
   |
   = help: the trait `std::fmt::Debug` is not implemented for `ColorWhen`
   = note: add `#[derive(Debug)]` or manually implement `std::fmt::Debug`

We do what we are told, and add the #[derive(Debug)] annotation:

#[derive(Debug)]
enum ColorWhen {
  // ...
}

Well, that what easy. Let’s move on to the next error.

Error 2 #

  | #[derive(StructOpt)]
  |          ^^^^^^^^^ the trait `std::str::FromStr` is not implemented for
  |                    `ColorWhen`

The compiler tells us it does not know how to convert the command line argument (a string) into the enum.

We don’t really remember what the FromStr trait contains. We could look up the documentation (but that would be cheating), so instead we write an empty implementation and see what happens:

impl std::str::FromStr for ColorWhen {

}

Error 3 #

Again, the compiler tells us what to do:

not all trait items implemented, missing: `Err`, `from_str`
  --> src/main.rs:10:1
   |
10 | impl std::str::FromStr for ColorWhen {}

missing `Err`, `from_str` in implementation
note: `Err` from trait: `type Err;`
note: `from_str` from trait:
  `fn(&str) -> std::result::Result<Self, <Self as std::str::FromStr>::Err>`

We need an associated type Err, and a from_str() function.

Let’s start with the Err type. We’ll need to tell the user about the invalid --color option, so let’s use an enum with a InvalidArgs struct containing a description:

#[derive(Debug)]
enum FooError {
  InvalidArgs { details: String },
}

Note how the compiler almost “forced” us to have our own error type, which is a very good practice!

Anyway, along with the from_str function.


impl std::str::FromStr for ColorWhen {
    type Err = FooError;

    fn from_str(s: &str) -> Result<ColorWhen, FooError> {
        match s {
            "always" => Ok(ColorWhen::Always),
            "auto" => Ok(ColorWhen::Auto),
            "never" => Ok(ColorWhen::Never),
            _ => {
                let details = "Choose between 'never', 'always', 'auto'";
                Err(FooError::InvalidArgs { details: details.to_string() })
            }
        }
    }
}

Error 4 #

error[E0599]: no method named `to_string` found
  for type `FooError` in the current scope

All custom error types should be convertible to strings, so let’s implement that:

impl std::string::ToString for FooError {
    fn to_string(&self) -> String {
        match self {
            FooError::InvalidArgs { details } => details.to_string(),
        }
    }
}

It compiles!

Let’s check error handling:

$ cargo run -- --color foobar
error: Invalid value for '--color <color_when>':
  Choose between 'never', 'always', 'auto'

Let’s check with a valid choice:

$ cargo run -- --color never
dry run: false
color: Never

It works!

The default #

There’s still a small problem: we did not use a Option for the color_when field, so the --color command line flag is actually required:

$ cargo run
error: The following required arguments were not provided:
    --color <color_when>

Can’t blame Rust there. That’s our fault for not having used an optional ColorWhen field in the first place.

Let’s try and fix that by using an Option<> instead:

// ...
struct Opt {
    // ...
    #[structopt(
        long = "--color",
        help = "Whether to enable colorful output"
    )]
    color_when: Option<ColorWhen>,
}

Well, since we did not do anything with the opt.color_when but print it, everything still works :)

Error 5 #

Had we tried to use the option directly like this:

fn force_no_color() {
  // ...
}

fn main() {
  let color_when = opt.color_when;

  match color_when {
    ColorWhen::Never => force_no_color(),
    // ..
  }

The compiler would have told us about our mistake:

ColorWhen::Never => force_no_color(),
^^^^^^^^^^^^^^^^ expected enum `std::option::Option`, found enum `ColorWhen`

And we would have been forced to handle the default value, for instance:

let color_when = color_when.unwrap_or(ColorWhen::Auto);

Side note #

There’s an other cool trick we can use to achieve the same result, by leveraging the default trait:

#[derive(Debug)]
enum ColorWhen {
    Always,
    Never,
    Auto,
}

impl std::default::Default for ColorWhen {
    fn default() -> Self {
        ColorWhen::Auto
    }
}

And calling unwrap_or_default() instead of unwrap_or():

fn main {
    let color_when = opt.color_when.unwrap_or_default();
}

The code is a bit longer but I find it more readable and more “intention revealing”.

(End of side note)

Conclusion #

I hope this gave you new insights about what a good compiler can do, or at least a feel of what writing Rust looks like.

I call this new workflow “compiler-driven development” and I find it nicely complements other well-known workflows like TDD.

Final note: to be honest we could have achieved better results by reading the documentation too: for instance, we could have used a custom string parser instead of the FromStr boilerplate, and implemented the Display trait on our custom error instead. Good docs matter too …

Cheers!

I am writing a book about CI

Fri, 11 Jan 2019 10:58:18 +0000

Why? #

I wanted to talk about CI for a very long time (more than 2 years). But it’s a huge topic, and it took me some time before I decided to start a blog post series about it.

But then I realized that was the wrong format: you see, writing and publishing one post at a time requires for you to know exactly what to say and in what order, and I found that was not the case for the “Tips From A Build Farmer” series. So instead, I am going to try and write the whole thing as a book.

How? #

For now, I am only focusing on the content: I just have a bunch of Markdown files I can turn into a big pdf thanks to pandoc.

Once done, I will probably publish the pdf and the sources on the Internet, available for free.

When? #

I have no clue at all, this is my first attempt at writing a book! But I will surely post updates there on the blog – wish me luck!

A definition of the Linux desktop

Wed, 02 Jan 2019 09:56:30 +0000

Quite some time ago, I came across a thread from Gary Bernhardt on twitter.

Here’s what he had to say about Linux, compared to macOS and Windows:

Unlike the other two operating systems, you have control. But nothing will ever work reliably, especially across upgrades. When it breaks, the community will frame it as your own moral failure.

As a guy who uses Linux for almost everything (both at work and at home), you may think I would disagree.

But no, I agree with almost everything, except I would rephrase the last sentence. Let me explain.

Control #

Let’s start at the beginning. What kind of control do you have depending on the operating system you use?

Control on macOS #

You have almost no control over hardware. You buy hardware that is manufactured by Apple, often in an “Apple store”.

Most applications have minimal configuration, simple things like preventing the laptop to suspend when closing the lid are hard to achieve ¹.

Finally, you often haven little control over your own data. Watching the movie you bought on iTunes on an non-Apple device is very hard.

You can put all your stuff on iTunes or iCloud, but if you loose access to your account you lose everything. ²

Control on Windows #

Compared to macOS, you have more control over hardware. For instance, you can buy a new battery when the one you use is failing instead of buying a completely new device.

But still, you get less and less control over what to update and when, and let’s not talk about privacy…

Control on Linux #

You control (almost) everything.

You can choose your distribution. You can choose its stability. For instance on Debian, you can choose between stable, testing, or unstable.

You can chose your desktop environment: KDE, Gnome, LXDE, XFCE, Mint, or even use no desktop environment at all.

You can can choose your window manager, your init system, your package manager …

You can choose when and what to update (less so on “rolling distributions” like Arch Linux, although it is still possible with extra care …), etc.

Control is everywhere and so ubiquitous it’s even scary. But it’s free: nothing prevents you from experimenting but time and energy.

When it breaks #

Let’s talk about what happen when software you are using breaks.

On macOS and Windows, all you can do is pray and find someone who has a solution. There are bug trackers of course (but on macOS you can only see the bugs you opened).

If you upgrade Windows and a third-party driver stops working, you’re basically screwed. I’ve seen my Mom loose access to its scanner for a few months because of this.

On Linux, things will break more often. They are less people to do QA, there are more unknowns, tons of different configurations (as we saw in the “control” section), and a pressure to release often and stay “bleeding edge”.

Almost everything is written and maintained by non-paid volunteers (and they do a terrific job by the way).

Upgrades are big and scary, and so things often break. No point discussing that.

A few months ago, after I upgraded Arch Linux on my laptop, the brightness control button ceased to work and I have no clue why.

But to be honest, I did not take the time to look for a solution, which leads us to the next section.

Your own moral failure #

Let’s take a look at some exchanges between an average user with a problem, looking for answer from the community, for instance in a public bug tracker or a forum:

First example:

> Q: I've upgraded *foo* to version 3.0 and now I cannot frob anymore
> A: Did you read the changelog before upgrading?

Second example:

> Q: I bought a new printer and now I can only print in black and white!
> A: Did you research compatibility with Linux before buying the new printer?

Let’s try to put ourselves in the shoes of those users. They might think they failed at something, or even that they are not even supposed to ask this kind of question³, which illustrates with the “moral failure” Gary talked about.

Rephrasing the last part #

So where do I disagree? Well, it’s the “framing” part I don’t like. I would rephrase the last sentence as: “When it breaks, it will be your (or the community’s) responsibility to fix it”.

Let me elaborate.

When stuff on Linux breaks, you can do something about it.

You can downgrade, you can switch to an other distribution, you can discuss the problem on a forum, you can send e-mails to mailing list, you can open an issue on the bug tracker.

You can even get the code and patch it yourself if you are not afraid. Sometimes it’s not that hard.

So I don’t think the community is framing it. I just think it’s how things work in the Linux world. You just can’t expect volunteers to always fix all the bugs for free. And you can’t expect the same level of quality as when QA is done by paid professionals.

So, yes, in exchange for the freedom of control you get some duties. Your duty is to help yourself and others when things break. And by the way, teaching users about reading changelogs and doing research before buying hardware is good advice, if they are willing to accept it.

For me those duties are a small price to pay because I value control and freedom more than convenience. If you are willing to accept those duties to gain this freedom and are not using Linux, give it a go!

If you are using Linux and want to “convert” other people, (like friends or family), explain this to them so they don’t get mad at you. Be ready to help them for things they don’t understand. You can even teach them a few things, or teach them how to learn. That’s how I got to use Linux in the first place almost 15 years ago.

And if you don’t want to try and fix stuff when it breaks, keep using macOS or Windows. There’s nothing wrong with that.

Cheers!

I spent a day looking for a solution, then decided I did not really needed it. But if you know how to do it, please let met know :) ↩︎
Ask duckduckgo if you think it’s a rare problem … ↩︎
This is not hypothetical. I’ve talked about that in an article called “Why I love Arch Linux” ↩︎

ruplacer: find and replace text in source files

Sun, 14 Oct 2018 12:34:43 +0000

Introduction #

Today I’d like to talk about a command-line tool I’ve been working on.

It’s called ruplacer and as the name suggest, it’s rually cool and written in Rust.

Basically, it finds and replaces text in source files. Here’s a screenshot of ruplacer in action:

Some nice features:

Skips files listed in .gitignore, as well as binary files
Runs in dry run mode by default (use --go to actually write the changes to the filesystem)
Defaults to searching the current working directory, although an other path maybe specified after the pattern and the replacement.
Uses Rust regular expressions, which means you can capture groups in the pattern and use them in the replacement. For instance:

# Replaces dates looking like DD/MM/YYYY to YYYY-MM-DD
$ ruplacer '(\d{2})/(\d{2})/(\d{4})' '$3-$1-$2'

Can be run with --no-regex if the pattern is just a substring and should not be used as a regular expression
Last, but not least there’s also a --subvert mode, which allows you to perform replacements on a variety of case styles:

$ ruplacer --subvert foo_bar spam_eggs
Patching src/foo.txt
-- foo_bar, FooBar, and FOO_BAR!
++ spam_eggs, SpamEggs, and SPAM_EGGS!

How it works #

Here’s how it works:

First, we build a structopt struct for the command-line arguments parsing. Depending on the presence of the --subvert or --no-rexeg flags, we build a Query, which can be of several types: Substring, Regex or Subvert.

Then we leverage the ignore crate to walk through every file in the source directory while skipping files listed in .gitignore. By the way, the ignore crates comes directly from ripgrep, an awesome alternative to grep also written in Rust.

Along the way, we build a FilePatcher from the source file and the query. The FilePatcher goes through every line of the file and then sends it along with the query to a LinePatcher.

The LinePatcher runs the code corresponding to the query type and returns a new string, using the Inflector crate to perform case string conversions if required.

Finally, if the string has changed, the FilePatcher builds a Replacement struct and pretty-prints it to the user. While doing so, it also keeps a record of the modified contents of the file. Finally, if not in dry-run mode, it overwrites the file with the new contents.

And that’s pretty much it :)

The idea of ruplacer started almost a decade ago when a colleague of mine showed me a shell function called replacer (thanks, Cédric!) It was basically a mixture of calls to find, sed and diff. You can still find it online.

Because I wanted better cross-platform support, a dry-run mode and a colorful output, I rewrote it in Python a few years ago. Along the way, the features, command line syntax and the style of the output changed quite a lot, but I’ve been using it regularly for all this time.

Finally, after hearing about ripgrep and fd, I decided to give Rust a go, and that’s how ruplacer, the third incarnation of this tool, was born. This makes me confident it’s good enough for you to try.

If you have cargo installed, you can get ruplacer by running cargo install ruplacer. Otherwise, you will find the source code and pre-compiled binaries on GitHub.

Cheers!

Syntax Highlighting Is Useless

Thu, 04 Oct 2018 13:30:31 +0000

… at least when it comes to Markdown files ;)

Introduction: editing this blog in Neovim #

Here’s what it used to look like to edit one of my articles in Neovim:

Take a quick look. Do you see what’s wrong?

First, the line about the summary in the front matter is colored in green. The front matter uses YAML syntax, but Neovim does not know it and thus highlights this line incorrectly.
Second, the python code is OK, but the language name is misspelt: pyton instead of python. I am using Neovim’s built-in spell checker, but since Neovim “knows” the stuff after ``` is source code, it disables spell checking for this part of the text.
Third, the link to ‘Using CMake and Ninja’ is built using hugo syntax, and if you were extra careful you may have noticed that the path to the article after ref starts with a double quote ("), but ends with a single quote (').

Turning syntax highlighting off #

For quite some time I did not think to much about this situation, until I read A case against syntax highlighting, by Linus Åkesson.

So one day I decided to try and turn the syntax highlighting off:

This time, the spell checker checks everything, and the typo in pyton is caught.
The “rendering bug” of the front matter is gone.
Also, he problem with the single quote is more obvious to spot. It’s less likely to be distracted by all the colors. That one may be subjective: it’s what my guts tell me, though.

One more thing: I’m also viewing the rendered HTML in a browser next to my Neovim window. Hugo rebuilds the page I’m editing after each save, and the browser automatically reloads.

This has several consequences:

First, some parts of the highlight are redundant. I’m able to see the italics in the browser, I don’t need to also view them as such in the editor.
Second, the problem with the {{ref usage gets caught by the hugo process itself:

ERROR 2018/10/04 16:36:23 error processing shortcode
  "_internal/shortcodes/ref.html"
  for page "post/0087-demo-syntax-highlight.md":
    template: _internal/shortcodes/ref.html:1:89:
    executing "_internal/shortcodes/ref.html" at <0>:
    invalid value; expected string

So let’s think about what we learned:

The syntax highlighting is buggy, and it will never be as good as the HTML rendering. (Especially with Markdown, where each rendering tool has its own quirks).
It “hides” important information, like spelling errors

So, why do we keep using it?

Conclusion #

For now, I’m just turning syntax off for Markdown files. Here’s the relevant section in my init.vim file:

augroup textfiles
  autocmd!
  autocmd filetype markdown :setlocal spell spelllang=en | syntax clear
augroup end

I’m still using syntax highlight in the other cases.

But, in the same way the syntax highlighting may be useless (and even harmful) for editing Markdown when a HTML renderer can be used, maybe syntax highlighting is also useless if a linter can be used directly to check the code while it’s being typed.

Food for thought …

Let's Build Chuck Norris! - Part 8: Using C++ in an iOS application

Tue, 18 Sep 2018 16:53:30 +0000

Note: This is part 8 of the Let’s Build Chuck Norris! series.

If you came this far after reading part 1 to 7, congrats!

In this article, we’ll use everything we learnt so far and write an iOS application able to show Chuck Norris facts. We’ll also learn a few things specific to iOS and Xcode. Let’s dive in!

Introduction: CocoaPods #

There are many ways to achieve our goal. Several tools exist, but for this post we’ll concentrate on CocoaPods. ¹

CocoaPods can be used in two modes:

First, you can run pod lib create foo. This will create a Foo.podspec file. The podspec describes how to build the foo library, a bit like a Conan recipe. You can then upload the podspec file and the associated sources to a repository.

Second, you can run pod init next to an existing Xcode project. This will create an Xcode workspace and a Podfile file. You can specify one or several dependencies in the Podfile. Then, when you run pod install, the podspecs will be fetched and the workspace will be configured so that you can build and use those dependencies in the original Xcode project.

That way you minimize the interactions with Xcode through its GUI and use code instead.

So, here’s the plan:

Create a CocoaPods library called ChuckNorrisBindings in ios/bindings/ChuckNorrisBindings.
Create a blank iOS application with in the ios/app/ChuckNorris directory, called ChuckNorris.
Use CocoaPods to create a dependency between ChuckNorrisBindings and ChuckNorris.
…
Profit!

The Bindings #

cocoapods is written in Ruby and can be installed with gem:

gem install cocoapods

We run pod lib create ChuckNorrisBindings and answer a few questions:

$ pod lib create ChuckNorrisBindings
Cloning `https://github.com/CocoaPods/pod-template.git` into `ChuckNorrisBindings`.
...
What platform do you want to use? [ iOS / macOS ]
> ios
What language do you want to use? [ Swift / ObjC ]
> ObjC
Would you like to include a demo application with your library? [ Yes / No ]
> No
Which testing frameworks will you use? [ Specta / Kiwi / None ]
> Specta
Would you like to do view based testing? [ Yes / No ]
> No
What is your class prefix?
CK
...
Running pod install on your new library.
...
Analyzing dependencies
Fetching podspec for `ChuckNorrisBindings` from `../`
Setting up CocoaPods master repo
...
 Ace! you're ready to go!
 We will start you off by opening your project in Xcode
  open 'ChuckNorrisBindings/Example/ChuckNorrisBindings.xcworkspace'

You will note CocoaPods has created lots of files. Among them:

ChuckNorrisBindings/Sources: the sources of the library.
ChuckNorrisBindings.podspec: the podspec used to build and depend on the ChuckNorrisBindings library
Example/ChuckNorrisBindings.xcodeproj the Xcode project to build the sources
Example/ChuckNorrisBindings.xcworkspace the Xcode workspace which we can use to:
- build the library
- build the test framework
- build and run the tests themselves.

If we try to run the tests directly from Xcode, it won’t work right away, instead we get this error message:

Scheme "Pods-ChuckNorrisBindings_Tests" is not configured for testing.
Edit the scheme to enable testing, or cancel the action.

We have to fiddle with the schemes CocoaPods generated for us:

Open the scheme editor:

Show the test scheme:

Click the + bottom at the bottom and select the test target.

This time we get an other error message:

Could not find test host for ChuckNorrisBindings_Tests ...

Edit the project settings, under “General” and switch the Host application form “Custom” to “None”:

And it works: a simulator is started and the tests run ².

Cross-compiling ChuckNorris for iOS #

Our plan to bind the C++ library for iOS is a combination of techniques we already seen in parts 5 and 6.

We’ll cross-compile ChuckNorris as a static library from macOS to iOS. And then we’ll compile the Objective-C code by giving it the paths to the libchucknorris.a file, the conan dependencies, and the chucknorris.h C header.

Before anything, let’s make sure Xcode command line tools are installed by running xcode-select --install. If you are following this tutorial at home, make sure to run this or nothing will work.

Let’s create a conan profile called ios and add a build dependency:

# In ~/.conan/profiles/ios
[settings]
os=iOS
os.version=9.0
arch=x86_64
compiler=apple-clang
compiler.version=9.0
compiler.libcxx=libc++
build_type=Release
os_build=Macos

[build_requires]
darwin-toolchain/1.0@theodelrieu/stable

For now, we’ve hard-coded the x86_64 architecture, because we’ll run everything in simulators, and Xcode simulators require this architecture. In order to run the code on actual devices, we’ll invoke conan with the correct --settings arch=<arch> flag. This is similar to what we did in Part 6.

Note that this time we don’t need to write the toolchain recipe ourselves, there is already one on conan-center.³

Then, as we did for Android, we run conan create for sqlite3.

cd conan/sqlite3
conan create . dmerej/test --profile ios --settings arch=x86_64

As we did for Android with the libc++shared.so file, we patch the ChuckNorris recipe to deal with the copies of all the .a files, both in the imports() and package() methods:

def imports():
      if self.settings.os == "Android":
          self.copy("*libc++_shared.so", dst="lib", keep_path=False)
      if self.settings.os == "iOS":
          self.copy("*.a", dst="lib", src="lib")

def package(self):
      self.copy("bin/cpp_demo", dst="bin", keep_path=False)
      self.copy("lib/*.so", dst="lib", keep_path=False)
      self.copy("lib/*.a", dst="lib", keep_path=False)

Then we can create the ChuckNorris package:

cd cpp/ChuckNorris
conan create . dmerej/test --profile ios --settings arch=x86_64

So far so good.

The podspec #

Note that all the .a files are somewhere inside ~/.conan/data/. Let’s copy them into a nativelibs/x86_64 directory next to the bindings code:

mkdir -p nativelibs/x86_64/
cp \
  ~/.conan/dataChuckNorris/0.2/dmerej/test/package/<id>/*.a
  nativelibs/x86_64/

So that we don’t forget, let’s add nativelibs to the .gitignore.

Next we can edit the podspec to specify:

The include directory: it’s inside a directory called pod_target_xcconfig.
The vendored libraries: that’s our libchucknorris.a and libsqlite3.a. They are called “vendored” because they won’t be compiled by CocoaPods itself, and are not already present on the target operating system either.

Pod::Spec.new do |s|
  ...

  s.pod_target_xcconfig = {
    'HEADER_SEARCH_PATHS' => "${POD_ROOTS}/../../cpp/ChuckNorris/include/",
  }

  s.vendored_libraries = Dir["out/*.a"]

Two remarks:

Since CocoaPods recipes are written in Ruby, we can use the overloaded [] operator for Dir objects to get the full list of files matching the out/*.a glob pattern.
The HEADER_SEARCH_PATHS string contains a ${POD_ROOTS} extension, that will be set by CocoaPods.

Note how it does not matter if we are building the ChuckNorrisBindings CocoaPods library or the ChuckNorris application: the resulting HEADER_SEARCH_PATHS will be the same.

Anyway, we can now run pod update from the Example directory to update the Xcode projects and workspace configurations.

Objective-C #

We can use C code directly in Objective-C.

It’s still dangerous to expose C code directly, so here’s how we can proceed:

Write a CKChuckNorris class. (It’s a convention to prefix all the classes in a pod library with the initials of the projects)
Write a CKChuckNorris+Private category to hide the C code from the consumers of the CKChuckNorris class.

/* In CKChuckNorris.h */
@interface CKChuckNorris: NSObject

@property void* ckPtr;

-(instancetype)init;
-(NSString*) getFact;

@end

/* In CKChuckNorris.m */
#import "CKChuckNorris.h"
#import "CKChuckNorris+Private.h"

@implementation CKChuckNorris

-(instancetype)init {
  self = [super init];
  return [self createCkPtr];
}

-(NSString *)getFact {
  return [self getFactImpl];
}

@end

/* In CKChuckNorris+Private.h */
#import "CKChuckNorris.h"

@interface CKChuckNorris (Private)

-(instancetype) createCkPtr;
-(NSString*) getFactImpl;

@end

So far we have just declared the createCKPtr and getFactImpl methods.

It’s time to define them in the CKChuckNorris+Private.m file:

/* In CKChuckNorris+Private.m */
#import "CKChuckNorris+Private.h"

#include "chucknorris.h"   // <- Our C header file

@implementation CKChuckNorris (Private)

-(instancetype)createCkPtr  {
  self.ckPtr = chuck_norris_init();
  return self;
}

-(NSString*)getFactImpl {
  const char* fact = chuck_norris_get_fact(self.ckPtr);
  return [NSString stringWithCString:fact encoding:NSUTF8StringEncoding];
}
@end

Note how the only file that depends on the chucknorris.h C header file is the private implementation of the CKChuckNorris class.

There’s a similar technique known as “PIMPL” (pointer to implementation) you can use in C++ code to achieve the same kind of isolation.

It’s now time to edit the tests:

/* In Tests.m */
describe(@"ChuckNorris", ^{

  it(@"can get version", ^{
    expect([CKChuckNorris versionString]).to.equal(@"0.1");
  });

  it(@"can get a fact", ^{
    CKChuckNorris* ck = [[CKChuckNorris alloc] init];
    NSString* fact = [ck getFact];
    expect(fact).toNot.beEmpty();
    NSLog(@"Got fact: %@", fact);
  });


});

And see if they pass:

Undefined symbols for architecture x86_64:
  "operator delete(void*)", referenced from:
      _chuck_norris_version in libchucknorris.a(c_wrapper.cpp.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Whoops, we forgot to add the dependency to the C++ library. It’s less hard than in the Android case, since they always have the same name for iOS:

Pod::Spec.new do |s|
  ...

  s.vendored_libraries = ...
  s.libraries = ['c++', 'c++abi']
end

libc++ and libc++abi are found on the target operating system, hence we use the libraries variable.

We can now try again, and this time the tests pass \o/.

The GUI #

We create the Objective-C application from Xcode.

Then we edit the main storyboard to add a stack view, a text field and a button:

We drag and drop the components from the designer view to the code, while keeping the ctrl key pressed.

For now, we’ll just set the text view to the string Hello when the button is clicked:

/* In ViewController.m */
- (IBAction)onClick:(id)sender {
  self.textView.text = @"Hello";
}

OK, that works.

Now we can run pod init and edit the Podfile:

target 'ChuckNorris' do
  pod 'ChuckNorrisBindings', :path => '../bindings/'
end

We took a small shortcut here. Rather than deploying the ChuckNorrisBindings to a spec repository, we just tell the Podfile to get the podspec directly from the file system.

Let’s check this works:

$ cd ios/app
$ pod update
...
Downloading dependencies
...
Sending stats

Looks OK.

All that’s left to do is add a CKChuckNorris* pointer to the controller, set it in viewDidLoad and call the getFact() method when the button is clicked:

/* In ViewController.h */
#import <UIKit/UIKit.h>
#import "CKChuckNorris.h"

@interface ViewController : UIViewController

@property CKChuckNorris* ck;

@end

- (void)viewDidLoad {
  [super viewDidLoad];
  self.ck = [[CKChuckNorris alloc] init];
}

-(IBAction)onClick:(id)sender {
  self.textView.text = [self.ck getFact];
 }

We can now run the application inside a simulator:

Yippee!

Dealing with the other architectures #

Let’s try to build an .ipa, i.e an application file that can be installed on real devices.

The first step is to generate an archive of our application. To do this, we can use the “Archive” entry in the “Product” menu.

Right now it’s disabled, but we can fix this by selecting the “Generic iOS Device” next to the play/stop buttons in the bottom left.

Let’s click on it:

ld: warning: ignoring file /nativelibs/x86_64/libchucknorris.a, file was built
  for archive which is not the architecture being linked (arm64)
error: Undefined symbols for architecture arm64:
  "_chuck_norris_version", referenced from:
  +[CKChuckNorris(Private) versionStringImpl] in
    libChuckNorrisBindings.a(CKChuckNorris.o)

You read this right: the fact the .a was not build for the correct CPU architecture is a warning, and after that we get a generic error about the symbol being not found. The morale of the story is: when you have a compilation failure, start at the top of the compiler output :)

So, let’s try and add support for Arm 64bits. We can start by cross-compiling sqlite3 like so:

cd sqlite3
conan create . dmerej/test --profile ios --settings arch=armv8

Note that Conan and Xcode use different names for this architecture (armv8 and arm64 respectively).

Now, for the x86_64 architecture we manually copied files from .conan/data to nativelibs/x86_64. This is not ideal. Instead, we can use conan install (to install the deps), then conan build (to build the ChuckNorris library), and finally conan package to copy the libraries and headers (to run the code that was in the package() function of the recipe, remember?):

$ cd cpp/ChuckNorris
$ conan install . --profile ios --settings arch=armv8 \
    --install-folder build/ios/armv8
$ conan build . --build-folder build/ios/armv8
$ connan package . --build-folder build/ios/armv8 \
    --package-folder nativelibs/armv8

Note how the “install folder” for conan install matches the “build folder” for conan build and conan package.

Also note how the profile and settings are only used by conan install. After this, everything conan needs to know can be found in the build directory.

Let’s do the same for ‘armv7’, ‘armv7s’, and ‘x86’. Armv7 and armv7s are architectures used by older iOS devices, and x86 is used by 32 bits simulators.

Since this is a bit tedious by hand, we’ll use the build_ios.py script. ⁴

Now, instead of trying to build the ChuckNorrisBindings several times, once per architecture, (which is hard to do when using CocoaPods), we can use fat libraries instead. Fat libraries are just like regular libraries, except they can contain code for multiple architectures.

So let’s create a nativelibs/universal folder containing fat liqsqlite3.a and fat libchucknorris.a libraries:

cd nativelibs/universal
lipo -create -output libsqlite3.a ../*/lib/libsqlite3.a
lipo -create -output libchucknorris.a ../*/lib/libchucknorris.a

We can check with lipo -info that this worked:

$ lipo -info libchucknorris.a
Architectures in the fat file: libchucknorris.a are:
  armv7 armv7s i386 x86_64 arm64

Then we fix the podspec:

- s.vendored_libraries = Dir["nativelibs/x86_64/*.a"]
+ s.vendored_libraries = Dir["nativelibs/universal/*.a"]

We re-generate the Xcode project with pod update --ro-repo-update. This time the “Archive” step triggers no errors, and we are faced with this window:

Signing the archive #

We now need to build a re-distributable application from the archive. The depends on your Xcode profiles and certificates. Here’s what worked for me, your mileage may vary:

Select “export” on the right pane
Choose “Ad Hoc” method of distribution
Then tick the “Rebuild from Bitcode” option.
Let Xcode do the signing and save the .ipa in a folder named based on the current date.

Finally, go to Xcode “Window” menu, select “Devices”, and drag and drop the “.ipa” from the date folder to the list of applications installed on this device.

And here’s the result (taken from my crappy Android phone):

Victory!

Conclusion #

That’s the end of the Chuck Norris series. I hope you had fun, I hope you learned something new. See you next time!

I heard Carthage is also a good option. Did not try it, though. ↩︎
That is, if you already generated a signing certificate. If not, follow the instructions printed in the logs. ↩︎
This recipe was written and shared by my nice colleague Théo Delrieu from tanker.io. Say thanks! ↩︎
Which is not far from the CI script we use at work by the way. ↩︎

Comparing directories: a case study

Sat, 08 Sep 2018 12:36:50 +0000

The problem #

Let’s say we have a directory called O that contains a large number of subdirectories and files.

We find ourselves with two different directories, A and B. Each of them started as a copy of O, but then, in each of them, some files were changed, some were added or removed, and we don’t know which ones.

We want to compare the contents of A and B, for instance to merge them back into O.

There are several ways to solve the problem, depending on the situation. Let’s take a look.

Using a GUI #

For our first case, let’s assume A and B are on our development computer and we have a graphical session opened.

We can use many GUI tools for this task, here’s an example with Kdiff3:

On the top left pane we have an overview of all the differences between the A and B directories, and we can see that:

The bar/eggs and bar/spam directories are identical between A and B
There’s a new file named bar/new.txt in A
There’s a new file named baz/six.txt in B
The file foo/a.txt was changed both in A and B

On the bottom panes, since a.txt is selected, we can see the differences in the contents.

Using the command line #

Now let’s assume A and B are both on a remote server running Linux¹ and we can’t open a remote graphical session there.

What are our options?

Well, we could use a coreutils command.

Coreutils is a package which is both very old and present in almost any Linux operating system. It contains a collection of useful, small command line programs. Even when it is not installed, an other package (like busybox) is usually here to provide similar functionality.

Anyway, there is a coreutils command named diff we can use for the task at hand:

$ diff --recursive a b
Only in a/bar: new.txt
Only in b/baz: six.txt
diff --recursive a/foo/a.txt b/foo/a.txt
1c1
< this is a
---
> this is modified a

Boom! All the info we need.

diff offers lots of ways to tweak its behavior². For instance, we can get more succinct info with the --brief option:

$ diff --recursive --brief a b
Only in a/bar: new.txt
Only in b/baz: six.txt
Files a/foo/a.txt and b/foo/a.txt differ

By the way, if git is installed on the remote server, we can also use it, with nice options like --stat or --word-diff³:

$ git diff --stat a b
 a/bar/new.txt => /dev/null | 1 -
 /dev/null => b/baz/six.txt | 1 +
 {a => b}/foo/a.txt         | 2 +-
 3 files changed, 2 insertions(+), 2 deletions(-)

Remote directories #

Now let’s assume A and B are on two different remote servers: let’s call them abbot and costello respectively.

Also, let’s assume A and B contains lots of big files, so we cannot copy the A directory from abbot to costello and use the previous technique. We can, however, transfer small files between the two servers. Here’s we can do.

First, we log in to abbot:

ssh user@abbot

Then, we go inside the a directory:

cd a

Then we run the following command:

find . | sort > ~/manifest-a

Explanation:

find . lists all the files and directories inside the current working directory.
The order of the files returned by find are not deterministic and can change form one file system to another, so we use a pipe (|) to take the output of find and pass it to the sort command. (sort is also part of the coreutils package).
Finally, we use an angle bracket (>) to write the output of sort to a file in the home directory: (~/manifest-a): we must not write the manifest-a file inside the a directory, otherwise the manifest may contain itself!

Now, we do the same on costello:

ssh user@costello
cd b
find . -type f | sort > ~/manifest-b

Then we use scp to transfer the A manifest to costello:

# On abbot
scp manifest-a user@costello:

Now we have two text files on costello, both containing a list of all the files. So, we use diff on the manifest files themselves:

diff ~/manifest-*
...
-./bar/new.txt
...
...
+./bar/six.txt

That takes care of the contents of the directories. What about contents of the files themselves ?

Well, we can use another coreutils tools called shasum.

shasum can be used to compute a checksum of the contents of a given file. It will generate the same output if the contents are the same, and if two files are different, there’s almost zero chance their checksum will be equal. ⁴

We can call shasum with a list of files as arguments, like this:

shasum file1.txt file2.txt

Here we need to call shasum with the whole list of files, so we call find . again but with the -type f option to select only the files and ignore the directories. This gives us a list of lines, so we use xargs (also in coreutils) to convert them into a list of command line arguments⁵, and finally write the result in a a.shasum file:

find . -type f | xargs shasum > ~/a.shasum

Here are the contents of the a.shasum file:

791c4ba196e0faea35e0c5fbe46e64da  bar/eggs/two.txt
23f4a2592b2e4dee0444983a6e53c23e  bar/new.txt
...

Now we can transfer the a.shasum file to costello:

$ scp a.shasum user@costello:

Finally, we can use shasum again with the --check option: it will read each line, parsing the file name and expected checksum, then compute the actual checksum for the given file and check if it matches the expected value:

$ cd b
$ shasum --check ~/a.shasum
./bar/eggs/two.txt: OK
./bar/spam/one.txt: OK
...
./foo/a.txt: FAILED
...
shasum: WARNING: 1 of 5 computed checksums did NOT match

And we’re done: from the differences of the manifests we know which file are missing or were added, and from the list of checksums we know which files differ.

Conclusion #

We saw that we can use nothing but command line tools from the coreutils package to compare the contents of two directories: the technique we used was working exactly the same way several decades ago, and will probably continue to work for a long time.

So, if you liked the techniques show here, here’s my advice to you: next time you are confronted with a task similar to the one we just studied, take a look at all of the coreutils documentation and try using them. If you do this often, after a while you’ll have gained a nice addition to your skill set.

Oh, and if you think “this is all useless”, please read this enlightening paper called Command-line Tools can be 235x Faster than your Hadoop Cluster :)

Cheers!

I hear there are people running operating systems other than Linux on remote servers. I’m sorry if this is the case for you. ↩︎
If you’d like to see all of what diff can do, feel free to run man diff and be amazed! ↩︎
Git is smart enough to see that neither A and B are inside a directory it controls, and if A and B were inside a git working tree, you could use the --no-index option, as explained in the documentation ↩︎
You may already have heard of md5 sum: it works exactly the same way but uses an older algorithm to compute the checksum and its usage is generally discouraged. ↩︎
If you use zsh you can also ask it to list all the files directly with a glob extension, like this: shasum **/*(.) > ~/a.md5. Yeah zsh! ↩︎

Tips From a Build Farmer - Part 2: Some concepts

Tue, 28 Aug 2018 18:17:04 +0000

Note: This is part 2 of the Tips From a Build Farmer series.

In this article we’ll define a few concepts that we’re going to use in the rest of this series.

But first, let’s look at some Java code! ¹

Fruits and apples #

class Fruit {
  // ...
}

class Apple extends Fruit {
	private final String type;

	public Apple(String type) {
		this.type = type;
	}
}

class Foo {

	void func() {
		List<Fruit> fruitList = new ArrayList<Fruit>();
		Apple golden = new Apple("golden");
		fruitList.add(golden);
	}
}

It’s clear that the Apple class is not really an apple. It’s a bunch of data and methods that identifies an apple via a type string. We can say that the Apple class is a metaphor for the real apple object.

But note that fruitList is not a list of fruits! It’s a slice of memory containing objects of a certain type, and if you forget that, you’re about to get nasty compilation failures such as this one:

Apple firstFruit = fruitList.get(0);

foo.java:21: error: incompatible types: Fruit cannot be converted to Apple

You can solve that with generics of course, but this is not my point.

My point is that as programmers we are surrounded with metaphors all day long, and sometimes without realizing it. After all, all we do is write text that ultimately is converted to a sequence of zeros and ones and ran on a piece of silicon.

We create abstractions and encapsulations to be able to reason about the code we write, and we often do that with metaphors.

We also really care about naming. For instance, you could argue the type field of the Apple class should be named variety instead, because this word already has several meanings for any developer.

But metaphor are not only used by developers.

Why you should care #

If you read about Agile or XP, you will find out there’s a programming practice called System Metaphor.

It goes with a bunch of other disciplines such as Test Driven Development, Sustainable Peace, and Collective Code Ownership. You can find the whole list on Wikipedia.

Among programmers we sometimes don’t think too much about metaphors because we can still look at the actual code implementation (or comments) to figure out what the name means.

Things change when we need to communicate with people outside the team, such as Product Owners, Marketing or Sales. We use metaphors when we talk to them about features or components of our systems.

Thus, coming up with good metaphors is the only way to communicate effectively between business and tech teams. We have to come up with a common vocabulary that everyone agrees on.

This means, among other things:

Avoiding using the same word for different things
Making sure each word means the same thing for everyone.

Anyway, this is why I wanted to talk about metaphors. The concepts I’ll be listing below can also be used as such to communicate between people both inside and outside your team.

Concepts #

The concepts are listed in a top-bottom manner. So if you encounter a term you don’t understand, just keep reading!

Continuous Integration #

Continuous Integration, or CI for short, is any situation where there exist scripts running on a build farm.

CI Scripts #

The scripts are usually written in an “scripting language” (hence there name).

They have access to some or all the source code and can do many things: compiling, running some tests, produce deliveries, deploy new servers, and so on ².

They can:

Run automatically upon certain events (like a merge request being created)
Run automatically on specific hours (for instance, everyday at 9 AM)
Be triggered by a manual action (like running a separate program or using a Web interface).

Build Farm #

A build farm is made of two things. A set of runners, and a coordinator.

There may be just one runner, and the coordinator may be running on the same machine as the runner, it does not matter; what matters is that runners and coordinators do different things.

Runner #

A runner is any machine on which the scripts run. That’s all.

Coordinator #

A coordinator:

Listens to the triggers listed above,
Dispatches the execution of the scripts on one or several machines among the runners,
Aggregates the results of those executions. Those can be test reports, logs, files, etc. …

Note that if you’re using Jenkins, the coordinator is called master, and the runners are called nodes or slaves.

Job #

A job is a script plus a configuration. The configuration contains at least the specifications for the triggers, but may also contain mappings of keys and values.

Build #

A build is the execution of a job.

Usually, builds have a number, and jobs have a name.

P.S: I know that “build” has many other meanings in the programming world, but I don’t have a better one.

Failure and success #

Builds have only one way to succeed, and many ways to fail. (The script could not run, the compilation failed, all the tests but one passed, etc. ).

By extension, we say that a job is failing when the most recent build failed. We say that the job is stable if the last build succeed. Thus a job can go from stable to failing, or be back to stable.

Jobs can have other states depending on the coordinator such as “skipped”, “starting”, or “cancelled”.

Side note: Jenkins assigns colors to builds, such as “yellow” when the compilation is OK but some tests are failing. For me a yellow build is just an other kind of failed build. ³

Notification #

When a job reaches a certain state, you may use a notification to communicate the event to some humans.

For instance, if a build triggered by a pull request fails, an e-mail may be sent by the coordinator to its author.

Step #

We saw that scripts can do many various tasks. Scripts often go through a series of steps, like our example in CI scripts are scary:

Fetch the code
Compile everything
Run the tests
Generate an archive

Usually a failure in one of the steps causes the script to terminate immediately (but not always).

Also, the series of steps may differ from one job to an other. For instance, you may want the “deploy to production” step to only run for certain triggers :P

Artifact #

An artifact is a file produced by one or several steps. Artifacts may be exchanged across jobs, in some cases via the coordinator, but other mechanisms may be used.

Delivery #

A delivery is an artifact that can be used by your customers. Deliveries are often deployed to a download page.

Deployment #

A deployment is a special kind of job that consists in either:

Uploading some files to a remote location
Updating some code that is running somewhere else

In the second case you may:

Upload the source files directly to the server. You may do this for sites written in PHP for instance.
Run some Ansible Playbooks
Build and upload a Docker image to a container registry and then ask some nodes in a cluster to use the new docker image
… or any other technique

Since deployments are just jobs, they also have only one way to succeed and many ways to fail …

Environment #

An environment is just a group of machines where a certain kind deployment occurs.

For instance, you may decide that:

Every deployment triggered by a push on the master branch will go to an environment called “pre-production”
Every deployment triggered by a tag starting with ‘v’ will go to an environment called “production”.

By the way, GitLab CI has an excellent support for environments.

Conclusion #

And that’s the vocabulary I will be using. I hope you’ll find it useful.

Next time we’ll answer the number 1 question: “What programming language should I use to write CI scripts?”.

You can skip directly to the concepts section if you want, but I wanted to talk about development practices a little bit. ↩︎
Note that I’m including Continuous Delivery (CD) inside CI. Your mileage may vary. ↩︎
Plus I’m colorblind ↩︎

Tips From a Build Farmer - Part 1: CI scripts are scary

Mon, 20 Aug 2018 10:35:17 +0000

Note: This is part 1 of the Tips From a Build Farmer series.

Here’s an simple yet somewhat realistic CI script, in Bash:

echo "fetching latest changes"
git fetch
git reset --hard @{u}
echo "compiling" ...
make
echo "running tests" ...
make test
echo "generating archive" ...
DESTDIR=/tmp/foo make install
cd /tmp
tar cvfz foo.tar.gz foo

This code fetches the latest changes, compile the code, runs the tests and finally generates an archive.

What’s so scary about those few lines? Isn’t this just boring?

Well, let’s have a look, and note that everything I’m about to tell you is based on true stories. ¹

One last thing before we start: let’s assume it takes 5 minutes to compile the code, and 10 minutes to run the tests.

The Slow Development Trap #

Right before your fist public release, you discover that the archive does not contain any README. You then add a single line of code to copy the README file from the sources to the directory used to generate the archive:

  ...
  echo "generating archive" ...
  DESTDIR=/tmp/foo make install
+ cp README /tmp/foo
  cd /tmp
  tar cvfz foo.tar.gz foo

Then, to see if your code is working, you have to wait the duration of the entire build before knowing if your code is correct.

So you wait 20 minutes, and the CI script dies right before generating the archive:

fetching latest changes
compiling ...
runinng tests ...
generting archive ...
cp: cannot stat 'README': No such file or directory

Oops, turned out the README was called README.md and not just README.

So you make an other change:

- cp README /tmp/foo
+ cp README.md /tmp/foo

You wait an other 20 minutes, and the build passes again.

Congrats, you just spent 40 minutes making a one-line patch (!).

That’s what I call “The Slow Development Trap”.

But wait, it gets worse.

Let’s keep our simplistic example and study a few bugs you may encounter.

CI bugs and where to find them #

The fatal typo #

First example: the script always crashes right before running any test:

echo "compiling" ...
make
echo "running tests" ...
make tests
...

make: *** No rule to make target 'tests'.  Stop.

Got it? The Makefile target is called test, singular, not tests, plural.

If you’re lucky, your team mates will wait until you fix the problem, but some may be entitled to by-pass CI completely because “we have run all the tests on our machines before running git push like we always do”.

Hint: they didn’t. No one ever does that all the time, and it’s the very reason we use CI in the first place. So better hurry before the master branch is full of failing tests. And remember the Slow Development Trap.

Forgetting something important #

Second example:

echo "fetching latest changes"
git reset --hard @{u}
echo "compiling" ...
make
echo "running tests" ...
make test

Your forgot to call git fetch. The logs look like the code was updated, but in reality you just keep resetting the working tree to the same commit. Since this was a good commit, all the tests always pass.

A week later you find several bugs have been introduced because the developers wrongly assumed the tests were run on the version of the code they just pushed (and who could blame them?). Also, it’s going to take quite some time before the team starts trusting the CI scripts again. ²

Un seul être vous manque … #

Third example: You are building a Qt application and in the Makefile you use to install, the rule to copy libQt5Widgets.so is missing.

The main program then crashes horribly the first time it is run right after installation.

error while loading shared libraries: libQt5Widgets.so.5:
cannot open shared object file: No such file or directory

This means that you can no longer ship, you can no longer do QA, and everyone gets stuck until you fix the bug (and we’re back to ‘development is slow’)

By now you should have realized that you are not allowed to have bugs in your CI scripts.

Scared much? Let’s talk about time travel for a change.

Back to the future #

Your team is working on a brand new product they intent to release in two months. You start writing CI scripts right away. But you still have to travel to the future and make sure the scripts you wrote today still work. Or at least, you can make them work quickly. Remember: “no bugs allowed”.

So, in addition to not be allowed to have any bugs, you also have to implement all the required features and you cannot miss the deadline. If you’re late, you can’t ship. Or worse, you’ll ship by doing the required steps by hand, and then you’ll get an whole new set of problems.

Forward from the past #

Two years ago you released version 2.5.2 of your flagship product, the last of the 2.x series.

Since then you made some pretty big redesigns, and released version 3.0 and 3.1. Most of your clients have made the switch, developers are happy to no longer have to work on the ancient code base, everything looks OK.

But then someone sends you an e-mail saying they’ve found a security bug in the 2.5.2 release, and they will disclose it next week no matter what.

Your team spends 1 day discussing and planning a fix. Then they take 3 days to actually implement it, because they have to find their way in an old code base no-one no longer knows very well. Now it’s time to QA the 2.5.3 release.

Better be sure the scripts you wrote two years ago still work, or everyone will be getting a very hard time!

OK, enough with scary stories, let’s talk about the craft of writing CI code. I’m afraid I have some bad news there too.

Your standard approach will not work #

Writing CI scripts is very different than writing production code or test code. It’s a whole different world. Let me elaborate.

Testing is hard #

Writing automated tests for CI scripts is difficult and almost never catch any bugs (at least, based on my experience). The best way to check that a CI script works is by running it it its entirety. And then you fall yet again into the Slow Development Trap.

Debug is hard #

CI scripts are hard to debug because they usually don’t run on developer’s machine but on runners where debug tools are seldom found.

If the worst case scenario (which happens quite often), your only option will be to add debug logs and this means getting caught by The Slow Development Trap all over again.

Eliminating fear #

By now you should start getting nervous.

TDD helped me getting rid of the fear of changing production code by teaching me a way of writing tests and production code in a nice loop that often leads to code that has a high quality and few defects.

For this series, I will be trying to do the same thing. I’ll tell you how to resist the Slow Development Trap, and how to avoid the bugs we discussed (and many others). Hopefully next time you’ll have to change a CI script it will be less scary.

See you next time, where we’ll introduce a few useful concepts.

I’m not going to give you proofs of these events because they’re quite embarrassing for everyone involved. ↩︎
That happened to me by the way: I was the guy who forgot to fetch. ↩︎

Introducing "Tips From a Build Farmer"

Sat, 18 Aug 2018 08:34:50 +0000

I’ve spent several years working on CI scripts and infrastructure. I’ve used many tools (Jenkins, Travis, GitLab CI, …). I’ve made a lot of mistakes, and I learned a few things I thought I might share here.

Here’s the list of articles published so far in these series:

Part 1: CI scripts are scary
Part 2: Some concepts

Update: This series has been canceled. I am writing a book instead!

Hello flake8

Mon, 13 Aug 2018 14:12:44 +0000

Introduction #

In my last article I explained why I decided to no longer use pylint.

In a nutshell, most pylint warning are now caught by other linters, and all that’s left are some warnings.

In an other article I mentioned flake8 briefly, saying that I preferred a simple bash script to drive the execution of the various linters.

I was concerned that flake8 did not include pylint (now I no longer care of course). Also, I did not like the fact that flake8 forces a specific version for all the linters it runs.

Well, I was wrong (again).

flake8 is awesome #

First off, I thought flake8 only combined the pyflakes and pycodestyle linters and nothing else. ¹. Well, it also includes mccabe by default.

Also, as explained in the FAQ, there are significant advantages in having the versions of the linters frozen this way.

Here are some other features I overlooked:

You can use the setup.cfg file to configure all the linters flake8 knows about
The output is consistent for all the linters
The checks of any linter can be be disabled on any line with just a # noqa comment

A nice surprise #

The last time I upgraded pylint, I only got one new warning. It was on a line looking like this:

my_set = set([elem for elem in my_list if some_condition(elem)])

The intent here is to build a unique set from a list of elements that satisfy a given condition.

pylint emitted the following warning:

R1718: Consider using a set comprehension (consider-using-set-comprehension)

Indeed, the code can also be written like this, using a set comprehension instead:

my_set = {elem for elem in my_list if some_condition(elem)}

Advantages:

The code is shorter
We no longer build a list (inside the square brackets) just to throw it immediately afterwards
The interpreter does not have to look up the set() function, so the code is faster

Well, there is already a flake8 plugin called flake8-comprehension that deals with these kind of issues.

In fact, there are a bunch of flake8 plugins available!

Plus, adding a new flake8 plugin is as easy as running pipenv install --dev <plugin name> and nothing else has to change :)

The future is bright #

Itamar Turner-Trauring ², in the comment section on dev.to gave an interesting example:

# Note: I've taken the liberty of making the code a bit less abstract
def greet(prefix, name):
    print(prefix, name)

greeters = list()
prefixes = ["Hi", "Hello", "Howdy"]
for prefix in prefixes:
    greeters.append(lambda x: greet(prefix, x))

for greeter in greeters:
    greeter("world")

You may think the following code would print:

Hi world
Hello world
Howdy world

but instead it prints:

Howdy world
Howdy world
Howdy world

This has to do with how the closures work in Python, and the bug is indeed caught by pylint:

$ pylint example.py
 W0640: Cell variable `prefix` defined in loop (cell-var-from-loop)

I did not find a flake8 plugin that could catch this bug right away, but I found flake8-bugbear, a plugin to “find likely bugs and design problems”.

The plugin is well-written, well-tested and easy to contribute to.

I’ve already tried porting some pylint warnings to flake8-bugbear and so far it has been much easier than I thought. ³

Therefore, next time I find a bug that could have been caught by inspecting the AST (which is what both flake8-bugbear and pylint do), I know how to write or contribute to a flake8 plugin in order to automatically catch it during CI.

Thus, I can slowly build a complete replacement for pylint, with just the warnings I care about, and without the configuration issues and false positives.

Bright future indeed!

The name of the project is somewhat misleading. ↩︎
If you haven’t already, you should definitively check his blog and subscribe to the Software Clown mailing list. ↩︎
Note that the pull request was actually not merged. In reality, I started by blindly adding a new check in flake8 before taking a look at the whole list of flake8 plugins. The pull request got closed because it turned out the check I added was already in the flake8-comprehensions plugin. Sometimes you have to lie to make a better story … ↩︎

Je cherche une asso

Thu, 09 Aug 2018 17:31:47 +0000

Mise à jour: J’ai trouvé :) Je donne dorénavant un cours sur Python un samedi sur 2 à l’École du Logiciel Libre. Merci à tous ceux qui m’ont répondu et ont fait circuler ce message!

Bonjour chers lecteurs!

Un article un peu spécial aujourd’hui, en Français pour changer.

Comme vous le savez sans doute si vous êtes un lecteur régulier ou si vous m’avez déjà croisé, j’adore le langage Python. Vous savez aussi que j’aime partager mes connaissances avec d’autres.

Je cherche donc une association (ou un cadre similaire) où je pourrais donner des formations dans ce langage, par exemple en tant que bénévole.

Quelques informations #

J’habite à Paris, donc idéalement il faudrait des formations pas trop loin.
J’ai déjà donné pas mal de conférences sur des sujets divers, que ce soit dans des salons ou pour mes collègues de travail. Je suis également mentor sur exercism.io où j’aide régulièrement des débutants dans leur découverte du langage en relisant et commentant leurs solutions à des exercices variés.
Je suis toujours employé en tant que développeur et compte le rester, donc je ne pourrai pas être formateur à temps plein, mais je suis disponible ponctuellement ou quelques heures par semaine.
Je me sens capable de donner des cours à la fois pour des débutants complets (en utilisant Python pour une introduction générale à la programmation), ou sur des sujets précis comme porter du code de Python2 à Python3, des sujets autour de Django, Flask, les tests, le refactoring, etc …).

Bien sûr, tout ceci est négociable!

Une bouteille à la mer #

Si vous connaissez une association qui convienne ou semblable, que vous souhaitez me poser plus de questions, ou si vous avez d’autres pistes à me conseiller, vous pouvez me contacter de plein de façons, qui sont toutes listées sur ma page de contact, ou simplement laisser un commentaire ci-dessous.

Si le cœur vous en dit, vous pouvez aussi faire circuler ce message autour de vous.

Comme d’habitude, merci de m’avoir lu jusqu’au bout et à la prochaine!

Bye-bye pylint

Tue, 07 Aug 2018 17:23:02 +0000

I’ve been using pylint for almost a decade now.

Fast-forward ten years later, and I’ve decided no longer use it.

Here’s why.

Introduction #

Let’s start with an example. Consider the following, obviously incorrect code:

def foo():
    ...

if __name__ == "__main__"
    foo(1, 2, 3)

Here’s what the output of pylint might look like when you run it:

$ pylint foo.py
foo.py:4: [E1121(too-many-function-args), ]
  Too many positional arguments for function call

Now let’s see a few problems I’ve encountered while using pylint.

Pain points #

Initial setup #

Initial setup of pylint is always a bit painful. However, if you follow some advice you can get through it.

False positives #

A recurring issue with pylint is the amount of false positives. That is, when pylint thinks something is wrong but the code is perfectly OK.

For instance, I like using the attrs library whenever I have a class that mostly contains data, like so:

import attr

@attr.s
class Foo:
    bar = attr.ib()
    baz = attr.ib()

Those few lines of code give me a nice human-readable __repr__, a complete set of comparison methods, sensible constructors (among other things), and without any boiler plate.

But when I run pylint on this file I get:

foo.py:3: [R0903(too-few-public-methods), Foo] Too few public methods (0/2)

Well, it’s perfectly fine to require at least 2 public methods for every class you declare. Most of the time, when you have a class with just one public method it’s better to just have a function instead, like this:


# What you wrote:
class Greeter
    def __init__(self, name="world"):
        self._name = name

    def greet(self):
    print("Hello", self.name)


# What you should have written instead:
def greet(name="world"):
    print("Hello" , name)

But here pylint does not know about all the nice methods added “dynamically” by attr and wrongly assumes our design is wrong.

Thus, if you run pylint during CI and you fail the build if any error is found, you have to insert a specially formatted comment to locally disable this warning:

import attr

# pylint: disable=too-few-public-methods
@attr.s
class Foo:
  ...

This gets old fast, especially because every time you upgrade pylint you get a new bunch of checks added. Sometimes they catch new problems in your code, but you still have to go through each and every new error to check if it’s a false positive or a real issue.

But so far I had managed to overcome those pain points. So what changed?

Turning the page #

Two things happened:

First, I’ve started using mypy and a “real” type system ¹.

What I found is that mypy can catch many of the errors pylint would catch, and probably more.

Also, since it uses type annotations mypy is both faster and more precise than pylint (because it does not have to “guess” anything).

Last but not least, mypy was also designed to be used gradually, emitting errors only when it is sure there’s something wrong.

Secondly, I decided to port one of my projects to Python3.7. I had to bump pylint from 1.9 to 2.1 (because older pylint versions do not support Python3.7), and I got 18 new pylint errors, which only one of them being actually relevant.

It was at this moment I decided to take a step back.

Categories #

As we saw in those examples, the pylint error messages contain a short name for the error (like too-many-function-args), and an numeric ID prefixed by a letter (E1121).

Each letter corresponds to a pylint category.

Here is a complete list:

(F)atal (something prevented pylint from running normally)
(E)rror (serious bug)
(W)arning (not so serious issue)
(I)nfo (errors like being unable to parse a # pylint: disable comment)
(C)onvention (coding style)
(R)efactoring (code that could be written in a clearer or more Pythonic way)

Note that Fatal and Info categories are only useful when we try to understand why pylint does not behave the way it should.

The rise of the linters #

I realized I could use other linters (not just mypy) for almost every pylint category.

Some of the Error messages can also be caught by pyflakes which is fast and produces very few false positive too.
The Convention category can also be taken care of by pycodestyle.
A few Refactoring warnings (but not all) can also be caught by mccabe, which measures code complexity.

So far I’ve been using all theses linters in addition to pylint, as explained in how I lint my Python

But what if I stopped using pylint altogether?

All I would lose would be some of the Refactoring messages, but I assumed most of them would get caught during code review. In exchange, I could get rid of all these noisy # pylint: disable comments. (34 of them for about 5,000 lines of code)

And that’s how I stopped using pylint and removed it from my CI scripts. My apologies to pylint authors and maintainers: you did a really great job all these years, but I now believe it’s time for me to move on and use new and better tools instead.

What’s next #

This is not the end of the story of my never-ending quest of tools to help me write better Python code. You can read the rest of the story in Hello flake8.

By the way, at the end of Giving mypy a go I said I was curious to know if mypy would help during a massive refactoring. Well, it did, even better than I would have hoped! ↩︎

Please verify your email on pypi.org and test.pypi.org

Tue, 17 Jul 2018 20:52:02 +0000

Just a quick post to kindly ask you to make sure your email is verified on both pypi.org and test.pypi.org

What to do #

Ff you have an account on either pypi.org or test.pypi.org, please make sure that you email is verified, and if not, ask for a confirmation email to be sent.

Whether or not you have an account, feel free to spread the news using those links:

You can also re-use this article on your own blog for instance ;) ¹

Why this matters #

pypi.org is essential to lots of python developers, but is unfortunately maintained by a tiny number of great people and they need our help.

If we don’t help them, pypi.org may become unable to send any email, period, which will make things really complicated for everyone.

More details #

The full explanation is also on GitHub if you want to take a look.

Thanks for your help in keeping pypi.org working!

Friendly reminder: this article (and all the others on my blog) is licensed or under a very permissive Creative Commons CC-by license ↩︎

Else after return: yea or nay?

Thu, 12 Jul 2018 15:38:15 +0000

Introduction #

As you may know, I use pylint for most of my Python projects.

A few weeks ago, I upgraded pylint and a new warning appeared. This tends to happen when you have a pretty large code base: new checks are added to pylint all the time, so new warnings are bound to show up.

Here’s a minimal example of the kind of code that triggered the new warning:

def foo():
    if bar:
        return baz
    else:
        return qux

$ pylint foo.py
...
Unnecessary "else" after "return" (no-else-return)

Indeed, the code after the first return will never execute if bar is true, so there’s no need for the else.

In other words, the code should be written like this:

def foo():
    if bar:
        return baz
    return qux

Well, the code is shorter. But is it better?

The problem #

If you think about it, the question about whether the code is better in the first form (let’s call it explicit else) or in the second form (let’s call it implicit else) is hard to answer because you have no clue about the meaning of the foo function, or the bar, baz and qux variables.

So let’s try to come up with better examples.

Guard clauses #

Sometimes you’ll find code written this way:

def try_something():
    if precondition():
         result = compute_something()
         return result
    else:
        display_error()
        return None

In other words, you are trying to do something but that’s only possible if a condition is true. If the condition is false, you need to display an error.

The else is explicit here.

The version with an implicit else looks like this:

def try_something():
    if precondition():
         result = compute_something()
         return result
    display_error()
    return None

So far, it’s not very clear what version is better.

Note there’s a third way to write the same code, by using if not precondition() instead:

# Implicit else, inverted condition
def try_something():
    if not precondition():
        display_error()
        return None

    result = compute_something()
    return result

Now, watch what happens when we add several preconditions:

# Explicit else
def try_something():
    if precondition_one():
        if precondition_two():
            result = compute_something()
            return result
        else:
            display_error_two()
            return
    else:
        display_error_one()

# Implicit else, inverted condition
def try_something():

    if not precondition_one():
        display_error_one()
        return

    if not precondition_two():
        display_error_two()
        return

    result = compute_something()
    return result

I hope you’ll agree the second version is better.

There’s one less level of indentation, and the line that displays the error is right after the line that checks for the error.

Clear win for the implicit else here.

Symmetric conditions #

Let’s take an other example.

Suppose you are writing a script that will check all the links in documentation written as a set of HTML pages.

You’ve got a list of all the possible pages, and then you need to check both internal links (with a href looking like ../other-page) and external links like (with a href like http://example.com).

Let’s take a look at the two variants:

# Implicit else
def check_link(link) -> bool:
    if is_internal_link(link):
        return check_internal_link(link)
    return check_external_link(link)

# Explicit else
def check_link(link) -> bool:
    if is_internal_link(link):
        return check_internal_link(link)
    else:
        return check_external_link(link)

This time, I hope you’ll agree the explicit else is better.

There are two things to be done, and visually they are at them at the same level of indentation.

The symmetry between the type of the link and the check that needs to be done is preserved.

We could say that the algorithm I’ve described as text in the last paragraph is better expressed in the second version.

Conclusion #

Pylint is a great tool, but be careful before deciding whether you want to follow its refactoring pieces of advice.

Second, make sure your code is easy to read and reveal your intention. Conciseness is not the only factor here.

Last, be careful with code samples that are too abstract :)

Cheers!

Twitter Timeline Cleaner

Sat, 07 Jul 2018 13:13:13 +0000

Note: this the second post in the Quantum of Ideas series.

Introduction #

I’ve already explained on this blog why I’ve decided to never follow more than 30 accounts on twitter.

To this day, I’m still convinced that if you follow more than 50 accounts, you should seriously considering cleaning your timeline too.

This may seem like a daunting task, especially if you’re already following hundreds of accounts. That’s when the Twitter Timeline Cleaner comes in.

How it works #

Remember, this is not an existing product, just a “work in progress” idea.

First, the TTC does a backup of the list of all your followers
Then it goes through the list, and for all of account you follow, it:
- Takes a random sample of recent tweets from this account and asks you to mark them as relevant or useless.
- Then it gives each account a score, something like the average interest of each tweet divided by the frequency of tweets.
That way, accounts that tweet rarely but are always interesting get a high score. Accounts that tweet several times per day every day, even if they sometimes have interesting content, get penalized.

When this is done, the TTC shows you a list of accounts sorted by decreasing score, and all you have to do is to select which irrelevant accounts you want to stop following.

By the way, unless they explicitly configured their account to do so, each account you unfollow will not get notified, so there’s a pretty good chance they won’t get mad at you.

Update: Turns out such a product already exists. Take a look at https://www.manageflitter.com/

Conclusion #

And that’s all there is to it. We could also extend the TTC so that it prevents you from becoming an account with low signal-to-noise ratio using the same technique. For instance, it would display messages like: “only 2% of readers found this tweet interesting”, or “you’ve already tweeted 3 times today, maybe go take a break”.

… or you could just use Mastodon where:

you can choose an instance that matches your interests
messages are more likely to get interesting due to the 500 characters limit
discussions are easier to follow because everything is sorted by chronological order
people are just overall nicer to each other or so it seems

Cheers!

Let's Build Chuck Norris! - Part 7: Android and JNA

Mon, 18 Jun 2018 12:17:17 +0000

Note: This is part 7 of the Let’s Build Chuck Norris! series.

Last time we managed to cross-compile and run C++ code for Android.

It’s now time to write some Java code, but we need to take a detour on the desktop first.

Java bindings #

Let’s create a new Java library project with gradle:

$ cd chucknorris
$ mkdir java && cd java
$ gradle init --type java-library

Gradle created a bunch of files, but for now we just care about the sources and the tests.

Our goal is to write a test that demonstrates we can indeed get some ChuckNorris facts.

Let’s start by removing cruft from the generated build.gradle:

dependencies {
-    api 'org.apache.commons:commons-math3:3.6.1'
-    implementation 'com.google.guava:guava:23.0'
}

Then let’s fix the source files that gradle created so that we have proper package:

$ tree java
├── build.gradle
├── gradle
└── src
    ├── main
    │   └── java
    │       └── com
    │           └── chucknorris
    │               └── ChuckNorris.java
    └── test
        └── java
            └── com
                └── chucknorris
                    └── ChuckNorrisTest.java

Now we can write a failing test:

/* In ChuckNorrisTest.java */
public class ChuckNorrisTest {

    @Test
    public void testGetFact() {
        ChuckNorris ck = new ChuckNorris();
        String fact = ck.getFact();
        assertThat(fact, containsString("Chuck Norris"));
    }
}

/* In ChuckNorris.java */

public class ChuckNorris {
    String getFact() {
        return "";
    }
}

Let’s run the tests:

$ ./gradlew test
> Task :test FAILED

com.chucknorris.ChuckNorrisTest > testGetFact FAILED
    java.lang.AssertionError at ChuckNorrisTest.java:14

1 test completed, 1 failed

# open build/reports/tests/test/index.html

java.lang.AssertionError:
Expected: a string containing "Chuck Norris"
     but: was ""

OK, this fails for the good reason.

Now we can try and load our shared library using JNA.

First, we add the dependency in the build.gradle file:

dependencies {
+   api 'net.java.dev.jna:jna:4.5.1'
    testImplementation 'junit:junit:4.12'
}

Then we’re ready to use JNA:

We use Native.loadLibrary() to load the shared library
We create a CLibrary interface that implements the C functions we want to call as methods. (just chuck_norris_init for now).
We call chuck_norris_init in the constructor of our ChuckNorris class, storing the result into a JNA Pointer:

public class ChuckNorris {
  private Pointer ckPointer;

  private static CLibrary loadChuckNorrisLibrary() {
    return (CLibrary) Native.loadLibrary("chucknorris", CLibrary.class);
  }

  public interface CLibrary extends Library {
    CLibrary INSTANCE = loadChuckNorrisLibrary();

    void chuck_norris_init();
  }

  public ChuckNorris() {
    ckPointer = CLibrary.INSTANCE.chuck_norris_init();
  }

  public String getFact() {
    return "";
  }
}

And we run the tests:

$ ./gradlew test
java.lang.UnsatisfiedLinkError: Unable to load library 'chucknorris':
  Native library (linux-x86-64/libchucknorris.so)
  not found in resource path (...)
  at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:303)
  at com.sun.jna.NativeLibrary.getInstance(NativeLibrary.java:427)
  ...

This is expected. We never told JNA where the libchucknorris.so file is.

As a reminder, the file currently lives in the build/default folder. Here’s how we built it:

$ cd build/default
$ conan install ../..
$ cmake -GNinja -DBUILD_SHARED_LIBS=ON ../..
$ ninja

There are several ways to tell JNA about the location of the shared library file. Here we’ll set a system property in the test block of the Gradle script:

def thisFile  = new File(project.file('build.gradle').absolutePath)
def projectPath = thisFile.getParentFile()
def topPath = projectPath.getParentFile()
def cppPath = new File(topPath, "cpp")
def cppBuildPath = new File(cppPath, "build/default/lib")

test {
    systemProperty 'jna.library.path', cppBuildPath
}

Now if we re-run the tests we get back our first failure:

$ ./gradlew test

java.lang.AssertionError:
Expected: a string containing "Chuck Norris"
     but: was ""

But we did manage to instantiate the ChuckNorris class, so this is progress :)

Let’s implement getFact(), and while we’re at it, add a .close() method:


  public interface CLibrary extends Library {
    // ...
    void chuck_norris_init();
    String chuck_norris_get_fact(Pointer pointer);
    void chuck_norris_deinit(Pointer pointer);
  }

  public ChuckNorris() {
    ckPointer = CLibrary.INSTANCE.chuck_norris_init();
  }

  public String getFact() {
    return CLibrary.INSTANCE.chuck_norris_get_fact(ckPointer);
  }

  public void close() {
    CLibrary.INSTANCE.chuck_norris_deinit(ckPointer);
  }
}

Re-run the tests:

$ ./gradlew test
BUILD SUCCESSFUL

Success!

Creating a new Android project #

Now we know:

How to compile the C++ code for Android.
How to load some C++ in Java, but only for the desktop.

It’s time to glue things together.

To do so, the best thing is to use Android Studio to create the gradle project, starting with a with a basic activity so we don’t have to deal with all the Android boilerplate.

Adapting the GUI #

Let’s pretend the ChuckNorris class already exists for now. ¹

We start by adding a text_view ID for the text view in the content_main layout.

Then we adapt the MainActivity.java file to update the text view when clicking on the floating button action:


public class MainActivity extends AppCompatActivity {
  private ChuckNorris chuckNorris;


  @Override
  protected void onCreate(Bundle savedInstanceState) {
    chuckNorris = new ChuckNorris();
    super.onCreate(savedInstanceState);
  }

  @Override
  protected void onDestroy() {
    chuckNorris.close();
    super.onDestroy();
  }

  // ...

  final TextView textView = (TextView) findViewById(R.id.text_view);
  FloatingActionButton fab = (FloatingActionButton) findViewById(R.id.fab);
  fab.setOnClickListener(new View.OnClickListener() {
    @Override
    public void onClick(View view) {
      String fact = chuckNorris.getFact();
      textView.setText(fact);
  });

Adding ChuckNorris sources #

One of Java’s slogan is “Write Once, Run Everywhere” ².

So let’s:

Add JNA in the dependencies
Add the ChuckNorris.java file we wrote earlier

And everything should work, right?

Fun with jnidispatch.so #

To check if our code works, let’s create an emulator, and click on play.

We’re faced with:

ChuckNorris has stopped

Open App Again

What? Chuck Norris can’t be stopped, this is unacceptable!

Time to look at the logs:

06-18 14:27:18.553 6890-6890/info.dmerej.chucknorris E/AndroidRuntime:
FATAL EXCEPTION: main
Process: info.dmerej.chucknorris, PID: 6890
java.lang.UnsatisfiedLinkError:
Native library (com/sun/jna/android-x86-64/libjnidispatch.so)
not found in resource path (.)
  at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath
  at com.sun.jna.Native.loadNativeDispatchLibrary
  ...
  at info.dmerej.chucknorris.ChuckNorris.loadChuckNorrisLibrary

That’s a fun one. Turns out the name of dependency changes when compiling for Android, you need a @aar prefix ³:

dependencies {
  // ...
  implementation 'net.java.dev.jna:jna:4.5.1@aar'
}

Let’s try again!

We get the same error message, but this time it’s the chucknorris.so library that is not found:

06-18 14:27:18.553 6890-6890/info.dmerej.chucknorris E/AndroidRuntime:
FATAL EXCEPTION: main
Process: info.dmerej.chucknorris, PID: 6890
java.lang.UnsatisfiedLinkError: Unable to load library 'chucknorris':
  Native library (android-x86-64/libchucknorris.so) not found

Fortunately, there’s a more or less standard solution.

If you put a .so file in a folder named src/main/jniLibs/<arch>, it will be included inside the Java application, and the Java code will be able to load it without any configuration.

The shared option #

For simplicity purposes, we built the ChuckNorris library as a static library, just to show that the C++ binary still needed libc++_shared.so to run. But JNA needs a shared library to run.

Remember in part 4 we had to call CMake with -DBUILD_SHARED_LIBS=ON to get a shared library.

We’ll do the same thing, but going through Conan this time.

First, let’s add the ChuckNorris:shared option in the android profile:

...
[options]
*:pic=True
ChuckNorris:shared=True

Then adapt the recipe:

class ChucknorrisConan(ConanFile):
    name = "ChuckNorris"
    ...
    options = {"shared": [True, False]}
    default_options = "shared=False"

    def build(self):
        cmake = CMake(self)
        cmake_definitions = {}
        if self.options.shared:
            cmake_definitions["BUILD_SHARED_LIBS"] = "ON"
        cmake.configure(defs=cmake_definitions)

    def package(self):
        self.copy("lib/libchucknorris.so", dst="lib", keep_path=False)
        self.copy("lib/libc++_shared.so", dst="lib", keep_path=False)

Then let’s re-create the Conan package:

$ conan create . dmerej/test --profile android --setting arch=x86_64
Exporting package recipe
...
package(): Copied 2 '.so' files: libchucknorris.so, libc++_shared.so
Package '<hash>' created

Finally let’s create symlinks to all .so files from the package.

$ cd android/app
$ cd src/main
$ mkdir -p jniLibs/x86_64
$ cd jniLibs/x86_64
$ ln -s ~/.conan/data/ChuckNorris/0.1/dmerej/test/<hash>/libchucknorris.so .
$ ln -s ~/.conan/data/ChuckNorris/0.1/dmerej/test/<hash>/libc++_shared.so .

Let’s try again:

Victory \o/

That’s all for today. See you next time!

This is also known as wishful thinking programming ↩︎
As always, the Wikipedia page contains lots of interesting stuff about this topic. ↩︎
You can find a note about this in JNA’s FAQ, but as far as I know, not anywhere else in the documentation. ↩︎

Giving mypy a go

Mon, 11 Jun 2018 10:17:17 +0000

Introduction #

This is a follow up to the I don’t need types article.

I left a teaser explaining I’ll be giving concrete examples using a Python project, so here goes.

As you may have guessed, I’m going to talk about type annotations and the mypy static type checker.

Update: 2021-11-15: Feel free to also read the case study about urllib3

How mypy works #

Here’s a quick overview of how mypy works, in case you are not already familiar with it.

Let’s use a rather contrived example:

def is_odd(num):
    return num % 2 == 0


if is_odd("this sentence contains %s"):
    print("ok")

We have a function is_odd, which obviously only works with numbers, and we call it with a string.

But since the string contains %s, Python will happily assume we are trying to format it, and the bug will go unnoticed. is_odd will simply return False, because we are also allowed to compare strings and numbers with == in Python.

Without type annotations, mypy detects nothing:

$ mypy foo.py
<nothing>

But if we add type annotations and re-run mypy, we do get an error message:

def is_odd(num: int) -> bool:
    return num % 2 == 0


if is_odd("this sentence contains %s"):
    print("ok")

$ mypy foo.py
error: Argument 1 to "is_odd" has incompatible type "str"; expected "int"

Thus, you can use mypy in a loop:

Start by annotating a few functions.
Run mypy on your source code.
If it detects some errors, either fix the annotations or the code.
Back to step one

This is called “gradual typing”.

Designing an experiment #

The goal of the experiment is to check if going through the trouble of adding type annotations is worth it.

Some definitions #

Let me define what I mean by “worth it”.

There are many things said about types annotations like:

It makes it easier for external contributors to understand the code
It helps during refactorings
It facilitates maintenance of large projects

Those may be true, but they are hard to measure.

So I asked myself something else:

One: even with everything else (linters, tests, reviews …), were there bugs that only type annotations would catch?
Two: were the changes required to have mypy run without errors improving the quality of the code?

I know the second question is a bit subjective. I’m still going to use this metric because it’s one that really matters to me.

Choosing a project #

To test this hypothesis, I needed an existing project. I used tsrc, the command-line tool we use at work to manage multiple git repositories and automate GitHub and GitLab review automation.¹

I chose it because:

It’s not too big, so the experiment will not take too long.
It’s not too small, so we’ll have enough experimental data.

The protocol #

So here is the protocol I used:

Go through the gradual typing loop until everything is type-annotated.
Make a note of every non-trivial patch. (That is anything that is not just adding annotations)
When the loop is finished, take a look at each of the patch, and ask if a bug was found, and whether it improved the quality of the code.

Before I continue, I should tell you I used mypy with three important options:

--strict: which means mypy emits an error for every missing type annotations.
--ignore-missing-imports: tsrc depends on libraries for which no type stub exist.
--strict-optional: If you have a string that can None, you must use Optional[str] instead of just str. I chose that because:
- Errors relative to None are quite frequent
- --strict-optional is going to be the default in a future mypy release.

Looking at patches #

Let’s look at trivial patches first:

Truthy string #

def find_workspace_path() -> Path:
    head = os.getcwd()
     tail = True
     while tail:
         tsrc_path = os.path.join(head, ".tsrc")
         if os.path.isdir(tsrc_path):
             return Path(head)
         else:
             head, tail = os.path.split(head)
     raise tsrc.Error("Could not find current workspace")

This function starts by checking there is a .tsrc hidden directory in the working path. If not, it goes through every parent directory and check if the .tsrc directory is here. If it reaches the root of the file system (the second value of os.path.split is None), it raises an exception.

mypy did not like that tail started as a boolean, and then was assigned to a string.

We can fix that by using a non-empty string, with a value that reveals the intention:

- tail = True
+ tail = "a truthy string"
     while tail:

An other way would be to use a type like Union[bool,str] but that would be more confusing I think.

Anyway, I’m not sure the quality of the code improved there. No point for mypy.

save_config #

class Options:
    def __init__(self, url, shallow=False) -> None:
        self.shallow: bool = shallow
        self.url: str = url


def save_config(options: Options):
    config = dict()
    config["url"] = options.url
    config["shallow"] = options.shallow

    with self.cfg_path.open("w") as fp:
        ruamel.yaml.dump(config, fp)

We are using save_config to serialize a “value object” (the Options class) into a YAML file.

mypy saw the first two lines, config = dict(), config["url"] = options.ul and wrongly deduced that config was a dict from strings to strings.

Then it complained about config["shallow"] that was assigned to a boolean.

We can fix that by forcing the config type to be a dict from string to Any: Any is a “magic” type we can use precisely for this kind of situation.

- config = dict()
+ config: Dict[str, Any] = dict()

This a bit annoying, but the type annotation makes it clearer what the config is. 1 point for mypy.

Encoding project names #

When you use the GitLab API, you often have to use the ‘project id’. The doc says you can use the <namespace>/<project_name> string if it is “url encoded”, like this:

Get info about the Foo project in the FooFighters namespace:
GET  /api/v4/projects/FooFighters%2Foo

In python, the naive approach does not work:

import urllib.parse

>>> urllib.parse.quote("FooFighters/Foo")
FooFighters/Foo

Instead you have to specify a list of “safe” characters, that is characters that you don’t need to encode.

The doc says the default value is “/”.

Thus, here’s what we use in tsrc.gitlab:

project_id = "%s/%s" % (namespace, project)
encoded_project_id = urllib.parse.quote(project_name, safe=list())

mypy saw that the default value was a string, so he complained we were using a list instead. We fixed it by:

-        encoded_project_name = urllib.parse.quote(project_name, safe=list())
+        encoded_project_name = urllib.parse.quote(project_name, safe="")

Here, mypy forced us to follow an implicit convention. There are two ways to represent a list of characters in Python. A real list: ['a', 'b', 'c'], or a string: "abc". The authors of the urllib.quote() function decided to use the second form, so it’s a good thing we follow this convention too.

An other win for mypy.

Bugs found #

We already use a lot of tools in the hope of catching bugs:

Every pull request was reviewed by other humans
Two static analyzers (pylint and pyflakes) were ran for each pull request
McCabe complexity was measured for each and every function and method of the code base and was not allowed to go above 10.
TDD was used throughout the development of the project
Test coverage was already at 80%

Despite of this, mypy found bugs that tests, all the linters and all the reviewers did not find.

Here are a few of them. Feel free to try and find the bug yourself, the answer will be given after the ⁂ symbol.

handle_stream_errors #

class GitLabAPIError(GitLabError):
    def __init__(self, url: str, status_code: int, message: str) -> None:
    ...

def handle_stream_errors(response: requests.models.Response) -> None:
     if response.status_code >= 400:
        raise GitLabAPIError(
            response.url, "Incorrect status code:", response.status_code)

This code wraps calls to the GitLab API, and arrange for a particular GitLabAPIError to be raised, when we make a “stream” request. (For instance, to download a GitLab CI artifact):

⁂

The problem here is that we inverted the status_code and message parameter. Easy to fix:

-        raise GitLabAPIError(
-           response.url, "Incorrect status code:", response.status_code)
+        raise GitLabAPIError(
+           response.url, response.status_code, "Incorrect status code")

The bug was not caught because the code in question was actually copy/pasted from a CI script (and you usually don’t write tests for CI scripts).

We actually don’t need streamed responses anywhere in tsrc, so this is in fact dead code (and it is now gone from the code base)

handle_json_errors #

def handle_json_errors(response: requests.models.Response):

    try:
         json_details = response.json()
     except ValueError:
     json_details["error"] = (
         "Expecting json result, got %s" % response.text
     )

    ...

    url = response.url
    if 400 <= status_code < 500:
        for key in ["error", "message"]:
            if key in json_details:
                raise GitLabAPIError(url, status_code, json_details[key])
         raise GitLabAPIError(url, status_code, json_details)
    if status_code >= 500:
        raise GitLabAPIError(url, status_code, response.text)

This one is slightly more interesting. It is located near the previous one and handles errors for the calls to the GitLab API which returns JSON objects.

We of course have to catch 500 errors, which hopefully do not happen often.

In case of a status code between 400 and 499, we know there was a problem in the request we made, but we need to tell the user why the request was rejected.

Most of the time the GitLab API returns a JSON object containing a error or message key, but sometimes neither key is found in the returned object, and sometimes the text of the response is not even valid JSON code.

So we have to check for both keys in the JSON object, and if not found (if we exit the for loop), just store the entire JSON response in the exception.

⁂

The bug is in the second raise GitLabAPIError. We are passing an entire object where the GitLabAPIError expected a string.

The fix was:

- raise GitLabAPIError(url, status_code, json_details)
+ raise GitLabAPIError(url, status_code, json.dumps(json_details, indent=2))

Again, this was hard to catch with tests. The case where the JSON returned by GitLab did not contain a error or message key only happened once in the lifetime of the project (which explain why the code was written), so manual QA and unit tests did not need to check this code path.

Anyway, note we did not blindly wrote something like str(json_details) to convert the JSON object to a string. We found out it was used in a message displayed to the end user, thus we use json.dumps(json_details), indent=2) to make sure the message contains neatly indented JSON and is easy to read.

LocalManifest #

class LocalManifest:
    def __init__(self, workspace_path: Path) -> None:
        hidden_path = workspace_path.joinpath(".tsrc")
        self.clone_path = hidden_path.joinpath("manifest")
        self.cfg_path = hidden_path.joinpath("manifest.yml")
        self.manifest: Optional[tsrc.manifest.Manifest] = None

    def update(self) -> None:
        ui.info_2("Updating manifest")
        tsrc.git.run_git(self.clone_path, "fetch")
        tsrc.git.run_git(self.clone_path, "reset", "--hard", branch)

    def load(self) -> None:
        yml_path = self.clone_path.joinpath("manifest.yml")
        if not yml_path.exists():
            message = "No manifest found in {}. Did you run `tsrc init` ?"
            raise tsrc.Error(message.format(yml_path))
        self.manifest = tsrc.manifest.load(yml_path)

    @property
    def copyfiles(self) -> List[Tuple[str, str]]:
        return self.manifest.copyfiles

    def get_repos(self) -> List[tsrc.Repo]:
        assert self.manifest, "manifest is empty. Did you call load()?"
        return self.manifest.get_repos(groups=self.active_groups)

After you run tsrc init git@example.com/manifest.git, the manifest is cloned inside <workspace>/.tsrc/manifest.

Thus, the contents of the manifest.yml is in <workspace>/.tsrc/manifest/manifest.yml>.

The LocalManifest class represent this manifest repository.

Here’s what happens when you run tsrc sync:

local_manifest.update() is called: the repository in <workspace>/.tsrc/manifest> is updated by running git fetch; git reset --hard origin/master
local_manifest.load() is called: the manifest.yml file is parsed, and its contents are stored in the self.manifest attribute.
Then the Syncer class calls local_manifest.get_repos() to find out the list of repositories to clone or synchronise.
Finally, the FileCopier class uses local_manifest.copyfiles to perform file copies.

⁂

It’s not really a bug, but mypy forced us to acknowledge that LocalManifest.manifest starts as None, and only gets its real value after .load() has been called.

We already have an assert in place in get_repos(), but mypy forced us to add a similar check in the copyfiles getter:

    @property
    def copyfiles(self) -> List[Tuple[str, str]]:
+       assert self.manifest, "manifest is empty. Did you call load()?"
        return self.manifest.copyfiles

GitServer #

Some of the tests for tsrc are what we call end-to-end tests:

Here’s how they work:

We create a band new temporary directory for each test
In it, we create a srv directory with bare git repositories
Then, we create a work directory and we run tsrc commands from there.

Thus, we don’t have to mock file systems or git commands (which is doable but pretty hard), and things are easy to debug because in case of problem we can just cd to the test directory and inspect the state of the git repositories by hand.

Any way, those tests are written with a test helper called GitServer.

You can use this class to create git repositories, push files on some branches, and so on:

Here’s what the helper looks like:

    def add_repo(self, name: str, default_branch="master") -> str:
        ...
        url = self.get_url(name)
        return url

    def push_file(self, name: str, file_path: str, *,
                  contents="", message="") -> None:
        ...
        full_path = ...
        full_path.parent.makedirs_p()
        full_path.touch()
        if contents:
            full_path.write_text(contents)
        commit_message = message or ("Create/Update %s" % file_path)
        run_git("add", file_path)
        run_git("commit", "--message", commit_message)
        run_git("push", "origin", "--set-upstream", branch)

    def tag(self, name: str, tag_name: str) -> None:
        run_git("tag", tag_name)
        run_git("push", "origin", tag_name)

    def get_tags(self, name: str) -> List[str]:
         src_path = self.get_path(name)
         rc, out = tsrc.git.run_git_captured(src_path, "tag", "--list")
         return out

You can use it like this:

def test_tsrc_sync(tsrc_cli: CLI, git_server: GitServer) -> None:
    git_server.add_repo("foo/bar")
    git_server.add_repo("spam/eggs")
    manifest_url = git_server.manifest_url
    tsrc_cli.run("init", manifest_url)
    git_server.push_file("foo/bar", "bar.txt", contents="this is bar")

    tsrc_cli.run("sync")

    bar_txt_path = workspace_path.joinpath("foo", "bar", "bar.txt")
    assert bar_txt_path.text() == "this is bar"

⁂

The bug is in get_tags:

    def get_tags(self):
         rc, out = tsrc.git.run_git(src_path, "tag", raises=False)
-        return out
+        return out.splitlines()

The get_tags methods is also dead code. It has an interesting story.

The need for the GitServer helper occurred when I started writing end-to-end tests for tsrc and discovered I needed a test framework to write those end-to-end test.

It was vital that the GitServer implementation uses clean code.

One, we must be sure GitServer has no bugs. Otherwise, we may get stuck looking for bugs in our production code when tests fail.
For the same reason, GitServer should be easy to change. There will almost certainly be features added in tsrc that will require adapting GitServer code in order to properly test it.

So to have a clean implementation of the GitServer I of course used the best technique I know of: TDD.

You can find them in the [test_git_server.py](https://github.com/dmerejkowsky/tsrc/blob/main/tsrc/test/helpers/test_git_server.py file.

Anyway, I was writing an end-to-end test for tsrc that involved tags.

I thought: “OK, I need a .tag() method in GitServer. So I also need a get_tags() method to check the tag was actually pushed”. Thus I wrote the get_tags method, forgetting to write a failing test for GitServer first (that still happens after 5 years of TDD, so don’t worry if it happens to you too.). At that point I only had my end-to-end test failing, so I made it pass and completely forgot about the get_tags() method. Oh well.

Executors and Tasks #

In the implementation of tsrc we often have to loop over a list of items, perform actions an each of them and record which one failed, and when the loop is done, display the error messages to the user.

For instance:

tsrc sync
* (1/2) foo/bar
remote: Counting objects: 3, done.
...
Updating 62a5d28..39eb3bd
error: The following untracked files would be overwritten by merge:
	bar.txt
Please move or remove them before you merge.
Aborting
* (2/2) spam/eggs
Already up to date.
Error: Synchronize workspace failed
* foo/bar: updating branch failed

Or:

$tsrc foreach
:: Running `ls foo` on every repo
* (1/2) foo
$ ls foo
bar.txt
* (2/2) spam
$ ls foo
ls: cannot access 'foo': No such file or directory
Error: Running `ls foo` on every repo failed
* spam

So in order to keep things DRY, we have some high-level code that only deals with loop and error handling:

First, a generic Task method.

(Note that you can have your own generic types with mypy. This is awesome because without it you get laughed at by C++ programmers)


class Task(Generic[T], metaclass=abc.ABCMeta):

    @abc.abstractmethod
    def description(self) -> str:
        pass

    @abc.abstractmethod
    def display_item(self, item: T) -> str:
        pass

    @abc.abstractmethod
    def process(self, item: T) -> None:
        pass

And then, a generic SequentialExecutor who knows how to execute a given task on a list of items and display the outcome:

class SequentialExecutor(Generic[T]):
    def __init__(self, task: Task[T]) -> None:
        self.task = task
        self.errors: List[Tuple[Any, tsrc.Error]] = list()

    def process(self, items: List[T]) -> None:
        if not items:
            return True
        ui.info_1(self.task.description())

        self.errors = list()
        for item in enumerate(items):
            ...
            self.process_one(item)

        if self.errors:
            self.handle_errors()

    def process_one(self, item: T) -> None:
        try:
            self.task.process(item)
        except tsrc.Error as error:
            self.errors.append((item, error))

    def handle_errors(self) -> None:
        # Display nice error message
        ...
        raise ExecutorFailed()

Thus we can inherit from Task to implement tsrc sync:

class Syncer(tsrc.executor.Task[Repo]):

    def process(self, repo: Repo) -> None:
        ui.info(repo.src)
        self.fetch(repo)
        self.check_branch(repo)
        self.sync_repo_to_branch(repo)

    def check_branch(self, repo: Repo):
        current_branch = tsrc.git.get_current_branch(repo_path)
        if not current_branch:
            raise tsrc.Error("Not on any branch")

Ditto for tsrc foreach:

class CmdRunner(tsrc.executor.Task[Repo]):

    def process(self, repo: Repo) -> None:
        full_path = self.workspace.joinpath(repo.src)
        rc = subprocess.call(cmd, ...)
        if rc != 0:
            raise CommandFailed

Did you spot the bug?

⁂

The bug is in here:

    def process(self, items: List[T]) -> None:
         if not items:
             return True

This is the result of a bad refactoring. Executor used to track success of task by looking at the return value the process() method.

After a while, I found out it was clearer to just use exceptions for that, mostly when I implemented the Syncer class. (You can just raise an exception instead of adding lots of ifs).

But the early return True was left. Here mypy found something that would have puzzled an external reader. Almost everything function or method dealing with Executors and Tasks in tsrc either return None or raise Exception. What is that boolean doing here?

There were of course no tests left that checked the return value of process (they got refactored at the same time of the rest of the code), so the bug went unnoticed.

run_git #

The last changed required by mypy was also pretty interesting. Here’s the deal.

In tsrc, we often need to run git commands, but we can run them in two very different ways:

We can either just run the command and let its output directly being displayed to the user. This is useful for things like git fetch which can take some times and for which the output contains an indication of progress for the user.
Or we just need to run the command, capture its output and deal with it as a value. For instance, if we want to get the url of the “origin” remote, we can call git remote get-url origin.
In both cases, we must handle the possibility that the command can fail.

Here’s what the implementation looked like:

def run_git(working_path, *cmd, raises=True):
    """ Run git `cmd` in given `working_path`

    If `raises` is True and git return code is non zero, raise
    an exception. Otherwise, return a tuple (returncode, out)

    """
    git_cmd = list(cmd)
    git_cmd.insert(0, "git")
    options = dict()
    if not raises:
        options["stdout"] = subprocess.PIPE
        options["stderr"] = subprocess.STDOUT

    process = subprocess.Popen(git_cmd, cwd=working_path, **options)

    if raises:
        process.wait()
    else:
        out, _ = process.communicate()
        out = out.decode("utf-8")

    returncode = process.returncode
    if raises:
        if returncode != 0:
            raise GitCommandError(working_path, cmd)
    else:
        return returncode, out

And here’s how to use it:

# run `git fetch`, leaving the output as-is
run_git(foo_path, "fetch")

# run git remote get-url origin
rc, out = run_git(foo_path, "remote", "get-url", "origin", raises=False):
if rc == 0:
    # Handle the case where there is no remote called 'origin'
    ...
else:
    # Do something with out

Here’s an other place we used run_git: we have a command named tsrc status, which displays a summary of the whole workspace. Here what’s its output looks like:

tsrc status
* foo master ↓1 commit
* bar devel ↑1 commit

And here is the implementation:

class GitStatus:

    def update_remote_status(self):
        _, ahead_rev = run_git(
            self.working_path,
            "rev-list", "@{upstream}..HEAD",
            raises=False
        )
        self.ahead = len(ahead_rev.splitlines())

        _, behind_rev = run_git(
            self.working_path,
            "rev-list", "HEAD..@{upstream}",
            raises=False
        )
        self.behind = len(behind_rev.splitlines())

Found the bug yet?

⁂

The code in update_remote_status() assumes that git rev-list outputs a list of commits, and then count the lines in the output.

This works well if there is a remote branch configured:

$ git revlist HEAD..@{upstream}
30f729a6a0ec3926cf063f5f8a3953b89d7b252e
ef564f0ef38a163beb3db52474ac4e256a6c2cd4

But if not remote is configured, the git command will fail with a message looking like:

$ git revlist HEAD..@{upstream}
fatal: no upstream configured for branch 'dm/foo'

Since update_remote_status() does not check the return code, self.ahead_rev and self.behind_rev both get set to 1, and the output looks like:

* foo other-branch ↑1↓1 commit

Oops.

⁂

But there is more to it than just this bug.

Notice the type of the return value of run_git depends on the value of the raises parameter.

At first I tried to annotate the function with a Union type, like this:

def run_git(working_path: Path, *cmd: str, raises=True) ->
      Union[Tuple[int, str], None]:
    pass

But then mypy complained for each and every line that used run_git with raises=False:

rc, out = run_git(..., raises=False)

foo.py:39: error: 'builtins.None' object is not iterable

I thought about this and found out it was cleaner to split run_git into run_git and run_git_captured:

def run_git(working_path: Path, *cmd: str) -> None:
    """ Run git `cmd` in given `working_path`

    Raise GitCommandError if return code is non-zero.
    """
    git_cmd = list(cmd)
    git_cmd.insert(0, "git")

    returncode = subprocess.call(git_cmd, cwd=working_path)
    if returncode != 0:
        raise GitCommandError(working_path, cmd)


def run_git_captured(working_path: Path, *cmd: str,
                     check: bool = True) -> Tuple[int, str]:
    """ Run git `cmd` in given `working_path`, capturing the output

    Return a tuple (returncode, output).

    Raise GitCommandError if return code is non-zero and check is True
    """
    git_cmd = list(cmd)
    git_cmd.insert(0, "git")
    options: Dict[str, Any] = dict()
    options["stdout"] = subprocess.PIPE
    options["stderr"] = subprocess.STDOUT

    returncode = process.returncode
    if check and returncode != 0:
        raise GitCommandError(working_path, cmd, output=out)
    return returncode, out

True, there’s a bit more code and slightly duplication but:

The multiple if raises in the original implementation are gone. Less if in a function is always a win.
The raises parameter did too very different things:
- One, it changed the return type of the function
- Two, it forced the caller to explicitly handle the case where the command fails.

Now the intent about whether you want to capture the output or not is encoded in the name of the function (run_git_captured instead of run_git).

Also note that you can’t forget about checking the return code anymore.

You can either write:

_, out = run_git_captured(repo_path, "some-cmd")
# We know `out` is *not* an error message, because if `some-cmd` failed, an
# exception would have been raised before `out` was set.

Or you can be explicit about your error handling:

rc, out = run_git_captured(repo_path, "some-cmd", check=False)
if rc == 0:
    # Ditto, out can't be an error message
else:
    # Need to handle error here

That alone would be a good reason to use mypy I think :)

What’s next #

Well, the mypy changes have been merged and the CI now runs mypy in strict mode on every pull request.

I’m still curious about the other benefits of type annotations I could not check (maintainability, code comprehension, ease of refactorings …).

I guess we’ll see how the next refactoring in tsrc goes.

Update, a few months later: Well, the next refactoring in tsrc went really well and mypy caught a ton of problems very early. I’m now using mypy in strict mode in almost all of my projects.

Conclusion #

We saw how mypy, while stilling making relative few false positives, still found inconsistencies, a few bugs, and even design problems.

So, my advice is for you to use it if you get a chance. Cheers!

If you want to know more, feel free to browse the documentation, or read the introduction post. ↩︎

Let's Build Chuck Norris! - Part 6: Cross-compilation for Android

Tue, 05 Jun 2018 10:17:17 +0000

Note: This is part 6 of the Let’s Build Chuck Norris! series.

Introduction #

After our little detour talking about how to wrap C++ in Python, we now are ready to tackle writing Android applications using C++ code.

We’ll start with a simple challenge: try and run a simple “Hello, world” program written in C inside a simulator and on our phone.

Let’s get started!

A naive try #

Let’s see what happens if we naively try to run some C code on our phone.

First, let’s install the Android SDK so that we can use the adb tool.
Then, let’s activate the developer mode on the phone, and plug it to our laptop.

Now let’s compile a simple C program:

/* in hello.c */
#include <stdio.h>

int main() {
  printf("Hello, world\n");
  return 0;
}

$ gcc hello.c -o hello

And now, let’s use adb push to copy the binary on our phone, and adb shell to try and run it:

(Note that /data/local/tmp is the only directory I found where we can run executables):

$ adb push hello /data/local/tmp
$ adb shell
$ cd /data/local/tmp
$ ./hello
/system/bin/sh: ./hello: not executable: 64-bit ELF file

Hum. What just happened ?

CPU architectures #

When you use a compiler, you get a binary file from the source code. You can think of this binary file as a list of instructions, ready to be used by a CPU.

The trick is that different CPU have different instructions sets.

For instance, it is very likely that the CPU you have on your laptop is a x86_64 CPU, and that the CPU of you phone is an armv7. ¹

This means that a binary you built for a x86_64 CPU will not run on a armv7 CPU.

The libc #

OK, but the Android SDK also comes with a simulator, and we can choose the CPU architecture.

Can’t we just use a x86_64 Android simulator ?

If we try the same thing with a x86_64 simulator, we get an even weirder error:

$ adb push hello /data/local/tmp
$ adb shell
$ cd /data/local/tmp
$ ./hello
/system/bin/sh: ./hello: no such file or directory

The file does exist, but Android does not know how to run it. Why?

Remember when we used LD_TRACE_LOADED_OBJECTS back in a previous post?

Well, if we re-run the binary we just built we can see it loads a few .so files:

$ export LD_TRACE_LOADED_OBJECTS=1
$ ./hello

  linux-vdso.so.1 (0x00007ffc1ad96000)
  libc.so.6 => /usr/lib/libc.so.6 (0x00007fb017147000)
  /lib64/ld-linux-x86-64.so.2 (0x00007fb017503000)

Those files do not exist on Android, because even if Android is based on Linux, it’s still a different operating system.

See the line about libc.so.6 ? That’s the library that contains the code for the printf function we just used.

We call such a library “the libc”, which is a bit misleading.

There are several implementations of “the libc”. On your laptop you are probably using glibc. There are other implementations like musl for instance. Android uses yet an other implementation called bionic.

So if want to achieve our goal (running the “Hello, world” C program on a simulator and on a phone), we have to do two things:

Make sure we can tell the compiler about the CPU architecture
Make sure we use the correct libc

This is tricky because by default, compilers use the same CPU architecture and the same libc used by the operating system they run on².

This is what we call “cross-compilation”.

Using the NDK to compile by hand #

Google provides a set of tools know as the NDK in order to help us cross-compiling code for Android.

Here we are using version r16.

If we download and extract the NDK, here’s how we can compile and link our “Hello, World” program:

export NDK_ROOT=/path/to/ndk

${NDK_ROOT}/toolchains/llvm/prebuilt/linux-x86_64/bin/clang
  --target=x86_64-none-linux-android
  --gcc-toolchain=${NDK_ROOT}/toolchains/x86_64-4.9/prebuilt/linux-x86_64
  --sysroot=${NDK_ROOT}/sysroot
  -isystem ${NDK_ROOT}/sysroot/usr/include/x86_64-linux-android
  -o  hello.c.o -c hello.c

${NDK_ROOT}/toolchains/llvm/prebuilt/linux-x86_64/bin/clang
  --target=x86_64-none-linux-android
  --gcc-toolchain=${NDK_ROOT}/toolchains/x86_64-4.9/prebuilt/linux-x86_64
  --sysroot  ${NDK_ROOT}/platforms/android-21/arch-x86_64
  -pie hello.c.o -o hello

Some notes:

We have to specify the Android API level (21 here) like in any other project targeting Android
Most of the magic is done by the --sysroot, --gcc-toolchain and --target options.
We have to specify -fPIE, a flag that means position independant executable. It serves the same kind of purpose as the -fPIC flag we met in a previous article.

And now we can upload and run the binary on the x86_64 simulator:

$ adb push hello /data/local/tmp
$ adb shell /data/local/tmp/hello
Hello, world

We can do the same thing for arm of course, but note how subtle the changes are to go from x86_64 to arm:

export NDK_ROOT=/home/dmerej/Android/Sdk/ndk-bundle

${NDK_ROOT}/toolchains/llvm/prebuilt/linux-x86_64/bin/clang
  --target=armv7-none-linux-androideabi
  --gcc-toolchain=${NDK_ROOT}/toolchains/arm-linux-androideabi-4.9/...
  --sysroot=${NDK_ROOT}/sysroot
  -isystem ${NDK_ROOT}/sysroot/usr/include/arm-linux-androideabi
  -o  hello.c.o -c hello.c

${NDK_ROOT}/toolchains/llvm/prebuilt/linux-x86_64/bin/clang
  --target=armv7-none-linux-androideabi
  --gcc-toolchain=${NDK_ROOT}/toolchains/arm-linux-androideabi-4.9/...
  --sysroot  ${NDK_ROOT}/platforms/android-21/arch-arm
  -pie hello.c.o -o hello

And if we try again:

$ adb push hello /data/local/tmp
$ adb shell /data/local/tmp/hello
Hello, world

Success!

Our next objective is to run the cpp_demo executable we used to test the Chuck Norris library in part 1 on our phone and on the Android simulator.

Things is going to be trickier because of the sqlite3 dependency and the fact that the code is written in C++.

But surely there is a better way to than guessing how to invoke the compilers and linkers.

Conan to the rescue #

Of course, we could have used the native plug-in of Android Studio directly.

Instead we will use Conan, the tool we talked about in part 2.

Using Conan is a good way to abstract the above complexity, without loosing any of the control (like depending on an IDE plug-in does).

Plus, we can apply what we learn using Conan in other contexts such as writing C++ code on iOS.

The standalone toolchain #

If you a look at the contents of the Android NDK, you’ll soon realize there are lots of stuff there.

Among the cross-compiler binaries, sysroot, include directories we used in the previous section we also have a full build system based on Makefiles called ndk-build.

The NDK also contains support files for every Android API from 14 to 27 and every processor architecture (aarch64, arm, mips, x86, x86_64).

That’s a lot of stuff and it takes about 3G of disk space.

However, in build/tools there’s a bash script called make-standalone-toolchain.sh. Let’s take a look:

Creates a toolchain installation for a given Android target.

The output of this tool is a more typical cross-compiling toolchain. It is intended to be used with existing build systems such as autotools.

Ah-ah! Sound like what we need, especially if later on we start depending on a library built with autotools from the ChuckNorris project.

Build requirements #

It’s now time to introduce a new Conan feature, the build requirements.

Basically build requirements are packages you only need when building something from sources. See the Conan docs for more information.

So, here’s the plan:

First let’s have a recipe to download and extract the NDK (hopefully just once)
Then let’s have a second recipe to run the make_standalone_toolchain script and set up what we need to cross-compile for Android with CMake.

Note before we begin: I’ll only show you a small fraction of the recipes code. This is for educational purposes: there are a lots of details I have to omit in order to keep things readable.

The NDK recipe #

Here is a simplistic recipe for the NDK:

class AndroidndkConan(ConanFile):
    name = "android-ndk"
    version = "r16"
    settings = "os_build", "arch_build"
    ...

    def source(self):
        url = "https://...android-ndk-%s-linux-x86_64.zip" % self.version
        tools.download(url, "ndk.zip")
        tools.unzip("ndk.zip", keep_permissions=True)
        tools.unlink("ndk.zip")

    def package_info(self):
        tools_path = os.path.join(self.package_folder, "build", "tools")
        ...
        env_info = self.env_info
        env_info.PATH.append(os.path.join(tools_path)

The source() method does nothing but fetching and extracting the NDK.

The package_info() is more interesting: it adds the build/tools directory in the env_info.PATH variable. Any recipe that has the NDK package as build requirement can thus call self.run() to run any binary from the tools directory, since PATH will be set accordingly.

The Android toolchain recipe #

We can now move on to the next recipe, the Android toolchain:

class AndroidtoolchainConan(ConanFile):

    name = "android-toolchain"
    lib_version = "r16"
    package_version = "r4"
    version = "%s-%s" % (lib_version, package_version)
    license = "GPL/APACHE2"
    url = "https://github.com/lasote/conan-android-toolchain"
    settings = "os", "arch", "compiler"
    build_requires = "android-ndk/r16@dmerej/test"
    ...

    def build(self):
      ...

      toolchain = get_toolchain_str(self.settings.arch)

      command = (
        "make-standalone-toolchain.sh"
          "--verbose"
          "--toolchan=%s",
          "--platform=android-%s"
        ) % (toolchain, self.settings.os.api_level)
        self.run(command)


    def package_info(self):
        sysroot = os.path.join(self.package_folder, "sysroot")

        self.env_info.CC =  ...
        self.env_info.AR = ...

        self.env_info.CFLAGS = ...

As you can see, we are able to run the make-standalone-toolchain.sh script directly with self.run().

The other important part is the package_info() method, which :

Set the sysroot
Tells the consumers of the package about the path to the binaries used during compilation and linking (CC and AR)
Set some compile flags via the CFLAGS variable.

Remember when we built our C binary by hand?

We can find traces of this data in the command line we used:

${NDK_ROOT}/.../bin/clang                  <- this is CC
  --sysroot  ${NDK_ROOT}/...               <- this is the sysroot
  --target=armv7-none-linux-androideabi \
  --gcc-toolchain=..                     | <- those are compile flags
  -pie                                  /
  hello.c.o
  -o hello

Note that the recipe depends on the architecture we want our compiled code to run on. This means we are going to build one android-toolchain package per CPU architecture. Also note that the NDK recipe has a setting to know about the Android API level.

Where can we specify all these configuration values?

Android Profiles #

There’s an elegant way to solve this.

We create a global Conan profile in ~/.conan/profiles/android:

[build_requires]
android-toolchain/r16@dmerej/test

[settings]
os=Android
os_build=Linux
os.api_level=21
compiler=clang
compiler.version=5.0

[options]
*:pic = True

There we define all the common settings between all the Android configurations (like the API level, the compiler and the compiler version).

Note the *:pic in the [options] section. This will make sure that everything is built with position independent code, a requirement for anything that runs on Android.

Then if we need to build form x86_64, we can invoke Conan this way:

$ conan install --profile android --setting arch=x86_64

All the settings from the android profile will be used, plus the arch we just set on the command line.

Cross compiling sqlite3 #

Let’s cross-compile sqlite3 for Android:

$ conan create . dmerej/test --profile android --setting arch=x86_64
sqlite3/3.21.0@dmerej/test: Exporting package recipe
...
Cross-build from 'Linux:x86_64' to 'Android:x86_64'
...
Requirements
    android-toolchain/r16@dmerej/stable
Packages
    android-toolchain/r16@dmerej/test
...
sqlite3/3.21.0@dmerej/test: Calling build()
-- Android: Targeting API '21' with
      architecture 'x86_64',
      ABI 'x86_64',
      and processor 'x86_64'
...
sqlite3/3.21.0@dmerej/test: Package built

Done :)

Cross compiling the chucknorris library #

Let’s now try to build ChuckNorris:

$ cd cpp
$ mkdir -p build/android/x86_64
$ cd build/android/x86_64
$ conan install ../../.. --profile android --setting arch=x86_64
....
Cross-build from 'Linux:x86_64' to 'Android:x86_64'
sqlite3/3.21.0@dmerej/test: Already installed!

OK, now we are ready to build:

$ cd build/android/x86_64
$ cmake -GNinja ../../..
$ ninja
-- The C compiler identification is GNU 7.3.0
-- The CXX compiler identification is GNU 7.3.0
-- Check for working C compiler: /bin/cc
-- Check for working C compiler: /bin/cc -- works
...
-- Check for working CXX compiler: /bin/c++
-- Check for working CXX compiler: /bin/c++ -- works
...
CMake Error at build/android/x86_64/conanbuildinfo.cmake:452 (message):
  Incorrect 'clang', is not the one detected by CMake: 'GNU'

Uh-Oh: CMake is using the default compiler /bin/cc. This is not going to work.

But Conan knew how to cross-compile sqlite3 for Android! Can’t we tell Conan to build chucknorris too?

Creating a recipe for chucknorris #

Of course we can! Let’s go to cpp/ChuckNorris and run conan new:

$ conan new ChuckNorris/0.1 --source

Then let’s edit the generated conanfile.py to have:

from conans import ConanFile, CMake


class ChucknorrisConan(ConanFile):
    name = "ChuckNorris"
    version = "0.1"
    license = "MIT"
    url = "https://github.com/dmerejkowsky/chucknorris"
    description = "Chuck Norris does not need a description"
    settings = "os", "compiler", "build_type", "arch"
    options = {"shared": [True, False]}
    default_options = "shared=False"
    generators = "cmake"
    exports_sources = "CMakeLists.txt", "src/*", "include/*"

    def build(self):
        cmake = CMake(self)
        cmake.configure()
        cmake.build()

    def package(self):
        self.copy("cpp_demo" dst="bin", keep_path="false")

Note that instead of having a source() method to fetch the sources from a remote location (as we did for sqlite3), we instead use exports_sources to tell Conan about the sources it needs to build the package.

Also note how we copy the cpp_demo binary in the package() method. We’ll use this to check that the binary we built with Conan can actually run.

Then we create the chucknorris package:

$ cd cpp/ChuckNorris
$ conan create .  dmere/test --profile android --setting arch=x86_64
ChuckNorris/0.1@dmerej/test: Exporting package recipe
...
Cross-build from 'Linux:x86_64' to 'Android:x86_64'
...
-- Build files have been written to: ...
../bin/clang++
  --target=x86_64-none-linux-android
  --gcc-toolchain=...
  -sysroot=...
  -isysroot=...
  -fPIC
  -fPIE -pie
  .../sqlite3.a
  ...

Well, it did build, and the commands ran by Ninja closely resemble the ones we wrote by hand in the previous section.

Let’s check it runs!

$ cd ~/.conan/data/ChuckNorris/0.1/dmerej/test
$ cd package/<hash>/bin
$ adb push cpp_demo /data/local/tmp/
$ adb shell /data/local/tmp/cpp_demo
CANNOT LINK EXECUTABLE "/data/local/tmp/cpp_demo":
  library "libc++_shared.so" not found

What?

The libc++ #

Remember when we talked about the libc? Well, for C++ on Android there are two possible choices.

You have to choose between the gnustl library, or the libc++ library, and they both come in two flavors (static or shared).

By default, our binary was compiled to link with shared version of libc++, hence the file name: libc++_shared.so.

Fortunately, we can use an other feature of Conan to help us: the imports() function. This function gets called before building and can be used to copy files from the dependencies packages.

Here’s what we can do:

Use imports() to copy libc++_shared.so from the android-toolchain package to the build directory.
Use keep_imports so that the imported files do not get removed from the build directory.
Add a copy() call in the package() method so that the libc++_shared.so file is present in the final package.

This is known as “repackaging” in Conan parlance.

class ChucknorrisConan(ConanFile):
    ...
    keep_imports = True

    def imports(self):
        self.copy("*libc++_shared.so", dst="lib")

    def package(self):
        self.copy("bin/cpp_demo", dst="bin", keep_path=False)
        self.copy("lib/libc++_shared.so", dst="lib", keep_path=False)

Now we can try again, using the LD_LIBRARY_PATH environment variable to tell the linker where to look for the shared libraries:

$ conan create .  dmere/test --profile android --setting arch=x86_64
$ cd ~/.conan/data/ChuckNorris/0.1/dmerej/test
$ cd package/<hash>/
$ adb push bin/cpp_demo /data/local/tmp
$ adb push lib/libc++_shared.so /data/local/tmp/
$ adb shell
$ cd /data/local/tmp
$ LD_LIBRARY_PATH=lib ./cpp_demo
When a zombie apocalypse starts, Chuck Norris doesn't try to survive.
The zombies do.

Hooray!

We can also check it works on a arm phone too:

$ cd cpp/conan/android-toolchain
$ conan create . dmerej/test --profile android --setting arch=armv7
$ cd cpp/conan/sqlite3
$ conan create . dmerej/test --profile android --setting arch=armv7
$ cd cpp/ChuckNorris
$ conan create . dmerej/test --profile android --setting arch=armv7
$ cd ~/.conan/data/ChuckNorris/0.1/dmerej/test
# This would be a different hash since the settings have changed:
$ cd package/<hash>/
$ adb push bin/cpp_demo /data/local/tmp
$ adb push lib/libc++_shared.so /data/local/tmp/
$ adb shell
$ cd /data/local/tmp
$ LD_LIBRARY_PATH=lib ./cpp_demo
Giraffes were created when Chuck Norris uppercutted a horse.

Conclusion #

We finally managed to run some C++ code directly on Android.

However, Android applications are written in Java (or Kotlin), so we still need to wrap the C++ library in Java.

See you in part 7 for the next episode :)

You may be using other architectures without realizing it, but don’t worry too much about it. ↩︎
It’s actually a good thing. Otherwise you won’t be able to run and debug the binaries you’ve just compiled. ↩︎

Adventures in CI land

Mon, 04 Jun 2018 18:21:30 +0000

Today at work I wrote a CI script to test a react application, and it turned out to be a bit trickier than expected.

Let’s try and reproduce the interesting issues I had and how I solved them.

Setting the stage #

Here’s what you are going to need if you want to try and reproduce what I did.

Node.js, yarn
Python3 and pipenv
The chromedriver binary.

Let’s start by creating a simple React application:

$ yarn global add create-react-app
$ create-react-app hello
$ cd hello
$ yarn

We now have a beautiful React application running in our favorite browser.

Let’s edit the App.js file to display Hello, world instead:

import React, { Component } from 'react';

class App extends Component {
  render() {
    return (
      <p>Hello, world!</p>
    );
  }
}

export default App;

Adding some end-to-end tests #

Let’s use pipenv to create a virtualenv with what we need:

$ pipenv install pytest
$ pipenv install selenium
$ pipenv shell

Now let’s add some end-to-end using selenium and pytest. ¹

# in test_hello.py
import selenium.webdriver


def test_home():
    driver = selenium.webdriver.Chrome()
    driver.get("http://127.0.0.1:3000")
    assert "Hello, world!" in driver.page_source

We can now run the tests with pytest as usual:

$ pytest
collected 1 item

test_hello.py .                            [100%]
1 passed in 4.77 seconds

OK, it works!

Now let’s imagine you have a team of people working on the application, and you would like these tests to run any time someone creates a merge request on this repo.

This is known as continuous integration (CI for short) and, trust me on this, it works a lot better than telling your teammates to remember to run the tests before submitting their changes for review!

Writing the CI script #

We use GitLab at work and are big fan of its CI features.

If you don’t know GitLab CI at all, here’s how it works:

You install and configure the gitlab-runner program on some machines (called runners)
Then you write a .gitlab-ci.yml file that contains the job description.

At my job we prefer to keep the .gitlab-ci.yml simple, and keep the code of the CI scripts separate, like this:

(note how we use python3 -m pipenv instead of just pipenv. This is to make sure pipenv runs with the expected version of Python)

# in .gitlab-ci.yml

stages:
 - check

check:
  stage: check
  script:
    - python3 -m pipenv install
    - python3 -m pipenv run python ci.py

# in ci.py

def main():
    # Call yarn here


if __name__ == "__main__":
    main()

We do this because it makes it easy to reproduce build failures found during CI locally. Any developer on the team can run python ci/ci.py on their machine directly instead of trying to copy/paste code from the yaml file.

Going headless #

Right now, the selenium tests use a full-fledged Chrome to run the tests. This is nice for the developers, but not so nice on a GitLab runner.

It would be much better to have those running in a headless Chrome instead, i.e without any GUI.

Let’s fix that by adding a --headless option:

# in conftest.py

import pytest

def pytest_addoption(parser):
    parser.addoption("--headless", action="store_true")


@pytest.fixture
def headless(request):
    return request.config.getoption("--headless")

# in test_hello.py

from selenium.webdriver.chrome.options import Options as ChromeOptions

def test_home(headless):
    options = ChromeOptions()
    options.headless = headless
    driver = selenium.webdriver.Chrome(chrome_options=options)
    ...

Now if we run pytest with the --headless option, the headless parameter of the test_home function will be set to True by pytest. That’s how pytest fixtures work.

Anyway, we can now check this is working by running:

$ pytest --headless

Writing the CI script #

So now we are faced with a new challenge: we need to run yarn start before running pytest, and kill the React script when the selenium tests have finished.

A nice way to do this in Python is to use the with statement, so let’s do that:


class BackgroundProcess:
    """ Run `yarn start` in the background. Ensure the yarn process
    is killed when exiting the `with` block

    """
    def __init__(self):
        self.process = None

    def __enter__(self):
        self.process = subprocess.Popen(["yarn", "start"])

    def __exit__(self, type, value, tb):
        self.process.terminate()

def main():
    with BackgroundProcess("yarn", "start"):
        subprocess.run(["pytest", "--headless"], check=True)


if __name__ == "__main__":
    main()

The __enter__ method will be called right before the contents of the with block, so before pytest starts. Then the __exit__ method will be called after pytest is done, even if an exception occurred, passing data about the exception as arguments to the __exit__() method. Since we don’t want do to anything other than re-raise if this happens, we just ignore them.

Anyway, this is much more readable than using try/except/finally, don’t you think?

We still need a tiny fix: by default, yarn start will open a new tab on our browser. This was great while we were working on the JavaScript code, but here we are working on the CI script, so we’d prefer to disable this behavior.

Fortunately, all we have to do is to set the BROWSER environment variable to NONE:

class BackgroundProcess:
    ...

    def __enter__(self):
        env = os.environ.copy()
        env["BROWSER"] = "NONE"
        self.process = subprocess.Popen(self.cmd, env=env)

Note: you may wonder why we did not just set the BROWSER environment variable directly in the .gitlab-ci.yml file. This would have worked, but here we create a special copy of the current environment variables, and we set the BROWSER environment variable just for the yarn process. Why?

Well, if you think of environment variables as nasty global variables (and you should: the environment of a process is just a big mutable shared state), it makes sense to limit their scope this way.

Anyway, back to the main topic:

The bug #

The rest of the article assumes you are using Linux. Things may work a bit differently (or not at all) on other operating systems.

Let’s see if the CI script works.

$ python ci.py
yarn run v1.7.0
$ react-scripts start
Starting the development server...
...
1 passed in 4.77 seconds

Let’s run it a second time just to check that the yarn process was indeed killed:

$ python ci.py
? Something is already running on port 3000. Probably:
  hello (pid 16508)

Would you like to run the app on another port instead? (Y/n)

Uh-oh.

Let’s run pgrep to check that the yarn process is dead:

$ pgrep yarn
[err 1]

The yarn process is dead. What gives ?

If we take a look at the .terminate() implementation, here’s what we find:

# in /usr/lib/python3.6/subprocess.py

class Popen:

      def send_signal(self, sig):
          """Send a signal to the process."""
          # Skip signalling a process that we know has already died.
          if self.returncode is None:
              os.kill(self.pid, sig)

      def terminate(self):
          """Terminate the process with SIGTERM
          """
          self.send_signal(signal.SIGTERM)

So, terminate() just sends the SIGTERM signal using the process ID (pid). The bug’s not there.

The naked truth #

The truth is we’ve just created an orphan (we’re monsters!)

When we ran yarn start, the yarn process looked at a section named start in the package.json and found something like this:

{
...
  "scripts": {
    "start": "react-scripts start",
    ...
  }
}

It then created a child process, namely react-scripts start, with a different PID.

So when we killed the parent process, the node process became an orphan since its parent was dead (poor little process).

On Linux at least, all orphans process get automatically re-attached to the first ever process that was created since the machine booted. (systemd on my machine). This process always has a PID equal to 1 and is often referred to as init.

We can check that by running pstree:

$ pstree
systemd─┬                               <- PID 1
...
        ├─node──                        <- our poor orphan
...
        ├─plasmashell─┬
                      ├─konsole─┬─zsh─  <- our shell

So how do we make sure the node child process gets killed too?

There are some fancy ways to fix these kind of problems (we could use cgroups for instance), but we can do it just with the Python stdlib.

Turns out we can use the start_new_session argument in the subprocess.Popen() call. This will create a session and attach the yarn process (and all its children) to it.

Then we can send the SIGTERM signal to the PID of the parent, and all the processes in the session will receive it:

import os
import signal

def __enter__(self):
  ...
  self.process = subprocess.Popen(self.cmd, start_new_session=True)

def __exit__(self):
    os.killpg(self.process.pid, signal.SIGTERM)

Now if we re-run our script, we can see that neither yarn or node remain alive when the CI script terminates:

$ python ci.py
$ pgrep yarn
[err 1]
$ pgrep node
[err 1]

That’s all for today. Cheers!

This is not the first time I’ve used those tools to write end-to-end tests for Web application. See Porting to pytest for instance. ↩︎

I don't need types

Sun, 27 May 2018 10:17:17 +0000

This is a story about me being wrong about tests and type systems.

It’s also a story of me trying something new and learning a few things.

Let’s start at the beginning.

C++ vs Python #

A few years ago, I was working in a team where the two main languages used were C++ and Python.

Here’s what I used to think:

C++ sucks:

I have to specify types everywhere:
- I need to know the difference between int, long, uin8_t, size_t even if all I want is a goddamn integer.
- I have to duplicate the signatures of the methods and functions in the header and the source files
I never know if a compiler warning or error indicates a bug or not
There’s still a lot of bugs the compiler does not catch (race conditions, dangling pointers ….)
Refactoring is hard: when you change a function signature, you have to fix the code everywhere before you can even run some tests.
Writing tests in hard: You can’t really write isolation tests because mocking and dependency injection in C++ don’t really work. Plus, gtest sucks.

Python rocks:

I don’t have to think about types if I don’t want to. If I really need it, I can write an abstract base class, but most of the time duck-typing “just works”.
I don’t need compile-time checks because if I misspell a variable or function name, or if I forgot to import a module, I’ll get a test failure immediately.
For each bug I can quickly add a non-regression test.
Refactoring is easy: all you have to do is run the tests, look at the failures and fix the code. When the tests pass, you know you’re done.
Writing tests is easy: dependency injection is trivial thanks to duck-typing, and you can monkey-patch or mock everything! Plus, pytest is awesome!
Static analysis for Python does not work, pylint is slow, and hard to configure. There are tons of bugs it does not catch, and it is often wrong.
Types annotations are useless (after all, the interpreter does almost nothing with them), except maybe for documentation purposes.

I now think most of these statements are wrong, but it was what I believed at the time.

Changing my mind #

So there I was. Tools and type systems did not matter, all that matter were tests and how easy it was to write them. I did not need a type system because tests were enough.

⁂

Boy, was I wrong… Here’s a list of things that contributed to burst my bubble.

Some linters #

pyflakes #

I started using pyflakes with vim-ale¹.

pyflakes is very easy to use, requires no configuration and is fast.

Suddenly a whole bunch of bugs disappeared: pyflakes is very good at finding misspelled variables or missing imports.

So maybe writing tests just to find misspelled variables or missing imports is overkill?

Of course, pyflakes does not catch other errors like calling a function with an incorrect number of arguments but still, it’s quite nice to catch these errors right after the file is saved, instead of later when a test fails.

pylint #

I’ve already mentioned how pylint can be very useful if you take the time to configure it properly, so I won’t repeat myself here.

Still, I learned that Python static analyzers did not have to suck after all, and could find defects faster than tests.

Today I use pyflakes, pylint and a few other linters for all my Python projects. You can read more about this in How I lint my Python.

At this point I had changed my mind a little bit. They were tools other than tests that could be useful. But I was still thinking that types did not help nearly as much as tests did.

A talk #

And then I watched a talk called ideology, by Gary Bernhardt. I highly recommend it.

Excerpt:

[This is important] mostly because it will make you better programmers, but also because it will stop you from making angry Hacker News comments.

I won’t summarize the talk here, but it helped me realizing what I really meant when I claimed “I have tests so I don’t need types”.

But this was rather abstract. Changing my mind about tests required me working with programming languages other than C++ and Python.

JavaScript #

Last year I had to make a refactoring in a JavaScript project.

There were no tests at all, and adding them would have been pretty challenging.

But there were flow type annotations everywhere. The errors weren’t always easy to understand and even sometimes misleading, but flow did help me gain confidence that I was not breaking everything during the refactoring.

That showed me that type annotations in a “dynamic” languages could actually be worth it.

Still, I was convinced we should have written tests for this project since day one, and not let the production code grow without tests.

Type annotations were required because they were no tests, and surely tests alone would have suffice.

Rust #

Rust was the last nail in the coffin.

I started re-writing a Python project in Rust, and suddenly all this stuff about “if it compiles, it works”, and “types system make unit tests unnecessary” finally started to make sense.

Here’s what I learned using Rust:

Specifying types is easy: all you need to annotate are function parameters and return values, and everything else is inferred by the compiler.
Error messages and warnings almost always indicate a bug or an inefficient way of doing things.
And types can actually help you!

Let me give you a few examples:

Anything that can fail returns a type (like Option or Result) that forces you to handle errors.
There is a Copy trait that tells you whether a type has copy semantics versus move semantics.
The Send and Sync traits define how you can use a type across threads.
and more!

In the mean time, using TDD with Rust is enjoyable and even recommended in the Rust book. I even wrote a test to make sure a certain bug would be caught at compile time.

And that’s how I completely changed my mind: type system do not have to suck, they can be very useful, and you can combine them with tests to get the best of two worlds.

Conclusion and teaser #

You see, we are terrible at spotting errors in our own code.

That’s why we try and multiply the techniques hoping each of them will find different types of mistakes:

We ask other humans to look at our code, during code reviews or peer programming
We ask other humans to find defects in the code for us, and we call this “a QA process”
We use static analyzers to find issues in the code automatically
We use tests to try and prove that the code works as it should
We use TDD to look at the code both from the ‘production’ perspective (when we go from red to green), and from a ‘quality’ perspective (when we go from green to refactor).
And we use static types or type annotations to improve correctness of the code in a way that complements all of the above methods.

That’s all for today.

If you still believe that Python does not need types if you have a good tests, I’ll have some very concrete examples to show you in a future blog post.

Cheers!

If using this kind of linters sounds interesting to you, you can read my blog post on the subject. ↩︎

A boolean's story

Fri, 18 May 2018 13:50:38 +0000

Earlier this month I told you about my pet project in Rust.

As a reminder, it’s a tool named rusync which contains some of the functionality offered by the rsync command-line tool.

Today I’d like to talk about a feature I’ve added recently, and take this opportunity to show you a few principles of good design along the way.

You can find the code on github and download it from crates.io.

The algorithm #

You run rusync by calling it with a source and destination directory, like so:

$ rusync src dest

The algorithm is pretty straightforward:

Recurse through all directories and files in the src directory
For each source file (for instance src/foo/bar.txt), check if the destination file (dest/foo/bar.txt in this case) exists. If it does exist, get its mtime (that is, the date it was last modified). Then, if the source is more recent than the destination, or if the destination is missing, proceed and copy the whole contents of src/foo/bar.txt to dest/foo/bar.txt.

One of the issues I had to deal with is the preservation of permissions. Suppose one of the files is executable, let’s say src/foo/bar.exe, then the destination (dest/foo/bar.exe) should be executable too.

So, after the destination file has been written, we should apply the necessary transformations so that the permissions of the destination file match the permissions of the source file.

The code #

Here’s what the implementation looked like at the time.

There’s a bug waiting to happen in the following code, can you find it ?

Let’s start by main():

#[derive(Debug, StructOpt)]
#[structopt(name = "rusync")]
struct Opt {
    #[structopt(parse(from_os_str))]
    source: PathBuf,

    #[structopt(parse(from_os_str))]
    destination: PathBuf,
}

fn main() {
    let opt = Opt::from_args();
    let source = &opt.source;
    let destination = &opt.destination;
    let mut syncer = Syncer::new(&source, &destination);
    syncer.sync()?;
}

We declare a struct Opt which will contain our command line options, then in main() we parse the command line arguments and instantiate a new Syncer object.

Here are the relevant parts of the Syncer implementation:


impl Syncer {
    fn new(source: &Path, destination: &Path) -> Syncer {
        Syncer {
            source: source.to_path_buf(),
            destination: destination.to_path_buf(),
            // ...
        }
    }


    fn sync(&mut self) -> io::Result<()> {
        let top_dir = &self.source.clone();
        self.walk_dir(top_dir)?;
        Ok(())
    }

    fn walk_dir(&mut self, subdir: &Path) -> io::Result<()> {
        for entry in fs::read_dir(subdir)? {
            let entry = entry?;
            let path = entry.path();
            if path.is_dir() {
                let subdir = path;
                self.walk_dir(&subdir)?;
            } else {
                self.sync_file(&entry)?;
            }
        }
        Ok(())
    }

    fn sync_file(&mut self, entry: &DirEntry) -> io::Result<()> {
        let rel_path = get_rel_path(&entry.path(), &self.source)?;
        let parent_rel_path = rel_path.parent();
        let to_create = self.destination.join(parent_rel_path);
        fs::create_dir_all(to_create)?;

        let src_entry = entry::Entry::new(&desc, &entry.path());

        let dest_path = self.destination.join(&rel_path);
        let dest_entry = entry::Entry::new(&desc, &dest_path);
        fsops::sync_entries(&src_entry, &dest_entry)?;
        Ok(())
    }

As you can see, the bulk of the work is done by the sync_file function, which calls fsops::sync_entries with a Entry type.

The Entry type is a container for the file path and its metadata. Reading metadata about the file (such as its mtime) is expensive, so we do that once in the Entry::new() function and the rest of the code can then use the public functions of the Entry struct to retrieve info about the file in question.

pub struct Entry {
    path: PathBuf,
    metadata: Option<fs::Metadata>,
    exists: bool,
}

impl Entry {
    pub fn new(description: &str, entry_path: &Path) -> Entry {
        let metadata = fs::metadata(entry_path).ok();
        Entry {
            metadata: metadata,
            path: entry_path.to_path_buf(),
            exists: entry_path.exists(),
        }
    }

    pub fn path(&self) -> &PathBuf { &self.path }
    pub fn exists(&self) -> bool { self.exists }

    pub fn metadata(&self) -> Option<&fs::Metadata> {
        self.metadata.as_ref()
    }
}

Finally, let’s take a look at the sync_entries function:

const BUFFER_SIZE: usize = 100 * 1024;

pub fn sync_entries(src: &Entry, dest: &Entry) -> io::Result<()> {
    let more_recent = more_recent_than(&src, &dest)?;
    if more_recent {
        return copy_entry(&src, &dest);
    }
    Ok(())
}

pub fn copy_entry(src: &Entry, dest: &Entry) -> io::Result<()> {
    let src_path = src.path();
    let src_file = File::open(src_path)?;
    let dest_file = File::create(dest_path)?;
    let mut buffer = vec![0; BUFFER_SIZE];
    loop {
        let num_read = src_file.read(&mut buffer)?;
        if num_read == 0 {
            break;
        }
        dest_file.write(&buffer[0..num_read])?;
    }
    copy_perms(&src, &dest)?;
}

fn copy_perms(src: &Entry, dest: &Entry) -> io::Result<()> {
    let src_meta = &src.metadata();
    let src_meta = &src_meta.expect("src_meta was None");
    let permissions = src_meta.permissions();
    let dest_file = File::create(dest.path())?;
    dest_file.set_permissions(permissions)?;
    Ok(())
}

As you can see, the sync_entries function takes care of copying the source file contents to the dest file chunk by chunk, and then calls copy_perms().

A new option #

After a while, it occurred to me that rusync will be annoying to use in case we copy from an ext4 partition to a fat32 partition.

Indeed, Linux cannot set permissions on a fat32 partitions, so the code from copy_perms would surely fail in this case.

I decided what I needed was a flag on the command line so that users of rusync could choose to turn off the “copy permissions” feature.

Well, this does not seem to hard, does it ? We just need to pass around a boolean called preserve_permissions all the way from the Opt struct in main to the sync_entries in function, across the Syncer struct.

Let’s do that!

First, let’s adapt main.rs:

struct Opt {

    #[structopt(long = "no-perms")]
    no_preserve_permissions: bool,

    // ...
}

fn main() {
    let opt = Opt::from_args();
    let source = &opt.source;
    let destination = &opt.destination;
    let preserve_permissions = !opt.no_preserve_permissions;

    let mut syncer = Syncer::new(&source, &destination);
    syncer.preserve_permissions(preserve_permissions);

    // ...
}

Then let’s adapt the Syncer:

impl Syncer {
    fn new(source: &Path, destination: &Path) -> Syncer {
        Syncer {
            source: source.to_path_buf(),
            destination: destination.to_path_buf(),
            // ...
            preserve_permissions: true,
        }
    }

    pub fn preserve_permissions(&mut self, preserve_permissions: bool) {
        self.preserve_permissions = preserve_permissions;
    }

    fn sync_file(&mut self, entry: &DirEntry) -> io::Result<()> {
        // ...
        fsops::sync_entries(&src_entry, &dest_entry, preserve_permissions);
        Ok(())
    }

And then let’s adapt sync_entries and copy_entry:

pub fn copy_entry(src: &Entry, dest: &Entry,
                   preserve_permissions: bool,) -> io::Result<()>  {
    let more_recent = more_recent_than(&src, &dest)?;
    if more_recent {
        return copy_entry(&src, &dest, preserve_permissions);
    }
    Ok(())
}

pub fn copy_entry(src: &Entry, dest: &Entry,
                  preserve_permissions: bool) -> io::Result<()> {
    let src_path = src.path();
    let src_file = File::open(src_path)?;

    // ...
    if preserve_permissions {
      copy_perms(&src, &dest)?;
    }
}

Now we run the tests, and it looks like nothing broke. Hooray!

Time for refactoring #

There are several things we can change to make the code more readable.

First, we can get rid of the double negative in main.rs:

impl Opt {
struct Opt {
    #[structopt(long = "no-perms")]
    no_preserve_permissions: bool,

    // ..
}

+ impl Opt {
+     fn preserve_permissions(&self) -> bool {
+         !self.no_preserve_permissions
+     }
+ }

fn main() {
    let opt = Opt::from_args();
    let source = &opt.source;
    let destination = &opt.destination;
-   let preserve_permissions = !opt.no_preserve_permissions;
+   let preserve_permissions = opt.preserve_permissions();
    let mut syncer = Syncer::new(&source, &destination);
    syncer.preserve_permissions(preserve_permissions);
  }
}

Yeah, I know it’s a little change, but it’s the kind of little details that matter in the long run. Double negatives are just as hard to understand in plain English as in code.

Then we have all these functions that take a boolean parameter.

As Uncle Bob would tell you, whenever you have a function that takes a boolean, you almost always want two functions instead.

So let’s think about our preserve_permissions boolean.

Turns out we can split sync_entries in two. One that only does the copy, and the other one that deals with permissions preservation.

We’ll keep the preserve_permission field in the Syncer, and only call the second function if we need to.

Let’s have a look at the patch:

- pub fn copy_entry(src: &Entry, dest: &Entry,
-                   preserve_permissions: bool,) -> io::Result<()>  {
+ pub fn copy_entry(src: &Entry, dest: &Entry) -> io::Result<()> {

  // ..

-    if preserve_permissions {
-        copy_permissions(&src, &dest);
-    }
  Ok(())
}
- pub fn sync_entries(src: &Entry, dest: &Entry,
-                      preserve_permissions: bool) -> io::Result<()> {
+ pub fn sync_entries(src: &Entry, dest: &Entry) -> io::Result<(SyncOutcome)> {
     if more_recent {
-        return copy_entry(&src, &dest, preserve_permissions);
+        return copy_entry(&src, &dest);
}

impl Syncer {

    fn sync_file(*mut self, entry: &DirEntry) -> io::Result<()> {

         let dest_path = self.destination.join(&rel_path);
-        fsops::sync_entries(&src_entry, &dest_entry, self.preserve_permissions)?;
+        fsops::sync_entries(&src_entry, &dest_entry)?;
+        if self.preserve_permissions {
+            fsops::copy_permissions(&src_entry, &dest_entry)
+          }
}

It seems we just moved the logic about permissions preservation from the low-level fsops module, to the high-level Syncer module, but the consequences of this refactoring may be more profound than you think.

A new design #

Note how the functions in fsops now know nothing about the command-line flags.

That’s us trying to adhere to Single Responsibility Principle. The SRP more or less states that each module should only have one reason to change.

Code in syncer.rs will have to change if we want to customize the behavior of rusync (like deleting the extraneous files in the destination folder). On the other hand, code in fsops.rs will have to change if we want to modify low-level implementation details of rusync (like using a more efficient algorithm like rsync does).

Note that in the first case, we’ll of course have to add code in fsops.rst to delete files or directories, but we are OK with it because the code will likely don’t have to be modified, we’ll just have to add more functions. (That’s an other principle at play here called the Open/Close Principle).

Anyway, found the bug yet?

The bug #

I actually discovered the “bug waiting to happen” I talked about in the beginning when I re-ran the test suite after I performed the refactoring I just discussed. One of the tests started failing.

Can you try and find it just by looking at the patches above ?

There is a clue about the bug in one of the previous post of this blog.

If you want to find out, click on the “spoiler” below.

Toggle Spoiler

It’s the same kind of bug I mentioned in the non-isomorphic C++ refactoring article.

You see, in Rust some types implement the Drop trait, and have a .drop() method called when they go out of scope.

In our case, before our refactoring, the dest file was created twice. Once in the copy_entry() function, and an other time in the copy_perms() function. Usually, File::create truncates the file, but when the first file handle was destroyed, its contents were magically preserved. (Software is weird sometimes)

After the refactoring, the second File:;create() was called after the first file handle was closed, and the destination filed ended up empty (but with the correct permissions …)

Introducing rusync

Tue, 08 May 2018 17:59:06 +0000

Today I wrote my first “real” rust project.

It’s a re-write of rsync in Rust called rusync.

Here’s what its installation and usage look like:

$ cargo install rusync
$ rusync test/src test/dest
:: Syncing from test/src to test/dest …
-> foo/baz.txt
-> foo/bar.txt
 ✓ Synced 2 files (1 up to date)

You can find the sources on github.

Feedback request #

I wrote this because I wanted to give Rust a try.

If you’re already are a Rust developer, I’d appreciate it if you could give me a honest review of the code I wrote.

See the contact page for all the possible ways to reach me, and many thanks in advance!

What’s next #

Here’s a list of features I plan to implement

Option to delete extraneous files
Global progress bar

The last one is interesting: we need to recursively walk through all the files in the source folder in order to estimate the total size of the transfer, but we want to do that while the transfer is in progress.

That will be an opportunity to play a little bit with Rust concurrency features :)

Cheers!

Update: well, I implemented this features using a pipeline of std::sync::mpsc channels, and it worked great! Here’s the relevant commit if you want to take a look.

Where Software Lives

Thu, 12 Apr 2018 06:05:15 +0000

Last night I had a beer with some old friends and colleagues.

We were discussing a particular library and one of us explained he recently found patches from almost all of us in the git history.

And then he said (I’m paraphrasing a little):

Software lives in the minds of the people who wrote it. Code and documentation are nothing but distant echoes of their thoughts at a given time.

I could not agree more.

Code is all but a means to an end, try and remember that.

Cheers!

Let's Build Chuck Norris! - Part 5: Python and cffi

Sat, 07 Apr 2018 17:58:52 +0000

Note: This is part 5 of the Let’s Build Chuck Norris! series.

Last week we wrote Python bindings for the chucknorris library using ctypes.

We managed to get some Chuck Norris facts from a Python program.

On the plus side, we did not have to compile anything. Everything was done directly in Python.

There were a few issues, though:

We had to pass the path to the libchucknorris.so shared library to ctypes.cdll.LoadLibrary.
We had to duplicate information about parameters types of the C function inside our Python program and mistakes were easy to made.

An other way to wrap C code in Python is to use a C extension: that is, a Python module written in C. In this case the Python module actually takes the form of a shared library, and is thus loaded by the python interpreter at runtime.

There are many ways to create a Python C extension, from directly writing the C code (using the Python C API), to generating the C code and then compile it.

In the past, I’ve used tools like boost::python and swig for this task.

I only started using cffi recently, but I find it easier to use, and, contrary to the above tools, it is compatible with pypy, which is kind of awesome.

First try with cffi #

I you browse the documentation you will see that cffi can be used in several modes. There is a ABI mode and an API mode. The ABI mode resembles the technique we used with ctypes, because it involves loading chucknorris as a shared library.

We are going to use the API mode instead, where all the code is generated using the chucknorris.h header, which minimizes the chance of mistakes.

This means we can go back to building chucknorris as a static library. That way we won’t have to care about the location of the library, and the chucknorris code will be used at compile time.(Sorry for the little detour).

All we have to do is re-run cmake and ninja:

$ cd cpp/ChuckNorris/build/default
$ cmake -DBUILD_SHARED_LIBS=OFF ../..
$ ninja
[1/7] Building C object CMakeFiles/c_demo.dir/src/main.c.o
[2/7] Building CXX object CMakeFiles/chucknorris.dir/src/c_wrapper.cpp.o
[3/7] Building CXX object CMakeFiles/cpp_demo.dir/src/main.cpp.o
[4/7] Building CXX object CMakeFiles/chucknorris.dir/src/ChuckNorris.cpp.o
[5/7] Linking CXX static library lib/libchucknorris.a
[6/7] Linking CXX executable bin/c_demo
[7/7] Linking CXX executable bin/cpp_demo

Now, let’s write a Python build script for our C extension using the cffi builder:

build_chucknorris.py:

from cffi import FFI
ffibuilder = FFI()

ffibuilder.set_source(
    "_chucknorris",
    """
    #include <chucknorris.h>
    """,
)

ffibuilder.cdef("""
typedef struct chuck_norris chuck_norris_t;
chuck_norris_t* chuck_norris_init(void);
const char* chuck_norris_get_fact(chuck_norris_t*);
void chuck_norris_deinit(chuck_norris_t*);
""")

We instantiate a FFI object we call ffibuilder.
In ffibuilder.set_source() we give the builder the name of the C extension: _chucknorris. It’s common for C extension names to be prefixed with an underscore.
We also give the FFI builder the C code it needs to compile the code it generates. (Here we only need to include the <chucknorris.h> header, but in a real project you may add things like macros or additional helper code).
Finally we list the functions and types we want exposed in our C extension as C declarations – directly copy/pasted from the chucknorris.h header – and pass them as a string to ffibuilder.cdef().

Keeping things DRY #

“But wait a minute!”, I hear you say. “You said cffi was better than ctypes because we did not have to duplicate information about types, but now you are telling us we still need to copy/paste C declarations inside the call to .cdef()! What gives?”.

Well, it’s true we usually try to keep things DRY when we write code, (DRY meaning “don’t repeat yourself”).

However, when using cffi it does not matter that much. Not following DRY is only dangerous when the duplicated code does not change at the same time and gets out of sync.

Let’s say you break the API of your library (for instance by changing the number of arguments of a C function). If you don’t reflect the change in ffibuilder.def(), you will get a nice compilation error, instead of a crash or segfault like we experienced with ctypes.

Adding a setup.py #

With that out of the way, let’s add a setup.py file we can use while developing our bindings, install our code, and re-distribute to others:

setup.py:

from setuptools import setup, find_packages

setup(name="chucknorris",
      version="0.1",
      description="chucknorris python bindings",
      author="Dimitri Merejkowsky",
      py_modules=["chucknorris"],
      setup_requires=["cffi"],
      cffi_modules=["build_chucknorris.py:ffibuilder"],
      install_requires=["cffi"],
)

Here’s what the parameters do:

py_modules: the list of Python modules to install. We only got one, called chucknorris, that will use the _chucknorris C extension and expose a more “Pythonic” API.
setup_requires: what the setup.py script needs in order to build the extension.
cffi_modules: the list of Python objects to be called when building the extension. Here it’s the ffibuilder object defined in the build_chucknorris.py file.
install_requires: the list of the dependencies of our module once it has been built and installed. We also need cffi at runtime, not just for compiling the extension.

Finally, we can write the implementation of the chucknorris Python module:

chucknorris.py:

from _chucknorris import lib, ffi


class ChuckNorris:
    def __init__(self):
        self._ck = lib.chuck_norris_init()

    def get_fact(self):
        c_fact = lib.chuck_norris_get_fact(self._ck)
        fact_as_bytes = ffi.string(c_fact)
        return fact_as_bytes.decode("UTF-8")

    def __del__(self):
        lib.chuck_norris_deinit(self.c_ck)


def main():
    chuck_norris = ChuckNorris()
    print(chuck_norris.get_fact())


if __name__ == "__main__":
    main()

We start by importing code from the _chucknorris C extension. lib contains what has been wrapped – declared with ffibuilder.cdef() – , and ffi contains various cffi helpers.
We hide the lib.chuck_norris_init() and lib.chuck_norris_deinit() under the __init__ and __del__ methods. (Exactly what we did when going from the C++ constructor and destructors to the C functions, but the other way around)
In the get_fact() method, we call lib.chuck_norris_get_fact(). chuck_norris_get_fact() returns a “C string”, which is just a char* array that ends with \0. We pass it to ffi.string() to get a bytes object, suitable for holding this kind of data. And finally, we convert the list of bytes to a real string using decode().
Finally, when the chucknorris.py script is called, we use our nice Python class as if no C code ever existed :)

Running the builder #

After installing the cffi package, we can finally try and build the code:

$ python setup.py build_ext
running build_ext
generating ./_chucknorris.c
...
building '_chucknorris' extension
gcc ...
  -fPIC
  ...
  -I/usr/include/python3.6m
  -c _chucknorris.c
  -o ./_chucknorris.o
_chucknorris.c:493:14: fatal error: chucknorris.h: No such file or directory

What happened?

python setup.py build_ext found out how to use our ffibuilder object.
It generated some C code in a _chucknorris.c file
It started building the _chucknorris extension using _chucknorris.c and the code we passed in ffibuilder.set_source() and ffibuilder.cdef().
The ffibuilder.compile() method knew about our old friend -fPIC, and about the path to the Python includes (-I/usr/include/python3.6m), but it could not find the chucknorris.h header and the compilation failed.

Tweaking the ffibuilder #

Clearly the ffibuilder needs to know about the chucknorris library and the chucknorris include path.

We can pass them directly to the set_source() method using the extra_objects and include_dirs parameters. ¹

build_chucknorris.py:

import path

cpp_path = path.Path("../cpp/ChuckNorris").abspath()
cpp_build_path = cpp_path.joinpath("build/default")
ck_lib_path = cpp_build_path.joinpath("lib/libchucknorris.a")
ck_include_path = cpp_path.joinpath("include")

ffibuilder.set_source(
    "_chucknorris",
    """
    #include <chucknorris.h>

    """,
    extra_objects=[ck_lib_path],
    include_dirs=[ck_include_path],
)
...

Note that we use the wonderful path.py library to handle path manipulations, which we can add to our setup.py file:


from setuptools import setup

setup(name="chucknorris",
      version="0.1",
      ...
      setup_requires=["cffi", "path.py"],
      ...
)

The missing symbols #

Let’s try to build our extension again:

$ python setup.py build_ext
running build_ext
generating ./_chucknorris.c
...
building '_chucknorris' extension
gcc ... -fPIC ... -I/usr/include/python3.6m -c _chucknorris.c -o ./_chucknorris.o
gcc ... -shared  ... -o build/lib.linux-x86_64-3.6/_chucknorris.abi3.so

OK, this works.

Now let’s run python setup.py develop so that we can import the C extension directly:

$ python setup.py develop
...
generating cffi module 'build/temp.linux-x86_64-3.6/_chucknorris.c'
already up-to-date
...
copying build/lib.linux-x86_64-3.6/_chucknorris.abi3.so ->
...

Note that setup.py develop takes care of building the extension for us, and is even capable to skip compilation entirely when nothing needs to be rebuilt.

Now let’s run the chucknorris.py file:

$ python chucknorris.py
Traceback (most recent call last):
  File "chucknorris.py", line 1, in <module>
    from _chucknorris import lib, ffi
ImportError: .../_chucknorris.abi3.so: undefined symbol: _ZNSt8ios_base4InitD1Ev

Damned!

That’s the problem with shared libraries. gcc happily lets you build a shared library even if there are symbols that are not defined anywhere. It just assumes the missing symbols will be provided sometime before loading the library.

Thus, the only way to make sure a shared library has been properly built is to actually load it from an executable ².

Again we are faced with the task of guessing the library from the symbol name. Since it looks like a mangled C++ symbol, we can using c++filt to get a more human-readable name:

$ c++filt _ZNSt8ios_base4InitD1Ev
std::ios_base::Init::~Init

Here I happen to know this is a symbol that comes from the c++ runtime library, the library that contains things like the implementation of std::string.

We can solve the problem by passing the name of the c++ library directly as a libraries parameter:

ffibuilder.set_source(
    "_chucknorris",
    """
    #include <chucknorris.h>

    """,
    extra_objects=[ck_lib_path],
    include_dirs=[ck_include_path],
    libraries=["stdc++"],

Note: we could also have set the language parameter to c++, and invoke the C++ linker when linking _chucknorris.so, because the C++ linker knows where the c++ runtime library is. ³

Let’s try again:

$ python setup.py develop
$ python chucknorris.py
ImportError: .../_chucknorris.abi3.so: undefined symbol: sqlite3_close

This one is easier: chucknorris depends on libsqlite3, so we have to link with sqlite3 too.

In the CMakeLists.txt we wrote back in part 2, when we were building the cpp_demo executable, we just called target_link_libraries(cpp_demo chucknorris). CMake knew about the dependency from the chucknorris target to the sqlite3 library and everything worked fine.

But we’re not using the CMake / conan build system here, we are using the Python build system. How can we make them cooperate?

The json generator #

Since conan 1.2.0 there is a generator called json ⁴ we can use to get machine-readable information about dependencies.

Here’s how we can use this json file inside our ffibuilder.

First, let’s add json to the list of conan generators:

conanfile.txt:

[requires]
sqlite3/3.21.0@dmerej/test

...

[generators]
cmake
json

Then, let’s re-run conan install:

$ cd cpp/python/build/default
$ conan install ../..
...
PROJECT: Installing /home/dmerej/src/chucknorris/cpp/ChuckNorris/conanfile.txt
...
sqlite3/3.21.0@dmerej/test: Already installed!
PROJECT: Generator cmake created conanbuildinfo.cmake
PROJECT: Generator json created conanbuildinfo.json

This generates a conanbuildinfo.json file looking like this:

build/default/conanbuildinfo.json:

{
  "dependencies": [
    {
      "version": "3.21.0",
      "name": "sqlite3",
      "libs": [
        "sqlite3",
        "pthread",
        "dl"
      ],
      "include_paths": [
        "/.../.conan/data/sqlite3/.../<id>/include"
      ],
      "lib_paths": [
        "/.../.conan/data/sqlite3/.../<id>/lib"
      ],
    }
  ]
}

Now we can parse the json file and pass the libraries and include paths to the ffibuilder.set_source() function:

build_chucknorris.py:

cpp_path = path.Path("../cpp/ChuckNorris").abspath()
cpp_build_path = cpp_path.joinpath("build/default")

extra_objects = []
libchucknorris_path = cpp_build_path.joinpath("lib/libchucknorris.a")
extra_objects.append(libchucknorris_path)

include_dirs = []
include_dirs.append(cpp_path.joinpath("include"))

libraries = ["stdc++"]

conan_info = json.loads(cpp_build_path.joinpath("conanbuildinfo.json").text())
for dep in conan_info["dependencies"]:
    for lib_name in dep["libs"]:
        lib_filename = "lib%s.a" % lib_name
        for lib_path in dep["lib_paths"]:
            candidate = path.Path(lib_path).joinpath(lib_filename)
            if candidate.exists():
                extra_objects.append(candidate)
            else:
                libraries.append(lib_name)
    for include_path in dep["include_paths"]:
        include_dirs.append(include_path)


ffibuilder.set_source(
    "_chucknorris",
    """
    #include <chucknorris.h>

    """,
    extra_objects=extra_objects,
    include_dirs=include_dirs,
    libraries=libraries,
)

And now everything works as expected:

$ python3 setup.py clean develop
$ python chucknorris.py
There are no weapons of mass destruction in Iraq, Chuck Norris lives in Oklahoma.

We can even build a pre-compiled wheel that other people can use it without need to compile the chucknorris project themselves:

On the developer machine:

$ python setup.py bdist_wheel
...
running build_ext
...
building '_chucknorris' extension
...
creating 'dist/chucknorris-0.1-cp36-cp36m-linux_x86_64.whl' and adding '.' to it
...

On an other machine:

$ pip install chucknorris-0.1-cp36-cp36m-linux_x86_64.whl
$ python -c 'import chucknorris; chucknorris.main()'

For this to work, the other user will need to be on Linux, have a compatible C++ library and the same version of Python, but as far as distribution of binaries on Linux usually go, isn’t this nice?

There’s an entire blog post to be written about distribution of pre-compiled Python binary modules, but enough about Python for now :)

See you next time, where we’ll use everything we learned there and start porting Chuck Norris to Android.

ffibuilder.set_source() uses the same API as the distutils Extension class. ↩︎
This also means you should really have at least one executable to test every shared library you write, but you already knew that, right? ↩︎
Not sure what the best move is here. If you have an opinion on it, please let me know. PS: I know we could also use static linking, but I’m saving that for the part where we build ChuckNorris on Android. ↩︎
Disclaimer: the json generator feature was added by yours truly. ↩︎

Let's Build Chuck Norris! - Part 4: Python and ctypes

Sat, 31 Mar 2018 12:44:43 +0000

Note: This is part 4 of the Let’s Build Chuck Norris! series.

Static and shared libraries #

C and C++ libraries come in two forms: static and shared.

In both cases, a library is collection of names (the symbols) and executable code. The difference between static and shared libraries is how those are used, for instance by an executable.

The code from a static library is directly integrated into the program: the compiler will take copies of the code the program uses from the static library and make it part of the program.

On the other hand, the code in a shared library is only referenced by the program. When the program is launched, the operating system will try and find the code to run in the shared library file.

We says the code in the static library is used at compile time whereas the code of a shared library is used at runtime.

For this reason, static libraries are also called archive libraries, and shared library are also called dynamic libraries.

Static and shared libraries usually have different extensions depending on the platform:

platform	static	shared
Linux	.a	.so
macOS	.a	.dylib
Windows	.lib	.dll ¹

So, which type of library do we need to write bindings in Python ?

From now on, everything takes place on a Linux machine. There are some differences when using macOS or Windows, but for the sake of simplicity I will only deal with one platform here. We will also use Python3 only. Again, in Python2 things are a little different, but let’s keep things simple.

Since we are using Python as a program (the /usr/bin/python3 binary), it’s obviously too late to do anything at compile time. So let’s build a shared library and see what we can do at runtime, shall we?

Building the shared library #

When we described the chucknorris library in our CMakeList.txt earlier, we did not specify its type.

The type of the library used in this case is controlled by a variable called BUILD_SHARED_LIBS which is OFF by default. So, let’s re-run CMake, setting this variable to ON instead.

$ cd build/default
$ cmake -GNinja -DBUILD_SHARED_LIBS=ON ../..
$ ninja
...
-- Library sqlite3 found /.conan/data/sqlite3/3.21.0/bincrafters/...libsqlite3.a
...
[1/7] Building C object CMakeFiles/c_demo.dir/src/main.c.o
[2/7] Building CXX object CMakeFiles/chucknorris.dir/src/c_wrapper.cpp.o
[3/7] Building CXX object CMakeFiles/cpp_demo.dir/src/main.cpp.o
[4/7] Building CXX object CMakeFiles/chucknorris.dir/src/ChuckNorris.cpp.o
[5/7] Linking CXX shared library lib/libchucknorris.so
FAILED: lib/libchucknorris.so
: && /bin/c++ ...
  -o lib/libchucknorris.so
  ChuckNorris.cpp.o
  c_wrapper.cpp.o
  ...
  libsqlite3.a
  ..
/bin/ld: libsqlite3.a(sqlite3.o): relocation R_X86_64_PC32
against symbol `sqlite3_version` can not be used when making a shared object;
recompile with -fPIC

The link fails.

What’s going on here is that we are trying to incorporate a static library (libsqlite3.a) inside a shared one. Most of the time this works fine, but not on Linux.

The compiler tells us what needs to be done: we have to recompile libsqlite3.a with -fPIC.

Here’s what man gcc has to say about fpic:

-fpic
  Generate position-independent code (PIC) suitable for use in a shared library.

Fair enough, let’s try to rebuild sqlite3 by generating position-independent code.

Patching a conan recipe #

The first step is to see if we can rebuild sqlite3 ourselves.

We will be using a different user name and channel. (@dmerej/test instead of @bincrafters/stable). As we explained earlier, conan is decentralized, so copying and modifying other people’s recipes in order to satisfy your requirements is possible and even encouraged.

Let’s fetch the recipe from the remote:

$ conan copy sqlite3/3.21.0@bincrafters/stable dmerej/test
Downloading conan_sources.tgz
[==================================================] 706B/706B
Copied sqlite3/3.21.0@bincrafters/stable to sqlite3/3.21.0@dmerej/test
Copied sources sqlite3/3.21.0@bincrafters/stable to sqlite3/3.21.0@dmerej/test

Here conan looked for the recipe in the remote and created a copy with a different name, but still inside the conan cache.

Then we copy the sources from the cache and put them in a conan/sqlite3 folder next to the C++ code:

$ cd ChuckNorris/cpp
$ mkdir -p conan/sqlite3
$ cd conan/sqlite3
$ cp -rv ~/.conan/data/sqlite3/3.21.0/bincrafters/stable/export/* .
'../conanfile.py' -> './conanfile.py'
'../conanmanifest.txt' -> './conanmanifest.txt'
'../LICENSE.md' -> './LICENSE.md'

Let’s try to build the package ourselves:

$ conan create . dmerej/test
sqlite3/3.21.0@dmerej/test: Exporting package recipe
sqlite3/3.21.0@dmerej/test: A new conanfile.py version was exported
...
sqlite3/3.21.0@dmerej/test: Installing package
..
sqlite3/3.21.0@dmerej/test: Attempting download of sources from:
  https://www.sqlite.org/2017/sqlite-amalgamation-3210000.zip
...
sqlite3/3.21.0@dmerej/test: Calling build()
CMake Error: The source directory "..." does not appear to contain CMakeLists.txt.

Turns out we need to also copy some files from the export_source folder:

$ cp -rv ~/.conan/data/sqlite3/3.21.0/bincrafters/stable/export_source/* .
'../CMakeLists.txt -> './CMakeLists.txt'
'../FindSQLite3.cmake' -> './FindSQLite3.cmake'

And now we can build:

$ conan create . dmerej/test
sqlite3/3.21.0@dmerej/test: Exporting package recipe
sqlite3/3.21.0@dmerej/test: A new conanfile.py version was exported
...
sqlite3/3.21.0@dmerej/test: Installing package
..
sqlite3/3.21.0@dmerej/test: Attempting download of sources from:
  https://www.sqlite.org/2017/sqlite-amalgamation-3210000.zip
...
sqlite3/3.21.0@dmerej/test: Calling build()
...
[1/2] Building C object CMakeFiles/sqlite3.dir/sources/sqlite3.o
[2/2] Linking C static library lib/libsqlite3.a
...
sqlite3/3.21.0@dmerej/test: Package '6ae331b7...' built
...
sqlite3/3.21.0@dmerej/test: Calling package()
sqlite3/3.21.0@dmerej/test package(): Copied 1 '.cmake' files: FindSQLite3.cmake
sqlite3/3.21.0@dmerej/test package(): Copied 2 '.h' files: sqlite3.h, sqlite3ext.h
sqlite3/3.21.0@dmerej/test package(): Copied 1 '.a' files: libsqlite3.a
...
sqlite3/3.21.0@dmerej/test: Package '6ae331b7...' created

Let’s sum up what happened:

Conan fetched the sources from sqlite.org
It called CMake using the CMakeLists.txt we copied from export_source
It built the CMake project using a function named build()
It copied some files using a function named package().

This is roughly what we did earlier when we built sqlite by hand.

Let’s take a closer look at the conan source files:

CMakeList.txt:

project(cmake_wrapper)

include(conanbuildinfo.cmake)
conan_basic_setup()

add_library(sqlite3 sources/sqlite3.c)

conanfile.py:


class ConanSqlite3(ConanFile):
    name = "sqlite3"
    version = "3.21.0"
    settings = "os", "compiler", "arch", "build_type"
    options = {"shared": [True, False]}
    ...

    def source(self):
        base_url = "https://www.sqlite.org/" + self.year
        ...
        download_url = "{0}/{1}.{2}".format(base_url, archive_name, archive_ext)
        tools.get(download_url)

    def build(self):
        cmake = CMake(self)
        ...
        cmake.configure()
        cmake.build()

    def package(self):
        self.copy(...)

The conan recipe is just a Python class that derives from the ConanFile class.

It contains the source(), build() and package() methods used to fetch the sources, build the package, and copy the relevant files we saw mentioned in the previous console output.

It uses a CMake class that knows how to run cmake.

Finally, the class contains a few attributes. name and version are self-explanatory, but we have to talk about the settings and options.

In both cases, settings and options are variables the consumers of the package can set. The settings cannot have default values, and if a setting changes, a different package must be produced. This is why the compiler (think gcc versus Visual Studio) is a setting. Settings are set globally, usually inside a profile, and apply to all the recipes. Options are different: they can have default values, they can be set package per package, and they have a pre-defined list of possible values.

We can see the conanfile.py already defines a shared option that can be true or false. We do not really want a shared sqlite3 library, we want a static sqlite3 library but built with position independent code.

There are several ways to do this. One of them is to introduce a new option called pic.

CMake knows how to convert the abstract concept of “position independent code” into concrete compiler flags such as -fPIC for gcc, so we just have to set the correct CMake variable:


class ConanSqlite3(ConanFile):

    settings = "os", "compiler", "arch", "build_type"
    options = {
      "shared": [True, False],
      "pic": [True, False],
    }


    def build(self):
        cmake = CMake(self)
        if self.options.shared:
            cmake.definitions["BUILD_SHARED_LIBS"] = "ON"
        if self.options.pic:
            cmake.definitions["CMAKE_POSITION_INDEPENDENT_CODE"] = "ON"
        ...

So now we can re-create the sqlite3 package:

$ conan create --option 'pic=True' . dmerej/test

Finally, we can change the conanfile.txt in cpp/ChuckNorris to reference our newly built package:

conanfile.txt:

[requires]
sqlite3/3.21.0@dmerej/test

Using the chucknorris shared library #

Let’s re-run conan install, using the --option command line flag again, and see if we can manage to build chucknorris the way we want.

Note that we prefix the pic=True option by the name of the package we want to apply the option on. If we did not do that, conan would have tried to set the option on every package.

$ cd build/default
$ conan install ../.. --option 'sqlite3:pic=True'
$ cmake -GNinja -DBUILD_SHARED_LIBS=ON ../..
$ ninja
[1/7] Building C object CMakeFiles/c_demo.dir/src/main.c.o
[2/7] Building CXX object CMakeFiles/chucknorris.dir/src/c_wrapper.cpp.o
[3/7] Building CXX object CMakeFiles/cpp_demo.dir/src/main.cpp.o
[4/7] Building CXX object CMakeFiles/chucknorris.dir/src/ChuckNorris.cpp.o
[5/7] Linking CXX shared library lib/libchucknorris.so
[6/7] Linking C executable bin/c_demo
[7/7] Linking CXX executable bin/cpp_demo

Success!

Side note: specifying the option about sqlite3 each time we call conan is a bit tedious, but we can just specify the option directly in the conanfile.txt:

[requires]
sqlite3/3.21.0@dmerej/test

[options]
sqlite3:pic = True

Anyway, we said earlier that it was the operating system that took care of loading code from the shared library at runtime. On Linux, this is done by a special shared library called ld-linux.so.

We can thus check that the libchucknorris.so file does get loaded when we run the cpp_demo executable, by asking ld.so to output debug information about the files it loads ²:

$ cd build/default
$ export LD_TRACE_LOADED_OBJECTS=1
$ ./bin/cpp_demo
./bin/cpp_demo
	linux-vdso.so.1 (0x00007f...)
	libchucknorris.so => .../build/default/lib/libchucknorris.so (0x00007f...)
    ...
	libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f...)
	libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f...)
    ...
Chuck Norris knows Victoria's secret

Our friends libpthread.so and libdl.so we had to take care of when we linked with sqlite3 by hand are involved, and we can see the full path of the ChuckNorris lib inside our build folder.

Using ctypes #

The Python standard library contains a module called ctypes that allows to do what ld.so does, but using Python code.

The documentation says we can use ctypes.cdll.LoadLibrary to get a “handle” from the .so, and then use the symbols in the shared library simply by calling methods with the right names on the handle.

Let’s try:

handle = ctypes.cdll.LoadLibrary("build/default/lib/libchucknorris.so")
ck = handle.chuck_norris_init()
fact = handle.chuck_norris_get_fact(ck)
print(fact)

$ python ck.py
zsh: segmentation fault (core dumped)  python ck.py

Whoops :/

Actually for this to work we have to specify the types of the parameters and return values for every method we call, like this:

handle = ctypes.cdll.LoadLibrary("build/default/lib/libchucknorris.so")
handle.chuck_norris_init.restype = ctypes.c_void_p
handle.chuck_norris_get_fact.restype = ctypes.c_char_p
handle.chuck_norris_get_fact.argtypes = [ctypes.c_void_p]
ck = handle.chuck_norris_init()
fact = handle.chuck_norris_get_fact(ck)
print(fact)

$ python ck.py
b'When Chuck Norris enters a rodeo the bull has to try and last 8 seconds.'

Almost there: we still have get rid of the b' prefix.

ctypes can’t really assume a char * in C code contains text, so the c_char_p type has been translated to a bytes object, suitable for representing binary data.

Assuming we were careful and only inserted valid UTF-8 encoded text in our sqlite3 database, we can call .decode(UTF-8) in our Python code, though:

fact_as_bytes = handle.chuck_norris_get_fact(ck)
fact_text = fact_as_bytes.decode("UTF-8")
print(fact_text)

$ python ck.py
When Chuck Norris enters a rodeo the bull has to try and last 8 seconds.

And we’re done.

This is all well and good, but there’s a more robust method to write our Python bindings. You can read more about it in part 5.

Actually, on Windows when you want to link with a shared library, you use both a .lib when linking and a .dll at runtime. It’s confusing, I know. ↩︎
This is the technique used by ldd, by the way. ↩︎

Let's Build Chuck Norris! - Part 3: A C wrapper

Sun, 25 Mar 2018 16:50:30 +0000

Note: This is part 3 of the Let’s Build Chuck Norris! series.

Introduction: when languages talk together #

C is kind like the lingua franca of programming languages. Many languages implementations are themselves written in C, and almost all of them know how to call C code. Often this is called using a Foreign Function Interface, or FFI for short.

Our goal with the Chuck Norris project is to use our library in a lot of various situations (such as in an iOS or Android application), so why did we not write the chucknorris library in C?

Well, C++ has many advantages compared to C:

Strings are easier to handle
Memory management is simpler
You can use nice tools such as classes and templates
… and more!

C and C++ are not so far apart: For instance, calling C from C++ works out of he box. sqlite3 is written in C, and in our code we just had to include <sqlite3.h> and everything “just worked”. ¹

But things get more interesting when we try to go the other way around.

Calling C++ code from C #

In our library we expose a class, but C does not know about classes. So we are going to declare a C API, and then implement the C API using C++ code.

We can do this because C++ is a “superset” of C.

There’s a few details to get right though, so let’s do this step by step.

Declaring a C API #

C does not know about classes, so we cannot use the ChuckNorris symbol anywhere.

Here’s what we can do:

include/chucknorris.h:

typedef struct chuck_norris chuck_norris_t;
chuck_norris_t* chuck_norris_init(void);
const char* chuck_norris_get_fact(chuck_norris_t*);
void chuck_norris_deinit(chuck_norris_t*);

Each function is prefixed with chuck_norris_ (because there are no namespaces in C)
We declare a chuck_norris_t struct type but do not bother to describe what’s inside. This works because the other functions will either return or take a parameter of the chuck_norris_t pointer type, so the compiler does not need to know what’s inside the struct. This is known as an opaque pointer. In the C++ implementation, we’ll have to perform casts between the “real” ChuckNorris* pointers and the opaque chuck_norris* ones.
Instead of letting the compiler handle creation and destruction of the C++ class, we have explicit functions: chuck_norris_init() and chuck_norris_deinit().
Instead of a getFact() method inside a class, we have a chuck_norris_get_fact() function that takes opaque chuck_norris pointer as first parameter.

Implementing the C API #

Here’s what our first attempt looks like:

src/c_wrapper.cpp:

#include <cstring>
#include <chucknorris.h>

#include <ChuckNorris.hpp>

chuck_norris_t* chuck_norris_init()
{
  auto ck = new ChuckNorris();
  return reinterpret_cast<chuck_norris*>(ck);
}

const char* chuck_norris_get_fact(chuck_norris_t* chuck_norris)
{
  auto ck = reinterpret_cast<ChuckNorris*>(chuck_norris);
  std::string fact = ck->getFact();
  const char* result = fact.c_str();
  return result;
}

void chuck_norris_deinit(chuck_norris_t* chuck_norris)
{
  auto ck = reinterpret_cast<ChuckNorris*>(chuck_norris);
  delete ck;
}

The only cast we can use is reinterpret_cast, which basically tell the compiler “trust us, what’s inside the pointer is of the right type!”. This means things will go terribly wrong if callers of the C API are not careful, but don’t worry, they’re used to it :P

To check it works, let’s add an other test executable, written in C this time:

src/main.c:

#include <chucknorris.h>
#include <stdlib.h>
#include <stdio.h>

int main()
{
  chuck_norris_t* ck = chuck_norris_init();
  const char* fact = chuck_norris_get_fact(ck);
  printf("%s\n", fact);
  chuck_norris_deinit(ck);
  return 0;
}

Now let’s adapt the CMake code to:

Add the c_wrapper.cpp file to the list of the sources of the chucknorris library.
Add a c_demo executable built with from the main.c file:

  add_library(chucknorris
    include/ChuckNorris.hpp
    include/chucknorris.h
    src/ChuckNorris.cpp
+   src/c_wrapper.cpp
  )

+ add_executable(c_demo
+   src/main.c
+ )
+
+ target_link_libraries(c_demo chucknorris)

And let’s try to compile:

$ cd build/default
$ ninja
[1/7] Building C object CMakeFiles/c_demo.dir/src/main.c.o
[2/7] Building CXX object CMakeFiles/chucknorris.dir/src/c_wrapper.cpp.o
[3/7] Building CXX object CMakeFiles/cpp_demo.dir/src/main.cpp.o
[4/7] Building CXX object CMakeFiles/chucknorris.dir/src/ChuckNorris.cpp.o
[5/7] Linking CXX static library lib/libchucknorris.a
[6/7] Linking CXX executable bin/c_demo
FAILED: bin/c_demo
: && /bin/c++ main.c.o -o bin/c_demo lib/libchucknorris.a ...
CMakeFiles/c_demo.dir/src/main.c.o: In function `main':
main.c:(.text+0x9): undefined reference to `chuck_norris_init'
main.c:(.text+0x19): undefined reference to `chuck_norris_get_fact'
main.c:(.text+0x35): undefined reference to `chuck_norris_deinit'

We can see the libchucknorris.a library was passed to the linker, so why were the symbols not found ?

Mangled symbols #

To understand, let’s look at the names of the symbols inside the libchucknorris.a library using a tool called nm:

$ nm --defined-only libchucknorris.a

ChuckNorris.cpp.o:
0000000000000000 V DW.ref.__gxx_personality_v0
...
000000000000020c T _ZN11ChuckNorris7getFactB5cxx11Ev

c_wrapper.o
...
0000000000000000 T _Z17chuck_norris_initv
00000000000000ac T _Z19chuck_norris_deinitP11ChuckNorris
000000000000003c T _Z21chuck_norris_get_factP11ChuckNorris

Hum. The names of the symbols do not match the ones we declared in the headers.

That’s because they were mangled by the C++ compiler. I won’t detail here the reasons why the symbols have to be mangled in the first place. Let’s just say it has to do with stuff like function overloading and things like that.

We can check that the cpp_demo binary contains a reference to the weird getFact symbol:

$ nm --defined-only cpp_demo
...
000000000000cefe T _ZN11ChuckNorris7getFactB5cxx11Ev

We can also use the --demangle option when calling nm and see the original names:

(note that the C symbols were mangled too)

$ nm --demangle --defined-only libchucknorris.a
...
00000000000001b0 T ChuckNorris::getFact[abi:cxx11]()
...
0000000000000000 T chuck_norris_init()
00000000000000ac T chuck_norris_deinit(ChuckNorris*)
000000000000003c T chuck_norris_get_fact(ChuckNorris*)

$ nm --demangle --defined-only cpp_demo
...
000000000000cefe T ChuckNorris::getFact[abi:cxx11]()

When we compiled c_demo.o, we used a C compiler. (CMake saw a .c extension on the source file, and thus told ninja to build main.c.o with a C compiler)

Since the C compiler does not mangle symbols at all, the final link between c_demo.o and libchucknorris.a failed.

The solution is to tell the C++ compiler to not mangle the symbols defined in the chucknorris.h header using the extern syntax


extern "C" {

  typedef struct chuck_norris chuck_norris_t;
  chuck_norris_t* chuck_norris_init(void);
  char* chuck_norris_get_fact(chuck_norris_t*);
  void chuck_norris_deinit(chuck_norris_t*);

}

But if we do that, we now get a compile failure because the C compiler does not understand the extern syntax:

$ ninja
/bin/cc  -o main.c.o -c main.c
In file included from ../../src/main.c:1:0:
chucknorris.h:3:8: error: expected identifier or ‘(’ before string constant
extern "C" {

Fortunately, the C++ compiler sets a __cplusplus define for us:


#ifdef __cplusplus
extern "C" {
#endif

  chuck_norris_t* chuck_norris_init(void);
  const char* chuck_norris_get_fact(chuck_norris_t*);
  void chuck_norris_deinit(chuck_norris_t*);

#ifdef __cplusplus
}
#endif

Now the build passes ², and we can double-check the names of symbols inside the archive:

$ ninja
[1/7] Building C object CMakeFiles/c_demo.dir/src/main.c.o
[2/7] Building CXX object CMakeFiles/chucknorris.dir/src/c_wrapper.cpp.o
[3/7] Building CXX object CMakeFiles/cpp_demo.dir/src/main.cpp.o
[4/7] Building CXX object CMakeFiles/chucknorris.dir/src/ChuckNorris.cpp.o
[5/7] Linking CXX static library lib/libchucknorris.a
[6/7] Linking CXX executable bin/cpp_demo
[7/7] Linking CXX executable bin/c_demo

$ nm --defined-only libchucknorris.a
ChuckNorris.cpp.o:
0000000000000000 V DW.ref.__gxx_personality_v0
...
0000000000000160 T _ZN11ChuckNorrisC1Ev
...
000000000000020c T _ZN11ChuckNorris7getFactB5cxx11Ev

c_wrapper.o
...
0000000000000000 T chuck_norris_init
00000000000000ac T chuck_norris_deinit
000000000000003c T chuck_norris_get_fact

The string bug #

Hooray, we managed to build our C code! Let’s run it:

$ ./bin/c_demo
���rU
 ./bin/c_demo
����BV
./bin/c_demo
`15R�U

Hum. Something is not right.

Let’s take a look again at the chuck_norris_get_fact implementation:

const char* chuck_norris_get_fact(chuck_norris_t* chuck_norris)
{
  auto ck = reinterpret_cast<ChuckNorris*>(chuck_norris);
  std::string fact = ck->getFact();
  const char* result = fact.c_str();
  return result;
}

Here’s what’s happening:

First we call chuck_norris->getFact() and create a local variable named fact
Then we get a char* pointer to the contents of the std::string with c_str()
We return the char* pointer
But then the fact variable gets out of scope and the contents of the std::string are freed. Note that this is what std::strings are designed to do: they handle memory management for us. We are only having this problem because we are playing with raw C pointers!
Now our char* pointer points to uninitialized stuff and we get garbage.

The solution is to call strdup to get a copy of the contents that we now own and to free it explicitly later on.

(Note that the returned pointer is no longer const.)

src/c_wrapper.cpp:

char* chuck_norris_get_fact(chuck_norris_t* chuck_norris)
{
  auto ck = reinterpret_cast<ChuckNorris*>(chuck_norris);
  std::string fact = ck->getFact();
  char* result = strdup(fact.c_str());
  return result;
}

src/main.c:

int main()
{
  chuck_norris_t* ck = chuck_norris_init();
  char* fact = chuck_norris_get_fact(ck);
  printf("%s\n", fact);
  free(fact);
  chuck_norris_deinit(ck);
  return 0;
}

And now the binary works:

$ ninja
$ ./bin/c_demo
Chuck Norris counted to infinity. Twice.;

That’s all for today. The C library is the building block we’ll use to write Python bindings and phone applications. Stay tuned for the rest of the story!

That’s a lie. There’s something special in the sqlite3.h file to make this work. But we’ll talk about that later. ↩︎
This little trick of #ifdef __cplusplus and extern "C" is used pretty often in the wild, and you do find it in the sqlite3.h header. I had to lie to preserve the flow of the article, sorry. ↩︎

Let's Build Chuck Norris! - Part 2: SQLite and conan

Sun, 18 Mar 2018 14:29:21 +0000

Note: This is part 2 of the Let’s Build Chuck Norris! series.

In the previous episode we wrote a simple C++ library that returned a hard-coded Chuck Norris fact.

It’s now time to put several Chuck Norris facts in a database, and select one at random when asked.

We are going to use sqlite for that.

First, let’s adapt our class declaration:

include/ChuckNorris.hpp:

#include <string>

#include <sqlite3.h>

class ChuckNorris
{
  public:
    ChuckNorris();
    // Make sure you cannot copy Chuck Norris
    ChuckNorris(ChuckNorris const&) = delete;
    ChuckNorris(ChuckNorris &&) = delete;
    ChuckNorris& operator=(ChuckNorris const&) = delete;
    ChuckNorris& operator=(ChuckNorris &&) = delete;
    ~ChuckNorris();

    std::string getFact();

  private:
    sqlite3* _db;
};

Then, it’s time to adapt the ChuckNorris.cpp file:

In the constructor, we open an in-memory database and fill it with lots of facts.

Then, in the getFact() method, we run a simple SQL query:

src/ChuckNorris.cpp:

// Note: Error handling omitted for brevity.

#include <sqlite3.h>
#include <ChuckNorris.hpp>

ChuckNorris::ChuckNorris()
{
  sqlite3_open(":memory:", &_db);

  auto const sql = R"(
CREATE TABLE chucknorris(id PRIMARY_KEY, fact VARCHAR(500));
INSERT INTO chucknorris (fact) VALUES
  ("Chuck Norris can slam a revolving door.");
INSERT INTO chucknorris (fact) VALUES
  ("Chuck Norris can kill two stones with one bird.");
  ...
  )";

  sqlite3_exec(_db, sql, 0, 0, nullptr);
}

ChuckNorris::~ChuckNorris()
{
    sqlite3_close(_db);
}

std::string ChuckNorris::getFact()
{
  // Note: error handling omitted for brevity

  sqlite3_stmt* statement;
  int rc;
  rc = sqlite3_prepare_v2(_db,
      R"(SELECT fact FROM chucknorris ORDER BY RANDOM() LIMIT 1;)",
      -1, &statement, 0);
  if (rc != SQLITE_OK) {
    // ...
  }
  rc = sqlite3_step(statement);
  if (rc != SQLITE_ROW) {
    // ...
  }
  auto sqlite_row = sqlite3_column_text(statement, 0);
  auto row = reinterpret_cast<const char*>(sqlite_row);
  auto res = std::string(row);
  sqlite3_finalize(statement);
  return res;
}

Now let’s try and compile this!

Third party libraries #

Since we are not the authors of the sqlite library, we say it is a third-party library. (As opposed to the chucknorris library we just wrote).

There are a lot of ways of adding a third-party dependency to a C++ program, from simply adding the sources to the project files, to installing them “in the system” (using homebrew on macOS or the package manager of your distribution on Linux).

In this article we will use a package manager called conan that will install the sqlite3 package somewhere in our home directory (called a cache).

This means we won’t need any administrative privileges (as opposed to installing the sqlite3 library in the system), but also that the library will usable from multiple C++ projects (as opposed to adding the sources of sqlite3 inside our source folder).

Using conan will come in handy when we start cross-compiling things later, but I’m getting ahead of myself.

Adding the dependency by hand #

Let’s start by doing the work “by hand” before we talk about how conan works.

Building sqlite3 #

First, we go to the download page of the sqlite3 project.

We then fetch the “almagation” archive, and extract it:

$ cd $HOME
$ mkdir -p 3rdpart/sqlite
$ cd 3rdpart/sqlite
$ wget https://www.sqlite.org/2018/sqlite-amalgamation-3220000.zip
$ unzip sqlite-amalgamation-3220000.zip
$ cd sqlite-amalgamation-3220000
$ ls
$ shell.c  sqlite3.c  sqlite3.h  sqlite3ext.h

OK, we just have a sqlite3.c to build and a sqlite3.h header that we can include.

Let’s build a static library:

$ gcc -c sqlite3.c -o sqlite3.o
$ ar qf libsqlite3.a sqlite3.o
$ ranlib libsqlite3.a

And, just to clean things up, let’s create a lib and include folder:

$ mkdir include lib
$ mv libsqlite3.a lib/
$ mv sqlite3.h include/

Now we have to adapt the CMakeLists.txt in the Chuck Norris sources to use our newly built library.

Finding and using the sqlite3 library #

In CMake parlance, sqlite3 is no longer a “regular” target, since it does not know how it was built.

So we need to create an imported target, and set the IMPORTED_LOCATION and INTERFACE_INCLUDE_DIRECTORIES properties on it:

project(ChuckNorris)

# ...

add_library(sqlite3 STATIC IMPORTED)
set_target_properties(sqlite3
  PROPERTIES
  IMPORTED_LOCATION /path/to/sqlite3-<version>/lib/libsqlite3.a
  INTERFACE_INCLUDE_DIRECTORIES /path/to/sqlite3-<version>/include
)

add_library(chucknorris
  # ...
)

Note that /path/to/sqlite3 is an hard-coded absolute path to the folder we used to compile sqlite3 by hand.

Obviously, hard-coded paths are bad, but this allows us to use target_link_libraries to add a dependency between our ChuckNorris library and the imported target, the same way we did to link cpp_demo with ChuckNorris:

target_link_libraries(chucknorris sqlite3)

Let’s build the ChuckNorris project again:

$ cd build/default
$ cmake -GNinja ../..
$ ninja -v
Build CXX object ChuckNorris.cpp.o
/bin/c++
  ../../src/ChuckNorris.cpp
  -I../../include
  -isystem /path/to/sqlite-<version>/include
  -o ChuckNorris.cpp.o
  ...
Linking CXX executable cpp_demo
/bin/c++
  main.cpp.o
  -o cpp_demo
  libchucknorris.a
  /path/to/sqlite-<version>/lib/sqlite3.a

You can see that CMake added the include path of sqlite3 in a -isystem flag¹ when compiling ChuckNorris.cpp, and that it added the libsqlite3.a file when linking the cpp_demo executable.

On my machine, I still got a build failure:

Linking CXX executable cpp_demo
FAILED: cpp_demo
/bin/c++
  main.cpp.o
  -o cpp_demo
  libchucknorris.a
  /path/to/sqlite-<version>/libsqlite3.a

libsqlite3.a(sqlite3.o): In function `pthreadMutexAlloc':
sqlite3.c:(.text+0x4275): undefined reference to `pthread_mutexattr_init'
...
libsqlite3.a(sqlite3.o): In function `unixDlOpen':
sqlite3.c:(.text+0x10bd8): undefined reference to `dlopen'

What’s going on?

Turns out that, on Linux at least, sqlite3 depends on two other libraries, namely pthread and dl. (This is common enough that I was able to guess the name of the libraries from the names of the missing symbols in the error message)

We can fix our compilation failure by telling CMake about the dependency from sqlite3 to pthread and dl:

if(CMAKE_SYSTEM_NAME STREQUAL "Linux")
  set_target_properties(sqlite3
    PROPERTIES
      INTERFACE_LINK_LIBRARIES "dl;pthread"
   )
endif()

$ ninja -v
/bin/c++
  main.cpp.o
  -o cpp_demo
  libchucknorris.a
  /path/to/sqlite-<version>/lib/libsqlite3.a
  -ldl
  -lpthread

That was not trivial, and we had to hard-code the location of the sqlite3 sources.

Let’s now see how conan can help us using sqlite3 more easily.

Installing conan #

Conan is written in Python and you can install it with pip:

$ python3 -m pip install conan --user

Afterwards, you should add the directory where the conan binary has been installed in your PATH

Linux: export PATH="${HOME}/.local/bin:${PATH}"
macOS: export PATH="${HOME}/Library/Python3.x/bin:${PATH}"
Windows: nothing to do ;-)

Conan lets you write recipes and upload binary packages.

A recipe is a piece of Python code that is used to build a package; and a package is simply an archive containing pre-compiled binaries.

In the conan world, any user can publish recipes and binary packages for any library. (Many package managers use a “first come, first served” approach to package names, but not conan).

As many other package managers, conan can look for packages in multiple locations called remotes: all you need is to host a conan server somewhere.

By default, conan comes with the conan-center remote pre-configured.

Creating a conanfile.txt #

Conan uses a simple configuration file format where you can specify the dependencies and some “generators”:

[requires]
sqlite3/3.21.0@bincrafters/stable

[generators]
cmake

The string sqlite3/3.21@bincrafters/stable is called a reference.

There’s the name: sqlite3, the version: 3.21, a user: bincrafters and a channel: stable.

You can think of channels as branches: the stable channel indicates that the recipes and binary packages have come through at least minimal quality assurance. (This is true for any package coming from the conan-center remote inside the stable channel).

The cmake generator tells conan to generate some files that contain information about the dependencies that are usable by CMake.

Here’s how to invoke conan:

$ cd build/default
$ conan install ../..
sqlite3/3.21.0@bincrafters/stable: Not found in local cache, looking in remotes...
sqlite3/3.21.0@bincrafters/stable: Trying with 'conan-center'...
...
PROJECT: Installing /path/to/conanfile.txt
Requirements
    sqlite3/3.21.0@bincrafters/stable from 'conan-center'
Packages
    sqlite3/3.21.0@bincrafters/stable:6ae331b72e7e265ca2a3d1d8246faf73aa030238
PROJECT: Retrieving package 6ae331b72e7e265ca2a3d1d8246faf73aa030238
...
PROJECT: Generator cmake created conanbuildinfo.cmake

Several things happened here:

First conan looked for the package in the local cache and did not find it.
It used a remote called conan-center and found a compatible package there with a certain ID (6ae331b72e7e265ca2a3d1d8246faf73aa030238).
Then it downloaded the binary package from the remote and stored it in the cache.
Finally, it generated a file called conanbuildinfo.cmake in the build folder.

Adapt CMakeLists.txt #

We can then revert our changes in the CMakeLists.txt file and instead include the file conan generated for us, using the include() CMake function and the predefined CMake variable CMAKE_BINARY_DIR that points to the build folder:

include(${CMAKE_BINARY_DIR}/conanbuildinfo.cmake)

Then we invoke the conan_basic_setup() method and refer to the targets defined by conan using the CONAN_PKG:: prefix:

project(ChuckNorris)
set(CMAKE_CXX_STANDARD 11)

include(${CMAKE_BINARY_DIR}/conanbuildinfo.cmake)
conan_basic_setup(TARGETS)

# ...

target_link_libraries(chucknorris CONAN_PKG::sqlite3)

We can now call cmake and ninja as we did in the end of the previous part:

$ cmake -GNinja ../..
...
-- Conan: Using cmake targets configuration
-- Library sqlite3 found /home/dmerej/.conan/data/.../lib/libsqlite3.a
...
$ ninja
[1/3] Building CXX object CMakeFiles/chucknorris.dir/src/ChuckNorris.cpp.o
[2/3] Linking CXX static library lib/libchucknorris.a
[3/3] Linking CXX executable bin/cpp_demo
$ ./bin/cpp_demo
The easiest way to determine Chuck Norris's age is to cut him
in half and count the rings.
$ ./bin/cpp_demo
If, by some incredible space-time paradox, Chuck Norris would ever fight
himself, he'd win. Period.

Seems to work!

In the next articles, we’ll write a C wrapper on top of our C++ API and then use it to implement a chucknorris module in Python.

See you soon :)

The -isystem flag works almost like the -I flag. You can find all about the gory details in the gcc man page. ↩︎

Let's Build Chuck Norris! - Part 1: CMake and Ninja

Sat, 10 Mar 2018 14:22:53 +0100

Note: This is part 1 of the Let’s Build Chuck Norris! series.

The core library #

We want a ChuckNorris class with a getFact() method that returns a random Chuck Norris fact.

Let’s start with a hard-coded answer for now:

include/ChuckNorris.hpp:

#pragma once
#include <string>

class ChuckNorris {
  public:
    ChuckNorris();
    std::string getFact();
};

src/ChuckNorris.cpp:

#include <ChuckNorris.hpp>

ChuckNorris::ChuckNorris()
{
}

std::string ChuckNorris::getFact()
{
  return "Chuck Norris can slam a revolving door.";
}

We have a header in include/ChuckNorris.hpp containing the class declaration, and a file in src/ChuckNorris.cpp containing the class definition.

The test program #

To make sure the library can indeed be used in other programs, let’s add a src/main.cpp to check we manage to get the hard-coded fact:

src/main.cpp

#include <ChuckNorris.hpp>
#include <iostream>

int main()
{
  ChuckNorris chuckNorris;
  std::string fact = chuckNorris.getFact();
  std::cout << fact << std::endl;
  return 0;
}

All we have to do is run the program without arguments, and check that something gets displayed. That will probably be enough to check our library works.

Targets #

Let’s enumerate what we have to do now to build everything:

Make sure that the include/ directory is used when we compile code in src/
Build a library using src/ChuckNorris.cpp
Build an executable named cpp_demo using src/main.cpp that links with the above library.

We can say we have two targets: a library called chucknorris, and an executable named cpp_demo:

Here’s what the commands to build them look like on Linux:

# Compile the .cpp file into a .o
$ g++ -c -I include/ src/ChuckNorris.cpp -o libchucknorris.o
# Create an archive containing the .o
$ ar cr libchucknorris.a libchucknorris.o
# Compile main.cpp into main.o
$ g++ -c -I include/ src/main.cpp -o main.o
# Tell g++ to link everything into an executable
$ g++ main.o libchucknorris.a -o cpp_demo
# Run the executable we've just built:
$./cpp_demo
Chuck Norris can slam a revolving door.

Phew!

Keeping track of all this by hand would get tedious very soon, but fortunately there are tools that can help us.

CMake #

CMake lets us describe the targets we want to build in code.

We do it by writing the following code in a file named CMakeLists.txt:

cmake_minimum_required(VERSION 3.10)
set(CMAKE_CXX_STANDARD 11)

project(ChuckNorris)

add_library(chucknorris
    include/ChuckNorris.hpp
    src/ChuckNorris.cpp
)


target_include_directories(
  chucknorris
  PUBLIC
    "include"
)

add_executable(cpp_demo
  src/main.cpp
)

target_link_libraries(cpp_demo chucknorris)

The add_library and add_executable commands describe the two targets (named chucknorris and cpp_demo) and the sources used to build them, and the target_link_libraries command tells CMake that cpp_demo depends on chucknorris.

The target_include_directories informs CMake that there are header files in the include directories that should be used both when builing the library itself, but also by consumers of the library. (We used the -I include/ flag both for building libchucknorris.o and main.co)

If the headers were used only to compile the library, we would have used the PRIVATE keyword, and if they were used only by consumers of the library, we would have used the INTERFACE keyword instead. You can read more about this in the CMake documentation.

Also note how we set the CMAKE_CXX_STANDARD variable. This will allow us to use modern C++ features (available in C++ 11 or later) such as the auto keyword or raw string literals later on.

Ninja #

Now that we have described what we want to build and how, we still need to perform the build itself.

CMake does not know how to actually perform the build. Instead it generates files that will be used by an other tool. It’s called a CMake generator.

There are plenty of generators available, but for now we’ll only talk about Ninja. I’ve already explain why I prefer using CMake with Ninja in an other blog post.

In our first attempt, we generated all the binaries (libchucknorris.a, the .o files and the cpp_demo executable) directly in the current working directory. It’s cleaner to have them put inside a dedicated build folder instead:

We’ll only have to put the build folder into our .gitignore file, instead of a bunch of files
If we want to, we can have several build folders, all using the same sources, but using different compilers or flags without risking mixing incompatible binaries.

So let’s create a folder named build/default and call CMake, asking it to use the Ninja generator. CMake uses the current working directory as the build folder, and you must specify the path to the folder containing the CMakeLists.txt file as the last argument on the command line:

$ mkdir -p build/default
$ cd build/default
$ cmake -GNinja ../..

And now we use ninja to build build and run our executable from the build folder:

$ cd build/default
$ ninja
[1/4] Building CXX object CMakeFiles/chucknorris.dir/src/ChuckNorris.cpp.o
[2/4] Building CXX object CMakeFiles/cpp_demo.dir/src/main.cpp.o
[3/4] Linking CXX static library libchucknorris.a
[4/4] Linking CXX executable cpp_demo
$ ./cpp_demo
Chuck Norris can slam a revolving door.

Done!

Note that CMake and Ninja cooperate so that you only rebuild what’s need to be rebuilt.

If we change just the main.cpp, we just have to rebuild main.cpp.o and relink the cpp_demo:

$ ninja
[1/2] Building CXX object CMakeFiles/cpp_demo.dir/src/main.cpp.o
[2/2] Linking CXX executable cpp_demo

But if we change ChuckNorris.cpp, everything except main.cpp.o needs to be rebuilt:

[1/3] Building CXX object CMakeFiles/chucknorris.dir/src/ChuckNorris.cpp.o
[2/3] Linking CXX static library libchucknorris.a
[3/3] Linking CXX executable cpp_demo

And if we change the CMakeLists.txt file, Ninja will re-run CMake for us:

$ ninja
[0/x] Re-running CMake...
-- Configuring done
-- Generating done
-- Build files have been written to: /path/to/build/default
...

That’s all for the first part. Stay tuned for part 2, where we’ll introduce an external dependency and get rid of the hard-coded fact.

Introducing the "Let's Build Chuck Norris!" Project

Sat, 10 Mar 2018 12:35:38 +0000

The ChuckNorris library #

The Chuck Norris library is written in C++ and contains a class able to give you random Chuck Norris facts when the getFact() method is called:

Here’s how to use it:

#include <ChuckNorris.hpp>

ChuckNorris chuckNorris;
std::string fact = chuckNorris.getFact();

You will find all the source code on GitHub.

Let’s build Chuck Norris! #

“Let’s Build Chuck Norris!” is a series of blog posts aiming at exploring various topics about C++ and build systems, and providing an example of the Salami method described by Adi Shavit.

Here are all of them, in chronological order. Enjoy!

Part 1 - Using CMake and Ninja

Part 2 - Managing third-party dependencies with conan

Part 3 - Exposing a C API on top of C++ code

Part 4 - Using Python with ctypes

Part 5 - Using cffi to build a Python extension

Part 6 - Cross-compiling for Android

Part 7 - Android and JNA

Part 8 - Using C++ in an iOS application

Porting to pytest: a practical example

Thu, 01 Mar 2018 16:34:19 +0000

Introduction #

The other day I was following the django tutorial.

If you never read the tutorial, or don’t want to, here’s what you need to know:

We have a django project containing an application called polls.

We have two model objects representing questions and choices.

Each question has a publication date, a text, and a list of choices.

Each choice has a reference to an existing question (via a foreign key), a text, and a number of votes.

There’s a view that shows a list of questions as links. Each link, when clicked displays the choices and has a form to let the user vote.

The code is pretty straightforward:

# polls/models.py
class Question(models.Model):
    question_text = models.CharField(max_length=200)
    pub_date = models.DateTimeField('date published')

    def was_published_recently(self):
        now = timezone.now()
        two_days_ago = now - datetime.timedelta(days=2)
        return two_days_ago < self.pub_date <= now


class Choice(models.Model):
    question = models.ForeignKey(Question, on_delete=models.CASCADE)
    choice_text = models.CharField(max_length=200)

Everything went smoothly until I arrived at the part 5, about automated testing, where I read the following:

Sometimes it may seem a chore to tear yourself away from your productive, creative programming work to face the unglamorous and unexciting business of writing tests, particularly when you know your code is working properly.

Well, allow me to retort!

Starting point: using tests from the documentation #

Here’s what the tests looks like when extracted from the django documentation:

import datetime

from django.utils import timezone
from django.test import TestCase

from .models import Question


class QuestionModelTests(TestCase):
    def test_was_published_recently_with_future_question(self):
        """
        was_published_recently() returns False for questions whose pub_date
        is in the future.
        """
        time = timezone.now() + datetime.timedelta(days=30)
        future_question = Question(pub_date=time)
        self.assertIs(future_question.was_published_recently(), False)

    def test_was_published_recently_with_old_question(self);
        """
        was_published_recently() returns False for questions whose pub_date
        is older than 1 day.
        """
        time = timezone.now() - datetime.timedelta(days=1, seconds=1)
        old_question = Question(pub_date=time)
        self.assertIs(old_question.was_published_recently(), False)


    def test_was_published_recently_with_recent_question(self):
        """
        was_published_recently() returns True for questions whose pub_date
        is within the last day.
        """
        time = timezone.now() - datetime.timedelta(hours=23, minutes=59, seconds=59)
        recent_question = Question(pub_date=time)
        self.assertIs(recent_question.was_published_recently(), True)


def create_question(question_text, days):
    """
    Create a question with the given `question_text` and published the
    given number of `days` offset to now (negative for questions published
    in the past, positive for questions that have yet to be published).
    """
    time = timezone.now() + datetime.timedelta(days=days)
    return Question.objects.create(question_text=question_text, pub_date=time)


class QuestionIndexViewTests(TestCase):
    def test_no_questions(self):
        """
        If no questions exist, an appropriate message is displayed.
        """
        response = self.client.get(reverse('polls:index'))
        self.assertEqual(response.status_code, 200)
        self.assertContains(response, "No polls are available.")
        self.assertQuerysetEqual(
            response.context['latest_question_list'],
            []
        )

    def test_past_question(self):
        """
        Questions with a pub_date in the past are displayed on the
        index page.
        """
        create_question(question_text="Past question.", days=-30)
        response = self.client.get(reverse('polls:index'))
        self.assertQuerysetEqual(
            response.context['latest_question_list'],
            ['<Question: Past question.>']
        )

    def test_future_question(self):
        """
        Questions with a pub_date in the future aren't displayed on
        the index page.
        """
        create_question(question_text="Future question.", days=30)
        response = self.client.get(reverse('polls:index'))
        self.assertContains(response, "No polls are available.")
        self.assertQuerysetEqual(
            response.context['latest_question_list'],
            []
        )

    def test_future_question_and_past_question(self):
        """
        Even if both past and future questions exist, only past questions
        are displayed.
        """
        create_question(question_text="Past question.", days=-30)
        create_question(question_text="Future question.", days=30)
        response = self.client.get(reverse('polls:index'))
        self.assertQuerysetEqual(
            response.context['latest_question_list'],
            ['<Question: Past question.>']
        )

    def test_two_past_questions(self):
        """
        The questions index page may display multiple questions.
        """
        create_question(question_text="Past question 1.", days=-30)
        create_question(question_text="Past question 2.", days=-5)
        response = self.client.get(reverse('polls:index'))
        self.assertQuerysetEqual(
            response.context['latest_question_list'],
            [
                '<Question: Past question 2.>',
                '<Question: Past question 1.>'
            ]
        )

We can run them using the manage.py script and check they all pass:

$ python manage.py test polls
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
........
----------------------------------------------------------------------
Ran 8 tests in 0.017s

OK
Destroying test database for alias 'default'...

OK, tests do pass. Let’s try and improve them.

I’ve set up a GitHub repository where you can follow the following steps commit by commit if you wish.

Step one: setup pytest #

I’ve already told you how much I love pytest, so let’s try to convert to pytest.

The first step is to install pytest-django and configure it:

$ pip install pytest pytest-django

# in pytest.ini
[pytest]
DJANGO_SETTINGS_MODULE=mysite.settings
python_files = tests.py test_*.py

We can now run tests using pytest directly:

$ pytest
========== test session starts ========
platform linux -- Python 3.5.3, pytest-3.3.1, py-1.5.2, pluggy-0.6.0
Django settings: mysite.settings (from ini file)
rootdir: /home/dmerej/src/dmerej/django-polls, inifile: pytest.ini
plugins: django-3.1.2
collected 8 items

polls/tests.py ........   [100%]

======== 8 passed in 0.18 seconds =======

Step two: rewrite assertions #

We can now use pytest magic to rewrite all “easy” assertions such as assertFalse or assertEquals:

- self.assertFalse(future_question.was_published_recently())
+ assert not future_question.was_published_recently()

Already we can see several improvements:

The code is more readable and follows PEP8
The error messages are more detailed:

# Before, with unittest
$ python manage.py test
    def test_was_published_recently_with_future_question(self):
        ...
>       self.assertFalse(question.was_published_recently())
E       AssertionError: True is not false

# After, with pytest
$ pytest
>       assert not question.was_published_recently()
E       AssertionError: assert not True
E        +  where True = <bound method was_published_recently() of Question>

Then we have to deal with assertContains and assertQuerysetEqual which look a bit django-specific.

For assertContains I quickly managed to find I could use response.rendered_content instead:

- self.assertContains(response, "No polls are available.")
+ assert "No polls are available." in response.rendered_content

For assertQuerysetEqual it was a bit harder.

def test_past_question(self):
    create_question(question_text="Past question.", days=-30)
    response = self.client.get(reverse('polls:index'))
    self.assertQuerysetEqual(
        response.context['latest_question_list'],
        ['<Question: Past question.>']
    )

This test checks that the context used to generate the response was passed correct latest_question_list value.

But it does so by checking the string representation of the Question object.

Thus, it will break as soon as Question.__str__ changes, which is not ideal.

So instead, we can write something like this and check for the content of the question_text attribute directly:

def test_past_question(self):
    create_question(question_text="Past question.", days=-30)
    response = self.client.get(reverse('polls:index'))
    actual_questions = response.context['latest_question_list']
    assert len(actual_questions) == 1
    actual_question = actual_questions[0]
    assert actual_question.question_text == "Past question"

While we’re at it, we can introduce small helper functions to make the tests easier to read:

For instance, the string No polls are available is hard-coded twice in the tests. Let’s introduce a assert_no_polls helper:

def assert_no_polls(text):
    assert "No polls are available" in text

- assert "No polls are available." in response.rendered_content
+ assert_no_polls(response.rendered_content)

An other hard-coded string is polls:index, so let’s introduce get_latest_list:

def get_latest_list(client):
    response = client.get(reverse('polls:index'))
    assert response.status_code == 200
    return response.context['latest_question_list']

Note how we embedded the status code check directly in our helper, so we don’t have to repeat the check in each test.

Also, note that if the name of the route (polls:index) or the name of the context key used in the template (latest_question_list) ever changes, we’ll just need to update the test code in one place.

Then, we can further simplify our assertions:

def assert_question_list_equals(actual_questions, expected_texts):
    assert len(actual_questions) == len(expected_texts)
    for actual_question, expected_text in zip(actual_questions, expected_texts):
        assert actual_question.question_text == expected_text

def test_past_question(self):
    ...
    create_question(question_text="Past question.", days=-30)
    latest_list = get_latest_list(self.client)
    assert_question_list_equals(latest_list, ["Past question."])

Step three: move code out of classes #

The nice thing about pytest is that you don’t need to put your tests as methods of a class, you can just write test functions directly.

So we just remove the self parameter, indent back all the code, and we are (almost) good to go.

We already got rid of all the self.assert* methods, so the last thing to do is pass the Django test client as a parameter instead of using self.client. (That’s how pytest fixtures work):

-    def test_two_past_questions(self):
-        ...
-        latest_list = get_latest_list(self.client)

+ def test_no_questions(client):
+    latest_list = get_latest_list(client)

But then we encounter an unexpected failure:

Polls/tests.py:34: in create_question
    return Question.objects.create(question_text=question_text, pub_date=time)

    ...

>       self.ensure_connection()
E       Failed: Database access not allowed, use the "django_db" mark, or the "db" or "transactional_db" fixtures to enable it.

Back when we used python manage.py test, django’s manage.py script was implicitly creating a test database for us.

When we use pytest, we have to be explicit about it and add a special marker:

import pytest

# No change here, no need for a DB
def test_was_published_recently_with_old_question():
    ...

# We use create_question, which in turn calls Question.objects.create(),
# so we need a database here:
@pytest.mark.django_db
def test_no_questions(client):
    ...

True, this is a bit annoying, but note that if we only want to test the models themselves (like the was_published_recently() method), we can just use:

$ pytest -k was_published_recently

and no database will be created at all.

Step four: Get rid of doc strings #

I don’t like doc strings, except when I’m implementing a very public API. There, I’ve said it.

I very much prefer when the code is “self-documenting”, especially when it’s test code.

As Uncle Bob said, “tests should read like well-written specifications”. So let’s try some refactoring.

We can start with more meaningful variable names, and have more fun with the examples:

def test_was_published_recently_with_old_question():
-   time = timezone.now() - datetime.timedelta(days=1, seconds=1)
-   old_question = Question(pub_date=time)
+   last_year = timezone.now() - datetime.timedelta(days=365)
+   old_question = Question('Why is there something instead of nothing?',
+                            pub_date=last_year)
    assert not old_question.was_published_recently()

def test_was_published_recently_with_recent_question():
-   time = timezone.now() - datetime.timedelta(days=1, seconds=1)
-   recent_question = Question(pub_date=time)
+   last_night = timezone.now() - datetime.timedelta(hours=10)
+   recent_question = Question('Dude, where is my car?', pub_date=last_night)

Time and date code is always tricky, and a negative number of days does not really make sense, so let’s make things easier to reason about:

def n_days_ago(n):
    return timezone.now() - datetime.timedelta(days=n)


def n_days_later(n):
    return timezone.now() + datetime.timedelta(days=n)

Also create_question is coupled with the Question model, so let’s use the same names for the parameter names and the model’s attributes.

And since we may want to create question without caring about the publication date, let’s make it an optional parameter:

def create_question(question_text, *, pub_date=None):
    if not pub_date:
        pub_date = timezone.now()
    ...

Code becomes:

-    create_question(question_text="Past question.", days=-30)
+    create_question(question_text="Past question.", pub_date=n_days_ago(30))

Finally, let’s add a new test to see if our helpers really work:


@pytest.mark.django_db
def test_latest_five(client):
    for i in range(0, 10):
        pub_date = n_days_ago(i)
        create_question("Question #%s" % i, pub_date=pub_date)
    latest_list = get_latest_list(client)
    assert len(actual_list) == 5

Do you still think this test needs a docstring ?

Step five: fun with selenium #

Selenium basics #

Selenium deals with browser automation.

Here we are going to use the Python bindings, which allow us to start Firefox or Chrome and control them with code.

(In both cases, you’ll need to install a separate binary: geckodriver or chromedriver respectively)

Here’s how you can use selenium do visit a web page and click the first link:

import selenium.webdriver

driver = selenium.webdriver.Firefox()
# or
driver = selenium.webdriver.Chrome()
driver.get("http://example.com")
link = driver.find_element_by_tag_name('a')
link.click()

The Live Server Test Case #

Django exposes a LiveServerTestCase, but no LiveServer object or similar.

The code is a bit tricky because it needs to spawn a “real” server in a separate thread, make sure it uses a free port, and tell the selenium driver to use an URL like http://localhost:32456

Fear not, pytest also works fine in this case. We just have to be careful to use super() in the set up and tear down methods so that the code from LiveServerTestCase gets executed properly:

import urllib.parse


class TestPolls(LiveServerTestCase):
    serialized_rollback = True

    def setUp(self):
        super().setUp()
        self.driver = selenium.webdriver.Firefox()

    def tearDown(self):
        self.driver.close()
        super().tearDown()

    def test_home_no_polls(self):
        url = urllib.parse.urljoin(self.live_server_url, "/polls")
        self.driver.get(url)
        assert_no_polls(self.browser.page_source)

If you’re wondering why we need serialized_rollback=True, the answer is in the documentation. Without it we may have weird database errors during tests.

Our first test is pretty basic: we ask the browser to visit the 'polls/ URL and check no polls are shown, re-using our assert_no_polls helper function from before.

Let’s also check we are shown links to the questions if they are some, and can click on them:

class TestPolls(LiveServerTestCase):
    ...
    def test_home_list_polls(self):
        create_question("One?")
        create_question("Two?")
        create_question("Three?")
        url = urllib.parse.urljoin(self.live_server_url, "polls/")
        self.driver.get(url)
        first_link = self.driver.find_element_by_tag_name("a")
        first_link.click()
        assert "Three?" in self.driver.page_source

Let’s build a facade #

The find_element_by_* methods of the selenium API are a bit tedious to use: thery are called find_element_by_tag_name, find_element_by_class_name, find_element_by_id and so on

So let’s write a Browser class to hide those behind a more “Pythonic” API:

# old
link = driver.find_element_by_tag_name("link")
form = driver.find_element_by_id("form-id")

# new
link = driver.find_element(tag_name="link")
form = driver.find_element(id="form-id")

(This is known as the “facade” design pattern)

class Browser:
    """ A nice facade on top of selenium stuff """
    def __init__(self, driver):
        self.driver = driver

    def find_element(self, **kwargs):
        assert len(kwargs) == 1   # we want exactly one named parameter here
        name, value = list(kwargs.items())[0]
        func_name = "find_element_by_" + name
        func = getattr(self.driver, func_name)
        return func(value)

Note how we have to convert the items() to a real list just to get the first element… (In Python2, kwargs.items()[0] would have worked just fine). Please tell me if you find a better way …

Note also how we don’t just inherit from selenium.webdriver.Firefox. The goal is to expose a different API, so using composition here is better.

If we need access to attributes of self.driver, we can just use a property, like this:

class Browser

    ...

    @property
    def page_source(self):
        return self.driver.page_source

And if we need to call a method directly to the underlying object, we can just forward the call:

    def close(self):
        self.driver.close()

We can also hide the ugly urllib.parse.urljoin(self.live_server_url) implementation detail:


class Browser:
    def __init__(self, driver):
        self.driver = driver
        self.live_server_url = None  # will be set during test set up

    def get(self, url):
        full_url = urllib.parse.urljoin(self.live_server_url, url)
        self.driver.get(full_url)


class TestPolls(LiveServerTestCase):

    def setUp(self):
        super().setUp()
        driver = selenium.webdriver.Firefox()
        self.browser = Browser(driver)
        self.browser.live_server_url = self.live_server_url

Now the test reads:


    def test_home_no_polls(self):
        self.browser.get("/polls")
        assert_no_polls(self.browser.page_source)

Nice and short :)

Launching the driver only once #

The setUp() method is called before each test, so if we add more tests we’re going to create tons of instances of Firefox drivers.

Let’s fix this by using setUpClass (and not forgetting the super() call)

class TestPolls(LiveServerTestCase):

    @classmethod
    def setUpClass(cls):
        super().setUpClass()
        driver = webdriver.Chrome()
        cls.browser = Browser(driver)

    def setUp(self):
        self.browser.base_url = self.live_server_url

    @classmethod
    def tearDownClass(cls):
        cls.browser.close()
        super().tearDownClass()

Now the browser is a class attribute instead of being an instance attribute. So there’s only one Browser object for the whole test suite, which is what we wanted.

The rest of the code can still use self.browser, though.

Debugging tests #

One last thing. You may think debugging such high-level tests would be painful.

But it’s actually a pretty nice experience due to just one thing: the built-in Python debugger!

Just add something like:


def test_login():
    self.browser.get("/login")
    import pdb; pdb.set_trace()

and then run the tests like this:

$ pytest -k login -s

(The -s is required to avoid capturing output, which pdp does not like)

And then, as soon as the tests reaches the line with pdb.set_trace() you will have:

A brand new Firefox instance running, with access to all the nice debugging tools (so you can quickly find out things like ids or CSS class names)
… and a nice REPL where you’ll be able to try out the code using self.browser

By the way, the REPL will be even nicer if you use ipdb or pdbpp and enjoy auto-completion and syntax coloring right from the REPL :)

Conclusion #

I hope I managed to show that you actually can get creative writing tests, and even have some fun.

See you next time!

Helping BBC subtitlers

Sun, 18 Feb 2018 13:52:40 +0000

Note: this the first post in the Quantum of Ideas series.

Introduction #

The other day I stumbled upon an article called Why subtitlers have one of the hardest jobs in TV. Go read it now, I’ll wait.

That got me thinking. What if there was a better way to transcribe what is said on TV into proper text, without having humans trying to repeat what they heard in a robotic voice, and then not-so-good “speech to text” programs converting robotic human voices to words?

An idea using old tech #

Turns out that writing text as fast as it is spoken is a very old problem. It’s called stenography (or shorthand), and according to Wikipedia, it’s been around since classical antiquity.

Heres’s what stenography looks like using the French “Prévost-Delaunay” technique:

In the same vein, converting human hand-writing to text is also a well-known problem. Just take a look at OCR history.

Speech recognition is quite new in comparison, and even if it has improved in the last decades, it’s still not entirely reliable, as the RadioTimes article shows.

So here’s the idea:

First, train people in stenography. In French, there used to be a job called “sténo-dactylo”, a secretary who used shorthand techniques to transcribe speech into an optimized hand-written notation, and then re-entered the text on a typing machine.

To be trained in stenography can be useful in a bunch of situations. For instance, sometimes video or audio recording is prohibited by the law, and it’s up to a court clerk to transcribe everything that is said during a trial: see this scene from “intolerable Cruelty” for an example.

Note how the problem of punctuation goes away, because you can just write ? instead of saying “question mark”.

Also, the issue of coloring the subtitles with different colors depending on who’s speaking could be solved by inventing a notation, like prefixing the spoken words by the name of the speaker.

Then, tweak an OCR Software so that it can work with the kind of input shown above. This may be a bit tricky because most of the time, the stenography techniques use a “phonetic” system, meaning that “eye” and “I” would both be written the same way. But I think we could train the software to automatically deduce the correct spelling.

There’s also the issue of proper nouns, but maybe we could fall back to regular writing for those.

Looks to me this would cost less in R&D, and could also lead to a better quality of life for the subtitlers. No more sore face at the end of the day, and no more voice tics.

An even simpler idea #

While I was doing research for this article, I found that there’s also a job called STTR. The “Speech To Text Reporters” use a dedicated machine, as seen on this demonstration video.

This seems to be already working just fine! Note how the names of the person speaking are handled, and how it’s a fully regulated profession. STTRs type at minimum 180 words per minute and with 97% accuracy, and are tested in live situations.

There’s a tiny issue because speech to text reporters have both their hands busy all the time, which means it would be hard for them to modify the position of the subtitles in real-time. But I believe the task of finding a ’text-free’ zone in the image and automatically move the subtitles could quite easily be done automatically by a small piece of software.

So the question becomes: what was the reason for the BBC not to hire STTRs for the purpose of subtitling all their programs?

I don’t have the answer, but if you ever need real-time speech to text conversion, please consider using a different technique than “respeaking”.