(blog . (coldboot . org))

Fixing Programmatic Tool Calling With Types

2026-03-10T13:37:00Z

Anthropic recently introduced Programmatic Tool Calling, which lets Large Language Models write Python programs to orchestrate multiple tool calls with loops, conditionals, and error handling. It is a very powerful method to have an agent run a huge number of tool calls without filling up its context window, and also avoid the latency of the usual call-one-tool-at-a-time loop.

This approach works great in demos, however it runs into a number of issues in production, and they all have the same root cause: Python is too expressive for this job. I created a small, restricted programming language called λ-Tool in OCaml to solve this problem, and it catches four common tool calling issues at compile time, before a single tool call is made.

The Issues

Say you have a support agent. A customer writes in: "I was charged twice." The LLM looks up the account, issues a refund, sends a reply. Here's the Python code it generates:

account = await lookup_account(ticket.customer_id)
refund = await issue_refund({
    "account_id": account["id"],
    "amount": account["last_charge_cents"]
})
await send_reply({
    "to": account["email"],
    "body": f"Refund of ${refund['amount'] / 100:.2f} processed. "
            f"Transaction: {refund['tx_id']}"
})

This passes the sandbox. Every test succeeds. And yet, it has at least three bugs:

Wrong field The account record has subscription_tier. The LLM writes account["plan"] somewhere in the conditional logic. The mock returns a dict with everything the test expects, so the KeyError never fires. In production it crashes, but only after side effects have already happened.
Double write issue_refund succeeds but the response times out. The LLM's error handling retries the call. The refund went through on the first attempt. Now the customer has been refunded twice.

try:
    refund = await issue_refund({"account_id": acct_id, "amount": 4999})
except TimeoutError:
    refund = await issue_refund({"account_id": acct_id, "amount": 4999})

Unhandled failure The reply gets sent even when the refund fails, because send_reply isn't conditional on issue_refund succeeding. The customer gets an email saying "your refund has been processed" when it hasn't.

The same bugs show up everywhere LLMs compose tools. An incident response bot retries a deployment rollback on timeout, re-deploying the broken version. A meeting bot creates a Jira ticket, the call fails, but sends a Slack message referencing the ticket key anyway. Wrong fields, unhandled errors, unsafe retry, unbounded loops.

λ-Tool

λ-Tool makes these four bugs compile-time type errors; the LLM generates λ-Tool code instead of Python, then a type checker verifies the code before any tool runs. If the code is wrong, the LLM gets a type error, fixes it, and resubmits. Nothing executed, so the retry is free.

Here's the same support agent in λ-Tool:

tool lookup_account: String -{Read}-> {id: String, subscription_tier: String,
                                       email: String, last_charge_cents: Int};
tool issue_refund: {account_id: String, amount: Int} -{Write}-> {tx_id: String};
tool send_reply: {to: String, body: String} -{Write}-> Unit;

match exec tool lookup_account "cust_12345" {
  Ok(account) =>
    match exec tool issue_refund {account_id = account.id,
                                   amount = account.last_charge_cents} {
      Ok(refund) =>
        match exec tool send_reply {to = account.email,
                                     body = refund.tx_id} {
          Ok(u) => "refund processed and customer notified",
          Err(e) => "refund succeeded but notification failed"
        },
      Err(e) => "refund failed, customer not refunded"
    },
  Err(e) => "account lookup failed"
}

Every tool declares its input type, output type, and effects (Read, Write). The types are how the checker knows what fields exist. If the LLM writes account.plan, it gets:

Field 'plan' not found in record type
  {id: String, subscription_tier: String, email: String, last_charge_cents: Int}

Every exec returns a Result. The only way to use it is match ... { Ok(x) => ..., Err(e) => ... }. The type checker rejects any program that uses a Result without matching both branches. send_reply is nested inside Ok(refund), so it only runs if the refund succeeded. In the Err branch, refund doesn't exist as a variable, so the LLM can't reference data from a failed call.

There is no retry mechanism. exec tool issue_refund ... evaluates exactly once, and the language has no try/except, no while, no recursion. You can't write a retry loop because the syntax doesn't allow it.

Iteration is bounded: map, filter, fold over finite lists; these require pure callbacks. Trying to call a tool inside map is a type error:

tool charge: {id: Int, amount: Int} -{Write}-> {tx_id: String};

map (fn o: {id: Int, amount: Int} => exec tool charge o)
    [{id = 1, amount = 100}]
(* Effect violation: allowed {} but got {Write} *)

Effectful iteration uses traverse, which short-circuits on the first error and forces Result handling:

match traverse (fn o: {id: Int, amount: Int} =>
    exec tool charge o
  ) [{id = 1, amount = 100}] {
  Ok(receipts) => "done",
  Err(e) => e
}

Unlike Python-based Programmatic Tool Calling implementations, λ-Tool does not require a sandbox because the restrictiveness is the sandbox.

Why Types

The idea of restricting a language to prevent errors goes back to Alonzo Church, who added types to his lambda calculus in 1940 specifically to forbid nonsense like applying a number to a number. The typed version could express fewer programs, and yet that was the point: the programs you lose are exactly the ones you don't want. Robin Milner took this further with ML, adding type inference and mandatory pattern matching, so the compiler could prove you'd handled every case before the code ran. The Curry-Howard correspondence formalized why this works: types are propositions, programs are proofs, and a type checker is a proof checker. When λ-Tool rejects a program that doesn't handle Err, it's rejecting an incomplete proof.

Each of λ-Tool's four guarantees comes from a specific result in this lineage. Row types, from Didier Rémy's work extending ML records in the early 90s, give us field safety: {id: String, email: String} means exactly those fields, and accessing anything else is a compile-time error. Mandatory Result matching is just ML's exhaustive pattern matching applied to tool calls, and bounded iteration (no while, no recursion) gives guaranteed termination.

The double-write problem, on the other hand, needed something newer. Jean-Yves Girard's linear logic introduced the idea that a fact can be used exactly once: when you use it, it's consumed. λ-Tool's !T annotation applies this to values:

fn token: !String => let a = token in let b = token in a
(* Linearity violation for 'token': used 2 times (must be exactly once) *)

A refund token, an API nonce, a deployment rollback: these are resources that should be consumed exactly once. Linear types enforce that at compile time.

Jane Street routes hundreds of billions of dollars through OCaml, a direct descendant of ML, because its type system catches at compile time what would be runtime crashes in Python. λ-Tool applies the same tradition to the problem of tool composition by language models.

Results

I ran both Claude Sonnet and Haiku against 8 PTC tasks, generating both Python and λ-Tool. Both models generated valid λ-Tool on the first try over 94% of the time. The remaining ~4% were parse errors, caught instantly before any tool runs.

In this experiment, every λ-Tool program that parsed also typed correctly, and every program that typed correctly executed without errors. The Python programs all ran without exceptions too, but that's the problem: several of them contained silent double-write bugs that the sandbox didn't catch. λ-Tool rejected the same patterns at compile time.

The full experiment setup and results are in the paper.

Implementation

λ-Tool is ~2,500 lines of OCaml, containing a lexer, parser, bidirectional type checker, and interpreter. The type checker has 178 tests across three suites, plus 200,000 QCheck-generated random well-typed programs, with zero counterexamples of the type system's guarantees.

The Python package wraps the OCaml CLI:

from lambda_tool import LambdaTool

lt = LambdaTool()

# Type-check -- no execution, no side effects
check = lt.typecheck(code)

# Execute with real tool implementations
result = lt.run(code, executors={
    "lookup_account": real_account_api,
    "issue_refund": real_refund_api,
    "send_reply": real_email_api,
})

minilambda is a basic (~200 LoC) demo agent that uses this Python package for tool calling. Claude generates λ-Tool code, the type checker verifies it, the interpreter executes it with real Python callbacks. It processes a batch of orders, which includes charging cards and sending receipts, with a simulated ~30% card decline rate. The type system forces Claude to handle the failures, because the code won't compile otherwise.

About Me

My name is Sarthak and I like using cross-disciplinary techniques to solve interesting problems. The ivory tower has many spires, and I like building bridges that connect them. If you have something interesting that you would like me to work on, please feel free to reach out.

Parameterized Packages: The Project Completion Update

2023-09-18T13:37:00Z

This post is a small completion update for my GSoC project on adding Parameterized Packages to GNU Guix. You can find the source code for the same on the main notabug repository or on the github mirror. This post is broken into two parts; the first is a simple overview of the uses of parameters and the second answers some commonly asked questions.

What is Package Parameterization?

In its current state, Package Parameterization makes it possible to add compile-time options to any Guix package, regardless of whether it was defined with support for a given compile-time option or not. This makes it a very powerful tool for hackers seeking to extend their packages in unorthodox and interesting ways. But that's not all! Parameters can be used to make any kind of changes to package definitions conditionally, making them very useful substitutes for unimplemented package transformation options in a lot of cases.

What are compile-time options?

Compile-time options are package features that can be selected only when a package is being compiled from source. Most binary-based GNU/Linux distributions pick these options for the users, often selecting options that the user might not need while leaving out some options that they might really need. These become especially relevant in the context of High-Performance Computing and computing on alternate ISAs, where greater flexibility with compile-time options is greatly needed.

What makes Package Parameterization special?

The strength of package parameterization lies in the fact that users will now be able to create generic compile-time options that they will be able to apply to any number of packages they like. It's almost akin to being able to create custom package transformation options, with the caveat that only an enumerated set of options will be available for choosing.

Generic Compile-time Options

The compile-time options created by parameters will be generic if the author chooses to write a global parameter for a given parameterization. This means that if, for example, I write a global parameter that enables text-to-speech support, it will work on any package that satisfies this parameter's predicate. The predicate is a user-defined function that returns #t or #f after checking a number of things about the package passed to it like the package's build system, inputs or arguments among other things. This helps prevent the usage of parameters on packages that they are not compatible with.

What does this look like in action?

I have demonstrated this in great detail in this post, for a quick example please head to the section where I've parameterized a bunch of variants of Emacs ("What does using parameters look like?"). All of the examples in the post work and you can try them out by applying the parameterization patch to a local copy of Guix.

Reasons for trying Package Parameters

In no particular order, here are some good reasons why you should try using Package Parameters

1. Speeding up packages that you use very often

To maintain stability and ensure that packages work across a wide variety of devices, Guix usually does not build packages with a lot of compile-time speed optimizations. However, with parameterization it is possible to speed up packages that you use very often for a much smoother and faster computing experience.

2. Easily creating and accessing multiple variants for packages

Parameters make it easy to succinctly write packages with multiple variants and switch between them with the --with-parameter transform. It is also possible to take an existing package and add a number of variants to it, like in the Emacs example mentioned in the previous heading.

3. Applying compile-time customizations to a number of packages

Using global parameters, it's possible to have common parameterized options across a huge number of packages. This makes it easy to use parameterized variants of packages without having to rewrite each of them with the compile-time option you want to use.

4. Learning more about your packages

This last reason is a bit opinionated, but I think that you get to learn a lot more about how the packages you're using work when you tweak them like this. It makes you mindful of all the packages that you are using, and sometimes even helps you discover new things about them that you otherwise wouldn't know about!

Answers to some commonly asked questions

Changes in the Syntax

The main point of confusion so far has been the syntax for package parameterization, which has gone through a couple of changes over time.

The Parameter List

The parameter list is a fundamental component of this project. At any place that accepts a list of parameters, the user passes a parameter list. Parameter lists are lists of the form ((parameter value) (parameter-2 value-2) ...) The first value in the cell is the symbol of the parameter, and the second value in the cell is the value of the parameter.

Negation and `!`

Previously, we were using ! to denote the negation of a parameter. This has been removed in newer versions. The rationale behind this removal is that ! does not clearly denote negation, and having arbitrary restrictions behind what kind of symbols are allowed and what kind are not would again cause confusion. Instead, we now use #:off to denote the negative value for any given parameter.

;; old style
(parameter-if ((a val) b!)
    do-thing)

;; new style
(parameter-if ((a val) (b #:off))
    do-thing)

Handling of Global Parameters

This topic is still under debate, but for now I have opted to use a separate namespace for global parameters. I'm accomplishing this with the help of a global hash-table. This is for two main reasons:

We can guarantee that global parameters are unique by adding checks for uniqueness in the global parameter definition macro
Parameter names can stay small inside the various parameter lists in the parameter-spec

Furthermore, a lot of the existing processing functions work on unique symbols denoting the parameters and not the parameters themselves, so a massive rewrite would be necessary if we want to pass them parameters instead.

Support for setting Global Parameter Values

Due to the very alpha state of parameterization, this has not yet been implemented. However parameters are a package transformation option and can be used just like any other package transformation option, so it's possible to create manifest files with parameterization for the packages a user is using.

Documentation and Tests?

I have not written texinfo documentation or unit tests for Package Parameterization yet as they're expected to change considerably in the process of being merged into master.

When will Parameters be merged to master?

While Package Parameterization itself works great, among other things a host of tools for writing parameterized packages more easily and measuring the complexity of parameterized packages are necessary before this project is merged to master. In the meantime, I would really appreciate feedback and bug reports on the current parameterized packages patch. It can be added to a source install of Guix with a simple patch that does not affect any other component of Guix. Please raise issues on the notabug repository or send me an email if you encounter any issues with this patch, it would greatly help with development.

Installing the patch

To install the patch, you will need to build a git copy of Guix. You can find instructions to do the same here. After building Guix from git, copy the patch into the directory and run git apply parameterization.patch to apply it to the source. Using ./pre-inst-env (see Running Guix Before It Is Installed) now should give you access to parameters.

Thanks

I owe a great deal of credit for this project to my mentors Pjotr Prins and Gabor Boskovits as their guidance was instrumental to its completion. I would also like to thank Ludovic Courtes, Arun Isaac and Efraim Flashner for their feedback and suggestions, which shaped this project into something much more beautiful than what I had originally envisioned. I also learned a lot about writing good Scheme code in the process, and I'm very thankful to Arun Isaac in particular for his detailed and insightful explanations about good practices for writing Scheme code. I also really appreciate the Guix community's enthusiastic and helpful comments on this project, which motivated me to keep working on it.

Thank you and Happy Hacking!

Parameterized Packages: The Second Update

2023-08-03T13:37:00Z

This is the third entry in my series of blog posts about adding Parameterized Packages to GNU Guix for my Google Summer of Code project. Parameterization here refers to making it possible for packages to be built with compile-time options, and I have gone over the reasons and benefits for doing so in detail in the first post about Parameterized Packages.

In the last update I talked about implementing support for boolean, non-negative parameters. In this update, I've implemented negation, enumeration and a few other features that will make parameters considerably more powerful!

What's New

A brief summary of additions is

Parametric Variants
Parameter Dependencies
'Negation' for parameter types
Enumerated Types

Parametric Variants

To make global parameters useful, it is necessary to be able to change the contents of a package in ways that Package Transformations might not be able to by themselves. Additionally, users might find themselves in situations where they wish to perform different operations for different values of an enumerated type.

"Parametric Variants" refers to matching against enumerated values and using methods of Defining Package Variants, such as package transforms, modify-inputs and procedures that return packages. Parameters now can use any of these methods instead of just using transformations.

For a better understanding of them, please look at any of the examples using the variant-match macro.

Parameter Dependencies

It's possible that enabling a parameter might require enabling another parameter or a package. For such situations, I've added a new dependencies field to the parameter record that lets users specify parameters or packages a given parameter depends on. You can also fine-tune values for the parameter and the parameters in dependencies.

Negation and Enumeration

Enumeration

In the previous version, parameters could only be in two states: on and off. This version makes it possible for parameters to take multiple states, as long as the user specifies all the possible states. This has a huge number of uses- for example, here's a parameter type for locales:

(define locale-parameter-type
  (parameter-type
   (name 'locale-type)
   (accepted-values
    '(ca_ES cs_CZ da_DK de_DE
            el_GR en_AU en_CA en_GB
            en_US es_AR es_CL es_ES
            es_MX fi_FI fr_BE fr_CA
            fr_CH fr_FR ga_IE it_IT
            ja_JP ko_KR nb_NO nl_NL
            pl_PL pt_PT ro_RO ru_RU
            sv_SE tr_TR uk_UA vi_VN
            zh_CN))
   (negation #f)
   (default 'en_US)
   (description "Type for Locales")))

Explanation
parameter-type is the record type for parameter types. Please note that parameter types are different from parameters. Parameter types are similar to variable types (character, boolean, symbol etc.) and parameters are similar to variables as they possess a type. The fields of this record are
- name which must be a symbol Unlike parameters, parameter-types are identified by the symbol bound by define (here locale-parameter-type) instead of their name field. The name field is purely for the sake of the Guix UI, while the bound name is what is used to utilize a record in a parameter. So to use this record in a parameter record, you would put locale-parameter-type in its type field.
- accepted-values which must be a list of symbols with at least two elements. This represents the entire set of values parameters belonging to this parameter type can take.
- negation which returns the 'negative' element. By default this is the first element of accepted-values, and if it is set to #f like here then negation is not supported for that parameter. It will throw an error if a user tries negating the parameter. Users can negate a parameter in a parameter list by using the #:off keyword:
```
(list ... (parameter-name #:off) ...)
```
  Normally, you would place a value from accepted-values in the right part of the cell, however, #:off gives the 'negative' value for all parameters that support negation.
- default is a value taken by the parameter if the keyword #:default is used on it. This works similar to negation, but you put #:default instead of a #:negation in the right part of the cell. In the case of default, its default value is the second element of accepted-values if negation is not set to #f, and the first element otherwise. In our example, it is set to 'en_US.
- description which provides a description of the parameter type.

Negation

Negation refers to being able to specify the opposite value for a parameter. If it is set to anything but #f for a given parameter-type record, any package record belonging to that parameter-type takes on that value when set to the special keyword #:off. Here's an example that uses it to run the without-tests transform on a package whenever the boolean parameter tests is set to #:off.

(define-global-parameter
  (package-parameter
   (name 'tests)
   (variants
    (parameter-variant-match
     (#:off #:transform (without-tests #:package-name))))
   (description "Toggle for tests")
   (predicate #t)))

Explanation
define-global-parameter is a macro that takes a parameter record and makes it global. This means that it can be referred to in any parameter definition, and that its name is guaranteed to be unique among all global parameters. package-parameter is the record type for parameters. Please note that the record is not called parameter, as it refers to something else entirely in Guile Scheme. The record accepts values for the fields name, type, variants, predicate and description.
- name is a symbol, similar to parameter-type's name field. However unlike parameter-type's name, package-parameter's name is very significant. We do not refer to parameters by any Scheme binding (through something like define or let), as it could lead to unexpected errors in logic validation fields. Parameters are only referred to by their name field, which is always unique for global parameters. If a global and local parameter share names, the local parameter is given preference. If two local parameters with the same name are added, an error will be signaled. The user can hence rest assured that in the context of any given package, each name has a unique meaning.
- type is the parameter-type to use as the basis for the parameter. By default, it is set to boolean which consists of the states on and off.
- variants is an associative list that assigns transforms, procedures, valid build systems etc. to parameter values. This replaces the transforms field from the last post. Users are expected not to write the alist themselves, but to instead use the parameter-variant-match macro that generates an alist based on a specification as seen here. This macro is somewhat similar to the build-system/transform-match macro from the last post. Users can also use parameter-variant if they want to match a single value. #:off matches the 'negative' value for any parameter, and _ matches all non-negative values. It is possible to match multiple values by putting them in a list like (_ #:off). Note that here it would have been possible to use the symbol off instead of the special keyword #:off, as our parameter belongs to the boolean type and its negative value is off. But it is a good idea to use #:off as it always matches against the negative symbol, regardless of the parameter-type's accepted values. Users can also specify the build system the value should match, as seen in the gcc-oflag parameter in the Bonus Examples section. This is not all there is to the magic of parameter-variant-match; to make parameterization more useful, it lets users get the package name, the package and the value of the parameter the statement matched against. These are accessed through keywords, such as the #:package-name keyword in this argument. Have a look at the gcc-oflag and static-lib parameters in Bonus Examples to learn more!
- predicate is set to #f by default. #f means that the given parameter can only be used when its been mentioned in a package's parameter-spec. Setting this to #t means that a global parameter can be applied to packages that do not have it in their spec. This is extremely dangerous and should only be used for extremely generic parameters. Otherwise, the user may set this to any lambda that takes a package record as its argument and returns #t or #f. The global parameter will be applied if this lambda returns #t.
- dependencies is a list of parameters and packages that a given parameter depends on. The list is punctuated by keywords to indicate parameter and package dependencies, with #:parameter and #:package respectively. If no keywords are given, the arguments are assumed to be parameters. Package dependencies have not been implemented yet.
```
(dependencies
`(#:parameter a b ...
  #:package git ,(package (name "some-package") ...) ...))
```
- description is a simple description of the parameter.

What does using parameters look like?

Here is an example use-case for parameterization, which packages Emacs' next, pgtk, xwidgets, wide-int and no-x variants in one package and also makes it possible to mix and match compatible variants.

Usage

The usage format for parameters is the same as that for other package transforms- you specify them through the CLI. In the future, it will also be possible to have a global set of transforms.

guix install emacs-parameterized \
     --with-parameter=emacs-parameterized=pgtk=on \
     --with-parameter=emacs-parameterized=tree-sitter=on \

Underlying Code

EDIT 9/18/2023: as the package record's source field is not thunked, I'm instead using a #:lambda inside the next package parameter to change it.

Under the hood, this is what the implementation looks like.

(package-with-parameters
 [parameter-spec
  (local
    (list
     (package-parameter 
      (name 'next)
      (variants
       (parameter-variant-match
        (_
         #:lambda
         (lambda (pkg)
           (package
            (inherit pkg)
            (version "29.0.92")
            (source
             (origin
              (inherit (package-source pkg))
              (method git-fetch)
              (uri
               (git-reference
                (url "https://git.savannah.gnu.org/git/emacs.git/")
                (commit (string-append "emacs-" version))))
              (file-name
               (git-file-name (package-name pkg) version))
              (patches
               (parameter-if #:package pkg (pgtk)
                             (search-patches
                              "emacs-exec-path.patch"
                              "emacs-fix-scheme-indent-function.patch"
                              "emacs-native-comp-driver-options.patch"
                              "emacs-pgtk-super-key-fix.patch")
                             (search-patches
                              "emacs-exec-path.patch"
                              "emacs-fix-scheme-indent-function.patch"
                              "emacs-native-comp-driver-options.patch")))
              (sha256
               (base32
         "1h3p325859svcy43iv7wr27dp68049j9d44jq5akcynqdkxz4jjn"))))))))))
     (package-parameter (name 'tree-sitter)
                        (dependencies '(next)))
     (package-parameter
      (name 'pgtk)
      (variants
       (parameter-variant-match
        (_ #:transform (with-configure-flag
                        #:package-name "=--with-pgtk"))))
      (dependencies '(tree-sitter x11)))
     (package-parameter
      (name 'xwidgets)
      (variants
       (parameter-variant-match
        (_ #:transform (with-configure-flag
                        #:package-name "=--with-xwidgets")))))
     (package-parameter
      (name 'wide-int)
      (variants
       (parameter-variant-match
        (_ #:transform (with-configure-flag
                        #:package-name "=--with-wide-int")))))))
  (one-of '((_ (x11 #:off) pgtk)
            (_ (x11 #:off) xwidgets)))]
 (inherit emacs)
 (name "emacs-parameterized")
 (arguments
  (parameter-substitute-keyword-arguments 
   (package-arguments emacs)
   [((x11 #:off))
    '(((#:configure-flags flags #~'())
       #~(delete "--with-cairo" #$flags))
      ((#:modules _) (%emacs-modules build-system))
      ((#:phases phases)
       #~(modify-phases #$phases
                        (delete 'restore-emacs-pdmp)
                        (delete 'strip-double-wrap))))]
   [(#:all (xwidgets on) (pgtk #:off))
    '(((#:configure-flags flags #~'())
       #~(cons "--with-xwidgets" #$flags))
      ((#:modules _) (%emacs-modules build-system))
      ((#:phases phases)
       #~(modify-phases #$phases
                        (delete 'restore-emacs-pdmp)
                        (delete 'strip-double-wrap))))]))
 (inputs
  (parameter-modify-inputs
   [(next) (prepend sqlite)]
   [(tree-sitter) (prepend tree-sitter)]
   [(xwidgets) (prepend gsettings-desktop-schemas
                        webkitgtk-with-libsoup2)]
   [((x11 #:off))
    (delete "libx11" "gtk+" "libxft" "libtiff" "giflib" "libjpeg"
            "imagemagick" "libpng" "librsvg" "libxpm" "libice" "libsm"
            "cairo" "pango" "harfbuzz" "libotf" "m17n-lib" "dbus")])))

Step-by-step Explanation

package-with-parameters: This macro takes a parameter-spec as its first argument and applies the parameter specification to the package in its body. The default parameters are then activated within the package.
parameter-spec: This record type contains all of the logic necessary to declare and resolve parameters for a package. This normally goes inside the properties field of the package record. In the previous post, it was necessary to put this record inside the properties, but now package-with-parameters handles that for us. The parameter specification record contains various fields, all of which are optional. I have gone over the fields in detail in the previous blog post, hence I will not explain all of them in detail here. The only big change is that one-of now has a functionality wherein if you start a list within it with _, you can have a case where none of the values in it are positive. Otherwise, it throws an error as one and only one value is expected to be positive. Also notice the usage of #:off to indicate negation. We have also not declared x11, which will hence be treated as a global parameter. In general global parameters must either have their predicate set to something that returns #t or be present anywhere in the parameter-spec to be applicable. Users are advised to put them in the optional field, as it was created with this use case in mind.
local parameter next: The local parameter next has an interesting #:lambda statement inside it. This statement takes the current package record as an argument and returns a new package record, which is then used in its place. This is an extremely powerful method for changing otherwise unchangeable options, such as the ones here; because the version and source fields are not thunked, this is the only way of modifying them. The #:lambda's functions can be passed 0, 1 or 2 arguments. In the case that it asks for 1 argument, the current package record is passed. If it asks for 2, the current package record along with the given parameter's value are passed. In all cases, the function is expected to return a package record that will then be taken as the new current package record.
package body: Within the package body, we have the usual fields you would expect. (inherit emacs) signifies that this package inherits all of emacs' base fields, and the rest of the fields are overrides of that. Please note that the name field cannot be influenced by parameters as it is not thunked.
parameter-match: note: this has been substituted by parameter-substitute-keyword-arguments in the latest edit This is an extremely useful macro that matches all the parameter lists that has any positive parameters. It is also possible to require all the parameters in a list to be positive by using #:all. Please keep in mind that it does not short-circuit by default like cond. It will keep matching parameters until all the lists have been combed through. A short-circuiting version exists in the form of parameter-match-case. I've gone over the functionality offered by this macro in detail in the previous blog post, however it has one small improvement: all conditionals now support checking if a parameter is set to a particular value instead of just checking if it is positive or not. This is very useful for enumerated types, where you might for example want to disable some features if and only if a parameter is set to the second positive value. To illustrate this, if you wanted to check whether a parameter y is set to v1 or if the parameter list z is non-negative, the list would be ((y v1) z). You can also use this to check for the default or negative value, with (parameter-name #:default) and (parameter-name #:off) respectively. _ is a similar special symbol which matches all non-negative values, but it is not necessary to use it since the parameter name by itself, say parameter-name is the same as (parameter-name _). We can see this in the ((y v1) z) example above, where z is matching all non-negative values of z even though we did not specify it as (z _). You can also use this in all of the fields in parameter-spec that require you to specify parameters. The parameter value list syntax is the same everywhere.
parameter-substitute-keyword-arguments and parameter-modify-inputs: As explained in the previous blog post, because modify-inputs is a macro, we cannot use parameter-match within it. To counter this, I have written parameter-modify-inputs that behaves a lot like if modify-inputs had a parameter-match macro within it. I have similarly written parameter-substitute-keyword-arguments for another commonly used macro called substitute-keyword-arguments.

I have gone over the rest of the conditionals in the previous blog post too, they remain more or less the same with the exception that we use #:all inside lists instead of all like last time. This is to make it obvious at a glance that #:all is not a parameter like the rest of the list.

Bonus Examples

Here are some bonus examples for enumerated parameters:

GCC Optimization Flags

gcc has a set of optimization flags that can be used to make programs faster or smaller at the expense of stability. This is a very basic attempt at adding that functionality to the gnu-build-system through the CFLAGS make-flag.

(package-parameter
 (name 'gcc-oflag)
 (type
  (parameter-type
   (name '_)
   (accepted-values '(-O0 -O1 -O2 -O3 -Os -Ofast -Og -Oz))
   (negation #f)))
 (variants
  (parameter-variant-match
   (_ #:build-system gnu-build-system
      #:lambda
      (lambda (pkg parameter-value)
        (package
         (inherit pkg)
         (arguments
          (substitute-keyword-arguments
           (package-arguments package)
           ((#:make-flags flags #~'())
            #~(append
               #$flags
               (list (string-append "CFLAGS="
                                    parameter-value))))))))))))

Static Libraries

In High-Performance Computing, it's often necessary to produce static builds of packages to share them with others. This parameter is a basic attempt at making it possible to do so with any given library.

(package-parameter
 (name 'static-lib)
 (variants
  (parameter-variant-match
   (_ #:transform
      (with-configure-flag #:package-name "=--disable-shared")
      (with-configure-flag #:package-name "=--enable-static")))))

Sneak Peak: A RESTful API for Parameterization

I recently made a post on Mastodon that claimed that the real advantage of Guix is that it's extensible with Guile Scheme. To back up this claim, once parameters have been merged to trunk I'll be writing a set of tutorials on hacking Guix with Guile Scheme. One of these planned tutorials is going to be about writing a RESTful API using Guile that'll allow users to request a package with specific parameters. Here is what the POST request for this API may look like:

POST /test HTTP/1.1
Host: guix.example
Accept: application/json
Content-Type: application/json
Content-Length: 194

{
  "User" : "guix-hacker",
  "Package" : "emacs",
  "Parameters" : [
      { "Parameter" : "next",
        "Value" : "on"},
      { "Parameter" : "tree-sitter",
        "Value" : "off"}
      ]
}

Future work

Here I have demonstrated a basic DSL that is more-or-less just S-expressions. There is however scope for making it a lot more convenient to use parameters, and thus there are plans on building a convenience syntax on top of this simple one. One example is using ~parameter-name to indicate the negation of a parameter. However, syntax like this may not be obvious to everyone at a glance, which is why we have decided to make a convenience DSL with these features only after heavy deliberation and discussion.

The next few updates will focus on the UI for Parameterization. The primary goals for the UI are to make it easy to discover parameterization options, tell what type a parameter is and to figure out parameter combinations that work for a given package.

Closing Thoughts

As can be seen with the Parameterized Emacs example in this post, parameterization will make it possible to join a large number of variations of packages and reduce the amount of code requiring maintenance. One of the aims of this project is to also create procedures that test parameter combinations and measure the combinatorial complexity brought about by parameterization, which should make testing parameteric variants easy too.

I expect parameterization to be particularly useful for running Guix on exotic hardware (such as static minimalistic targets) or on High-Performance Computing Systems (specific architecture optimizations) and make it generally easy to tailor a lot of packages for a particular system's requirements.

This update marks the completion of this Google Summer of Code project's midterms. I'd like to thank my mentors Pjotr Prins and Gábor Boskovit as well as Ludovic Courtès, Arun Isaac and Efraim Flashner for their guidance and help, without which I don't think I'd have been able to reach this milestone. I'm also very grateful to the many wonderful people in the Guix community that provided me with a lot of useful advice and suggestions.

Stay tuned for updates, and happy hacking!

Parameterized Packages: An Update

2023-06-22T13:40:00Z

I started work on adding Parameterized Packages to GNU Guix 3 weeks ago, and this post is a short status update on the things I've done so far.

1. Summary of additions

I have written

Record types for parameters
Processors for boolean, non-negative parameters
Macros for using parameters inside package definitions

I started my work by writing draft parsers that acted on S-expressions, before moving on to records. The parameterization process follows 3 phases:

Reading values from the parameter-spec record
Resolving the values against user-input, and returning a final parameter list
Applying transforms and macros in accordance with the final parameter list

The parsers can currently resolve boolean and non-negative parameters. This means that parameters can either be on or off, and the parser does not automatically assume that parameter! implies (not parameter). I am now working on adding support for enumerable parameters (i.e. parameters with multiple values which are not just on and off) and adding support for negation of boolean parameters. I next wrote Guix-style record types for parameters, parameter types and parameter specification. I also wrote sanitizers and macros to make it convenient to input values in these records, as the code otherwise gets a bit boilerplate-y. I then ported the previously written parsers to these newly made records, and also wrote a few macros that let you use parameters inside special if and cond-style expressions inside the package definition to have conditional options for fields depending on if one or all of the required parameters are switched on. Next I wrote a macro package-with-parameters that returns a package definition with the default parameter transforms applied. This means that if by default my package uses two parameters a and b, the macro will return the package with both their parameter transforms applied to it. Lastly, I wrote a modified modify-inputs macro that accepts parameters as conditions for the traditional modify inputs actions.

2. Illustrative example

EDIT 9/18/2023: The example has been updated to match the latest syntax. The following code builds Guile from the git source. Please keep in mind that the syntax is still heavily subject to change, however the underlying mechanisms will remain consistent.

(define-public guile-parameterized
  (package-with-parameters
   (parameter-spec
    (local
        (list
         (package-parameter
          (name "git")
          (variant 
           (parameter-variant-match
            (_ (with-git-url #:package-name
                             "https://git.savannah.gnu.org/git/guile.git")))))))
    (defaults '((git off))))
   (inherit guile-3.0)
   (name "guile-parameterized")
   (inputs (parameter-modify-inputs (package-inputs guile-3.0)
                                    (((git on)) (append autoconf)
                                     (append automake)
                                     (append libtool)
                                     (append gnu-gettext)
                                     (append flex)
                                     (append gperf)
                                     (append texinfo))))
   (arguments (append
               (parameter-if ((git on))
                             (list #:make-flags #~'("VERBOSE=1"))
                             '())
               (package-arguments guile-3.0)))))

The package-with-parameters macro takes in a parameter-spec and a package definition and gets the default parameter alist from it, and applies transformations according to it. Here we're defining a version of guile that inherits guile code and adds a parameter-spec property. Inside the parameter-spec definition, we start by defining local parameters. These parameters are only available to the package, and override any global parameters with the same name. This overriding feature will make parameter specifications resilient to globalization of previously local parameters, and also make it possible to globally declare what local parameters you'd like to access across packages. The name field of the parameter record accepts both strings and symbols, but when referring to a parameter in other fields one must use a symbol. The variants field is explained in further detail in the next blog post. The transforms follow the usual cons cell syntax used when defining package variants. Next we have the defaults field, which takes a list of parameters which are expected to be switched on by default. Some fields of the parameter-spec record omitted from this example are:

optional, for declaring global parameters that can optionally be used
one-of, which is a list of lists of parameters of which only one can be used per list
required, which is a list of parameters that are absolutely required. It exists mostly for global parameters
canonical, which contains canonical combinations, a proposed feature for solving the substitute problem
parameter-alist, (not meant to be modified by the user) contains the final list of active parameters and their values, on or off.

These all come together to make it possible to define an arbitrary combination of parameters in arbitrary states and test them against the parameter-spec to see if they work and apply them if they do. The package-with-parameters macro is proof of this working, it calculates transforms pertaining to default values and applies them to the package record defined inside it based on the contents of the parameter-spec.

3. Parametric Conditionals

I have written a number of conditional macros that check if a given parameter is set to on in the parameter-alist and update the package record appropriately. parameter-if and parameter-modify-inputs have been used in the example above, and below is an explanation of how they work:

3.1. parameter-if

parameter-if takes a parameter or a list of parameters and checks if any of them are on. If they are, it returns the first expression, but if all of them are off, it returns either nothing or the second expression. It behaves similarly to Guile's if macro. It is being used in this snippet from the guile-parameterized example:

;; inside package definition
(arguments (append
            (parameter-if ((git on))
                          (list #:make-flags #~'("VERBOSE=1"))
                          '())
            (package-arguments guile-3.0)))

Here, the arguments field is given a list formed by appending #:make-flags with the value "VERBOSE=1" if the parameter git is set to on, or appending an empty list '() otherwise.

3.2. parameter-match

parameter-match is somewhat similar to Guile's cond, but also very different. It takes in a set of lists of the form ((parameters ...) clauses ...), wherein if any in the list of parameters is set to on, the clauses are executed. This behavior is not short-circuiting, and the other lists are checked once one is evaluated regardless of the result. A list may be prefixed with #:all if all parameters are required to be switched on. Also, a _ can be used to always match. For example, the parameter-if example above can be rewritten with parameter-match like this:

(arguments (append
            (parameter-match
             (((git on)) (list #:make-flags #~'("VERBOSE=1")))
             (_ '()))
            (package-arguments guile-3.0)))

parameter-match has a variant called parameter-match-case which is the same as parameter/match, but it short-circuits when a matching list is found.

3.3. parameter/modify-inputs

The modify-inputs macro is used very frequently when defining package variants, but due to it being a macro we cannot use parameter-match inside it to pick arguments. To this end, I have defined a new macro called parameter-modify-inputs that takes in a list of parameters and a corresponding list of arguments to modify-inputs that can be used instead of it. _ can be used to always execute the clauses, and #:all may be used to require all parameters to be positive. In the example package above, it has been used like this:

;; inside the package definition
(parameter-modify-inputs (package-inputs guile-3.0)
                                    (((git on)) (append autoconf)
                                     (append automake)
                                     (append libtool)
                                     (append gnu-gettext)
                                     (append flex)
                                     (append gperf)
                                     (append texinfo)))

Here, if the parameter git is switched on, autoconf, automake, libtool, gnu-gettext, flex, gperf and texinfo are added to the package's inputs. This is quite useful as these inputs are required for building guile from its git source.

4. Global Parameters

The handling of global parameters is an important topic that needs more discussion. Right now, the idea is to require all global parameters to be defined in one file and to access them through a hash-table called %global-parameters. This is to ensure that global parameter symbols are both short and unique. To make the process of adding values to this hash-table easier, I've written a macro called define-global-parameter that takes a parameter definition and makes it global. For example, if I wanted to define a global parameter that disables tests for guile-3.0, I can do it like this:

(define-global-parameter
  (package-parameter
    (name "guile-3.0-tests!")
    (description "Disables tests for Guile 3.0")
    (variants
     (parameter-variant-match
      (_ (without-tests . "guile-3.0"))))))

Now any package that uses this global transform will have guile-3.0's tests disabled.

5. Results

It is now possible to define a package with parameters and change the parameter-alist to use the parameters. Next, I'll be working on parsing negated and enumerated parameters, along with adding support for modify-inputs and package-rewriting in the parameter record itself.

EDIT: This work has been done, and can be seen in the next blog post.

Stay tuned for updates, and happy hacking!

Running a Patched DWM on GNU Guix

2023-06-12T13:37:00Z

If like me you use a lot of patches with dwm, you probably use dwm-flexipatch for configuring dwm. You probably also have an existing build of dwm with a lot of customizations in config.def.h. Research online might lead you to the conclusion that you need to set up a git repository and a guix channel with your dwm folder to make it work, but that's not necessary at all. In this post I'll go over how I got my dwm-flexipatch build to work on GNU Guix without any of that.

We want to start by making a directory for our dwm folder and build file.

mkdir DWM_MODULE

Next, copy your dwm folder here.

cp -r /path/to/dwm DWM_MODULE/

Now, this step is very important: run make clean once in the dwm folder, otherwise guix will run into a RUNPATH error during the build phase.

cd DWM_MODULE/dwm
make clean
cd ..

And finally, inside the DWM_MODULE directory create a file named USER-dwm.scm, where USER can be your username or anything else. Then put this in the file:

(define-module (USER-dwm))
(use-modules (gnu packages suckless)
             (gnu packages xorg)
             (gnu packages image)
             (guix gexp)
             (guix packages))

(define-public USER-dwm
  (package
    (inherit dwm)
    (name "USER-dwm")
    (inputs (modify-inputs (package-inputs dwm)
              (append libxcb)
              (append imlib2)))
    (source (local-file "dwm" #:recursive? #t))))

USER-dwm

Here I have added libxcb and imlib2 to inputs as I have patches requiring them. If you are using a patch requiring some other library, you can add it by putting another (append libname) statement in the modify-inputs expression. The source expression searches for a folder named dwm in the file's pwd and uses its contents as the package's source. Running guix package -f USER-dwm.scm will now install your flexipatched dwm. If you want to add this to your system configuration, modify your config.scm like this:

(use-modules (USER-dwm) ...)

(operating-system
  ...
  (packages (list USER-dwm ...)))

And now, reconfigure the system with the following command:

guix system reconfigure /path/to/config.scm -L $HOME/DWM_MODULE

If you don't want to run the reconfigure command with -L every time, add $HOME/DWM_MODULE to GUIX_PACKAGE_PATH.

Here's a customary screencap:

An Introduction to Parameterized Packages

2023-05-09T02:40:00Z

I will be working on bringing Parameterized Packages to GNU Guix this summer as a part of the Google Summer of Code program under the mentorship of Pjotr Prins and Gábor Boskovits. This post will go over the basic ideas behind Parameterized Packages, their benefits and how I am planning on implementing them.

What are Parameterized Packages?

Parameterized packages will provide users with a simple way to configure many aspects of packages, à la Gentoo USE flags.

GNU Guix is the only GNU/Linux distribution capable of achieving a full-source bootstrap, and this comes with many unique advantages. Every package in Guix can be built from source, and as a result it is possible to configure a wide variety of options for each package that just aren't exposed on binary-based distributions using Package Transformations. While package transformations are extremely powerful, they require some experience with packaging software for Guix and are generally expected to be used by power users for applications such as High-Performance Computing.

Parameterized Packages aim to not only bring the benefits of package transformations to all users, but to also make it possible to globally specify some aspects to include or exclude from packages similar to Gentoo's USE flags.

Some of the benefits of Parameterized Packages are:

Significantly smaller binaries
More fine-grained control over the entire system
Access to additional features only accessible through compile-time options

Among many more.

Example Usage

(Edited - 15th Sep 2023) To avoid confusion, I have removed the example from this section. Owing to community feedback, the syntax for parameterization has changed a lot from the original draft. To see a concrete example of parameters in action, please see the second update.

What if a package record does not contain the parameter value?

In the instance that parameter values are not specified, the package will be used in its default state by all of the packages depending on it. In general, parameters propogate to dependencies if a valid configuration can be acheived with them, and if this is not possible the default state of the package is used. This will help with the gradual adoption of package parameters, as not every package will have to specify parameters and at the same time the packages specifying parameters will be able to use them even if their dependencies do not have the given parameter.

One awesome feature of this arrangement is that a user could have two packages with conflicting parameters but they would both work on the system thanks to Guix building both versions of dependencies. This would not work on imperative package managers with similar functionality. For example if a music player application depends on mpd built with pulseaudio while another depends on it built with jack, Guix will create two versions of mpd, one built with each so that both of these packages may coexist.

More on parameter symbols

Fairly generic options such as x11, gcc-oflag or locale will be accepted as global parameters. Most global parameters will have pre-defined standard transforms for every build system they are valid for. Note that having too many parameters in a given package's definition will lead to combinatorial explosion of states, and thus it is best to limit the number of parameters to a manageable amount. I will be attempting to add 10~20 global parameters after finishing this project.

Conclusion

Parameterized Packages have the potential to add more functionality to GNU Guix for all users, however they will require the feedback and support of the entire Guix community. I would immensely appreciate any kind of suggestions and comments in the new thread on Parameterized Packages which can be found in the mailing list, especially suggestions on what parameter symbols users would like to use.

Happy Hacking!

The First Week of January 2023

2023-01-10T13:37:00Z

I decided to start writing a blog to talk about my ventures in programming and system administration to document my progress and motivate me to do more. This entry covers what I did in the first week of January, 2023.

Torus Wrapper

One of the projects I poured a bunch of time into was Torus Wrapper, which was an effort to wrap transparent PNGs into tiles that can be tiled to make tiling wallpapers, tiling game assets, et cetera. A high-level overview of how this is done is that the program takes in a transparent png, finds the biggest rectangle inside it and then wraps the parts not fitting in the rectangle atop it. Here's an example of a tile created with it:

Lots of Websites

I worked on a lot of websites during this break; my personal site, BITS SOS' site and the Journal Club's new website. I used static site generators to build BITS SOS and JC's sites, as they provide a simple way to theme huge websites; however, these are not free of hiccups. Themes often come configured wrong and can cause CSS and images to not load if the paths are not set right. Luckily, there were no other major problems setting up either of them with HUGO. For my personal site, I wrote the main site in ClojureScript (with Reagent) and the blog site in Frog. The blog (this site) is based on Racket lang's blog.

Tiny Improvements to my R6RS Infix Calculator

I abstracted away a lot of the recursion in my R6RS infix expression calculator by defining a new macro to handle functions that iterate over a carry-forwarded list and adding the Clojure thread-first macro (->). I planned on implementing something called the Nestedness Theorem, which is that the rightmost open paren is always safe to evaluate. However, my current approach is more-or-less similar to this, hence I did not rewrite it. I also added additional error checking; the program now checks if parens are balanced.

Adding a DDNS to my Home Server

I was previously connecting to my home server through a static IP, however I realized that my usage needs could be met with a dynamic IP using a dynamic DNS. I thus set up a DDNS on the server and bound its internal IP to ensure that I could boot into it after it rebooted. Home servers are excellent for maintaining private remote project repositories, hosting messenger bots or services like IRC bouncers and testing out dynamic websites.