- s390xbl0v3's blogZola2025-01-01T00:00:00+00:00https://bl0v3.com/tags/s390x/atom.xmlPorting NixOS to IBM mainframes2025-01-01T00:00:00+00:002025-01-01T00:00:00+00:00
bl0v3
https://bl0v3.com/Blog/porting-nixos-to-an-ibm-mainframe/<img class="no-hover"alt="todo_mainframe_and_nix_logo.jpg"src="TODO.jpg"/>
<p><strong>TODOS:</strong></p>
<ul>
<li>link to neofetch software about.txt or sth</li>
<li>maybe include a picture of the neofetch</li>
<li>show how to make containers/tarballs without kernel</li>
<li>show how to run nixos-install</li>
<li>show how the iso thing works</li>
<li>at some point mention that “lol emulators cool but you will see
that they aint the same shit” (dasd ccwgroup)</li>
<li>show like an abstract from real mainframe configuration (the one on paper)
and how to write them in hercules</li>
</ul>
<h1 id="what-does-this-article-address">What does this article address?</h1>
<p>Like said previously this article documents part of the process I underwent
to get NixOS running on IBM’s mainframe architecture.</p>
<p>While it obviously is somewhat focused on the s390x/Z architecture. A few
concepts/technologies/tricks found here can of course be either partially
or fully applied to the process on other architectures. These roughly are the
points covered:</p>
<p><strong>TODO clean up, order correctly, give enough insight up here to let
people wishing to only look at a specific section find it</strong></p>
<p>-all</p>
<p>-the</p>
<p>-bootstrap</p>
<p>-shit</p>
<p><strong>TODO sum divider</strong></p>
<p>kernel/os/boot … etc … shit:</p>
<ul>
<li>cross compiling a nixos rootfs tarball and booting this up <strong>TODO maybe also test binfmt</strong></li>
<li>alpine kexec</li>
<li>building iso</li>
<li>zipl (usage , packaging and linker section fun)</li>
<li>qemu ( focus on differences in hw)</li>
<li>getting nix to run on alpine</li>
<li>installing nixos without nixos</li>
<li>bootstrap process</li>
<li>various obscure package fixes</li>
<li>cross compilation of nixos systems</li>
</ul>
<blockquote class="note">
<p class="alert-title">
<i class="icon"></i>Note</p>
<p>This article may be perceived as rather long. Keep in mind that it does
cover quite a few things not all of these are essential.</p>
</blockquote>
<blockquote class="important">
<p class="alert-title">
<i class="icon"></i>Important</p>
<p>Some sections are specific to problems I faced due to the setup
I was doing this in and other than <em>oh hey I didn’t know that</em> may
not really provide you with a lot of actually relevant information.</p>
</blockquote>
<h1 id="the-situation">The situation:</h1>
<p>Well where do I start … I’m not even quite sure anymore what motivated me to give this a shot but it more or less boils down to
these 3 points:</p>
<ul>
<li>
<p>a mainframe somehow managed to appear in front of me … <small>(I guess that doesn’t happen too often)</small>
<img class="no-hover"alt="me with a mainframe"src="https://files.catbox.moe/7qze3s.jpg"/></p>
</li>
<li>
<p>I wanted to add a platform to Nix(OS) and see how things work <em>“from the ground up”</em> <small>(but nearly everything was already added even risc-v)</small></p>
</li>
</ul>
<p>I thought it would be funny to use it to post in in some sort of desktop thread with this <small>but lying ain’t cool
and it actually running there may make it even more funny (or at least I thought so)</small>
<img class="no-hover"alt="first_neofetch_s390x_showcase_screencap"src="https://files.catbox.moe/v08l22.jpg"/></p>
<p>I quickly came to the realisation that unlike initially assumed “just running NixOS” (and consequently neofetch) wasn’t as simple as I thought it would be as:</p>
<ul>
<li>
<p>there are no premade s390x isos/containers/…</p>
</li>
<li>
<p>not even the bootstrap tools were available <em>(thus no <code>pkgs.stdenv {}</code>)</em></p>
</li>
<li>
<p><a href="https://en.wikipedia.org/wiki/IBM_System/390">s390(x)</a>/<a href="https://en.wikipedia.org/wiki/Z/Architecture">z/Architecture</a> is an <a href="https://en.wikipedia.org/wiki/Endianness">big endian</a> architecture</p>
</li>
<li>
<p>emulators(<a href="https://en.wikipedia.org/wiki/Binfmt_misc">binfmt</a>) are not only broken
but also insanely slow and don’t scale too well across multiple threads/cores</p>
</li>
<li>
<p>cross compilation is a pain but you somehow need to build some sort of bootstrap/dev system before you can actually run/build
things on s390x itself. <small>(as nobody bothered to build them before)</small></p>
</li>
</ul>
<p>Though with the help of various people (folks working at IBM , friends and a bunch of people involved with NixOS). A little patience and
having spend your teenage years running linux on various cursed platforms ….. these challenges were overcome :D</p>
<h2 id="where-to-start">where to start?</h2>
<p>In order to get things running on an s390x host you obviously need to compile your software for the
s390x architecture/target. So you either:</p>
<ul>
<li>
<p>run your compiler/buildenv natively
<small>(this assumes that you have these already for the s390x architecture)</small> but as we don’t have
these we can’t.</p>
</li>
<li>
<p>you cross-compile from another architecture such as x86 <small>(this did partially work but
is known to cause issues in general)</small></p>
</li>
</ul>
<p><strong>TODO: include a small showcase on how to cross compile things on nix</strong></p>
<p>One of the things that makes cross compiling rather hard is that you can’t really
execute the binaries you just build. This usually is not a problem and to some degree
nix even accounts for this.</p>
<p><strong>TODO explain how nix differenciates various buildInputs = specifications</strong>.</p>
<p>Sadly at times there is software that <strong>(often even independently of nix!)</strong> is either very hard to cross compile
OR upright not designed to <small>( … like <code>zipl</code> the bootloader we gonna have to use eventually)</small>.
Thus unless we get native compilation working we can use “<em>the next best thing</em>”:</p>
<h4 id="getting-a-non-s390x-nixos-host-to-run-s390x-binaries">getting a non s390x NixOS host to run s390x binaries:</h4>
<p>I expected this to work out of the box but it didn’t, luckily the issue was easy to overcome
and this ended up being a rather quick fix: <a href="https://github.com/NixOS/nixpkgs/pull/327665">added s390x option type via magic attributes #327665</a>.</p>
<p>As all that had to be done was more or less
adding the <a href="https://github.com/qemu/qemu/blob/b69801dd6b1eb4d107f7c2f643adf0a4e3ec9124/scripts/qemu-binfmt-conf.sh#L95">magic extension sequence</a> to get the kernel to forward execution via <a href="https://en.wikipedia.org/wiki/Binfmt_misc">binfmt</a> of all s390x binaries in question to qemu-s390x.</p>
<p><strong>(TODO short showcase of how to enable binfmt with this fix. How to make a nixos system
“belive” it supports this arch. And show like a binaries headers of idk gnu hello and how we
run it on sth that <code>/proc/cpuinfo</code> denotes as lets say x86)</strong></p>
<h2 id="moving-beyond-binfmt">moving beyond binfmt</h2>
<p>After a bunch of crude hacks, overlays and disabled tests I managed to cross-compile the first nixos-s390x rootfs tarball.</p>
<p><strong>TODO: share it here + share the drvs needed to do so</strong></p>
<p><strong>TODO show neofetch and explain how one would build and export a nix system in tarball format</strong></p>
<p>While on first glance this looked promising. As expected when running an operation such as <code>nix-shell -p</code> or whatever deals with .drv’s that aren’t already known by hash to nix would fail.</p>
<p><strong>TODO show the actual error maybe</strong></p>
<p>Since I guess nobody before me bothered giving this a try <small>(maybe for a good reason)</small> nix wasn’t
properly aware of how to actually build “literally anything” on s390x for s390x. This is due to the bootstrap package
missing and consequently not being able to build <code>stdenv</code>/<code>pkgs.stdenv</code>.</p>
<blockquote class="note">
<p class="alert-title">
<i class="icon"></i>Note</p>
<p>Commands like <code>nixos-rebuild</code> or <code>nixos-install</code> won’t work
without the means (<code>stdenv</code>) to compile stuff locally either.</p>
</blockquote>
<p>So either one has to accept being unable to build pretty much anything on s390x
itself and stick to cross compilation.</p>
<p>Or alternatively we <strong>get stdenv working!</strong>.</p>
<h3 id="bootstrap-tools-and-overcoming-infinite-regression">“bootstrap tools” and overcoming infinite regression</h3>
<p>When building a system from the ground up <strong>(independently of nix)</strong> it usually goes something like <q>you need a compiler to build a compiler to build a libc to build a compiler that builds a compiler ….</q>. But where would
that <em>“first compiler”</em> come from?</p>
<p>The problem presents itself a little bit similar to the chicken and egg
problem as in both cases at the core its <a href="https://en.wikipedia.org/wiki/Infinite_regress">infinite regress</a>
that we need to overcome. Luckily in our case it can be solved but first we should establish some terminology:</p>
<h3 id="what-exactly-are-bootstrap-tools">what exactly are “bootstrap tools”?</h3>
<p>With nix we don’t exactly start with just a compiler, we actually ship multiple executables.</p>
<p><strong>TODO actually build the thing and just show an abstract raw from the terminal</strong></p>
<p><strong>TODO give https://trofi.github.io/posts/240-nixpkgs-bootstrap-intro.html some credit</strong></p>
<p><strong>TODO look at the most current drv as well at https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/linux/make-bootstrap-tools.nix</strong></p>
<p>These roughly look like:</p>
<p><strong>TODO do we want to talk about how its linked esp when trofi already did it?</strong></p>
<pre class="z-code"><code><span class="z-text z-plain">busybox (statically linked against musl)
</span><span class="z-text z-plain">gcc (this package and below are dynamically linked against glibc)
</span><span class="z-text z-plain">glibc binutils coreutils tar bash findutils diffutils sed grep awk
</span><span class="z-text z-plain">gzip bzip patch patchelf gmp mpfr mpc zlib isl libelf
</span></code></pre>
<p>The collection of these utilities is what one refers to in this context when talking about “bootstrap tools”.</p>
<p><strong>TODO talk about how on nix its very important that this is pure and why shipping lets say busybox makes
a lot of sense instead of using the system coreutils or pretty much anything from the system</strong></p>
<h3 id="how-to-obtain-bootstrap-tools">how to obtain “bootstrap tools”</h3>
<p>While there are actually 2 ways to go on about obtaining/building these “bootstrap tools”, I will focus on the
approach which cross compiles them like seen with risc-v for example.</p>
<blockquote class="note">
<p class="alert-title">
<i class="icon"></i>Note</p>
<p>For good reasons its very uncommon to build these “bootstrap tools” and
unless the architecture lacks support there is no need to do so.</p>
<p><strong>TODO: maybe show the error nix throws if the bootstrap tools don’t exist yet</strong></p>
<p><strong>TODO fact check: I think unless you run the maintainer script nix will not build
them and just give up upon finding out that they aren’t hosted anywhere as at some point
we declare them with “=” for the arch so It should trigger some sort of attribute not found
error or sth instead of attempting to build or complaining the drv lacks</strong></p>
</blockquote>
<p>To get nix to build these bootstrap tools a few things need to be done before we can proceed:</p>
<ol>
<li>describe the target in <a href="https://github.com/NixOS/nixpkgs/blob/master/lib/systems/examples.nix">lib/systems/examples.nix</a> …
<small>this was actually the only thing already done and only requires like one line</small>
Taking a look at the entries here for lets say m68k , s390 as well as s390x</li>
</ol>
<pre data-lang="nix" class="language-nix z-code"><code class="language-nix" data-lang="nix"><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span>
</span><span class="z-source z-nix"> <span class="z-comment z-line z-number-sign z-nix"># ...</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">m68k</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">config</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">"</span>m68k-unknown-linux-gnu<span class="z-punctuation z-definition z-string z-double z-end z-nix">"</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix">
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">s390</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">config</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">"</span>s390-unknown-linux-gnu<span class="z-punctuation z-definition z-string z-double z-end z-nix">"</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix">
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">s390x</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">config</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">"</span>s390x-unknown-linux-gnu<span class="z-punctuation z-definition z-string z-double z-end z-nix">"</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"> <span class="z-comment z-line z-number-sign z-nix"># ...</span>
</span><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset z-nix">}</span>
</span></code></pre>
<p>one sort of gets the point. Some “systems” are more involved though providing variables for libc/gcc(fpu+arch)/rustc/xcode/…
luckily we don’t need to do that here.
<small>there are still some platform specific things to take care of as you will see later</small></p>
<ol start="2">
<li>
<p>with <a href="https://github.com/NixOS/nixpkgs/blob/master/lib/systems/examples.nix">lib/systems/examples.nix</a> from above now providing us with <code>pkgsCross.s390x</code> we can make the adequate change to <code>pkgs/stdenv/linux/make-bootstrap-tools-cross.nix</code> via <a href="https://github.com/NixOS/nixpkgs/pull/327715">bootstrap-tools-cross: add s390x to make-bootstrap-tools-cross.nix #327715</a> which should get NixOS’s CI “hydra” actually attempt to build this</p>
</li>
<li>
<p>Modifying <code>CROSS_TARGETS</code> in <a href="https://github.com/NixOS/nixpkgs/commit/313fc26d22d34e0dfb9c0922b3a687d03452199f">maintainers/scripts/bootstrap-files/refresh-tarballs.bash</a> to include <code>"s390x-unknown-linux-gnu"</code>. As running this script will provide us with the urls/hashes and some metadata about these bootstrap files. This relies on 1) & 2) of course. The end result looks something like this:</p>
</li>
</ol>
<pre data-lang="nix" class="language-nix z-code"><code class="language-nix" data-lang="nix"><span class="z-source z-nix"><span class="z-comment z-line z-number-sign z-nix"># Autogenerated by maintainers/scripts/bootstrap-files/refresh-tarballs.bash as:</span>
</span><span class="z-source z-nix"><span class="z-comment z-line z-number-sign z-nix"># $ ./refresh-tarballs.bash --targets=s390x-unknown-linux-gnu</span>
</span><span class="z-source z-nix"><span class="z-comment z-line z-number-sign z-nix">#</span>
</span><span class="z-source z-nix"><span class="z-comment z-line z-number-sign z-nix"># Metadata:</span>
</span><span class="z-source z-nix"><span class="z-comment z-line z-number-sign z-nix"># - nixpkgs revision: 8ba481d65eb21a4f9e6b1e812de3f83079eb8016</span>
</span><span class="z-source z-nix"><span class="z-comment z-line z-number-sign z-nix"># - hydra build: https://hydra.nixos.org/job/nixpkgs/cross-trunk/bootstrapTools.s390x-unknown-linux-gnu.build/latest</span>
</span><span class="z-source z-nix"><span class="z-comment z-line z-number-sign z-nix"># - resolved hydra build: https://hydra.nixos.org/build/267960435</span>
</span><span class="z-source z-nix"><span class="z-comment z-line z-number-sign z-nix"># - instantiated derivation: /nix/store/hqmllvbilxslp393ci4lkj66psh5iv6a-stdenv-bootstrap-tools-s390x-unknown-linux-gnu.drv</span>
</span><span class="z-source z-nix"><span class="z-comment z-line z-number-sign z-nix"># - output directory: /nix/store/wnr3zf16ci8ajxnv0v6w3dn8lm93gp5z-stdenv-bootstrap-tools-s390x-unknown-linux-gnu</span>
</span><span class="z-source z-nix"><span class="z-comment z-line z-number-sign z-nix"># - build time: Sun, 28 Jul 2024 14:47:36 +0000</span>
</span><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">bootstrapTools</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-support z-function z-nix">import</span> <span class="z-string z-unquoted z-spath z-nix"><nix/fetchurl.nix></span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">url</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">"</span>http://tarballs.nixos.org/stdenv/s390x-unknown-linux-gnu/8ba481d65eb21a4f9e6b1e812de3f83079eb8016/bootstrap-tools.tar.xz<span class="z-punctuation z-definition z-string z-double z-end z-nix">"</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">hash</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">"</span>sha256-fuKIRXznA8hU8uGpxldAUNvuJBZ/xiyJUByNbpBCaH8=<span class="z-punctuation z-definition z-string z-double z-end z-nix">"</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">busybox</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-support z-function z-nix">import</span> <span class="z-string z-unquoted z-spath z-nix"><nix/fetchurl.nix></span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">url</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">"</span>http://tarballs.nixos.org/stdenv/s390x-unknown-linux-gnu/8ba481d65eb21a4f9e6b1e812de3f83079eb8016/busybox<span class="z-punctuation z-definition z-string z-double z-end z-nix">"</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">hash</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">"</span>sha256-R6nAiaIOgShKiu+qcOP9apVpnuJgTAGAsJxWSHsH4/A=<span class="z-punctuation z-definition z-string z-double z-end z-nix">"</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">executable</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-language z-nix">true</span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset z-nix">}</span>
</span></code></pre>
<p>and located at <code>pkgs/stdenv/linux/bootstrap-files/s390x-unknown-linux-gnu.nix</code>. We then need to source this in <code>pkgs/stdenv/linux/default.nix</code>. You can see the commits achiving this here: <a href="https://github.com/NixOS/nixpkgs/pull/332462/commits/398058603f13a57ca0ed891e81dcf6d39b9a9e05">stdenv: add bootstrap files for s390x</a>.</p>
<blockquote class="note">
<p class="alert-title">
<i class="icon"></i>Note</p>
<p>the hydra builds get flushed within 2 weeks or something so don’t let too much time pass when having the tarballs uploaded. This is done manually
still.</p>
<p><em>As you can see below this mixed with another problem actually ended up in me having to push the bootstrapTools
urls/hash file twice …</em></p>
</blockquote>
<p><a href="https://github.com/NixOS/nixpkgs/pull/332462/commits/bdb500ace1fdc02ab754c58302583e9c4f8ec17f">bintools-wrapper: add dynamicLinker for s390x</a> is also needed as otherwise while we got the bare bootstrap files working we can’t properly link at further stages as the shared library loader would be wrong.</p>
<h3 id="unforseen-consequences-of-forgetting-a-commit-message">unforseen consequences of forgetting a commit message</h3>
<p><strong>TODO mention that technially seen it wasn’t required to put the pasta</strong></p>
<p>I wasn’t sure if I should mention this as its quite unlikely that this will happen again. But I think its
interesting enough to still mention it <strong>whomever reads this probably really won’t
have to care about this again though</strong>. So here we go:</p>
<p>I did make a tiny mistake with consequences a little bigger than anyone up until then expected. <a href="https://discourse.nixos.org/t/bootstrap-files-updates-amplifiy-exploit-of-any-package-into-exploit-of-every-package/50534">Bootstrap-files updates amplifiy exploit of <em>any</em> package into exploit of <em>every</em> package</a> was the result of that mistake. I won’t attempt to explain the issue here once more since its already touched upon in discourse.nixos.org but its still funny to note that essentially the only thing that lead to the discovery of this was:</p>
<p><a href="https://github.com/NixOS/nixpkgs/pull/332462">Bootstrap tools cross add s390x #332462</a></p>
<p>not including the copypasta like seen in:</p>
<p><a href="https://github.com/NixOS/nixpkgs/pull/334334">pkgs/stdenv/linux: update s390x-unknown-linux-gnu bootstrap-files #334334</a>.</p>
<blockquote>
<p><em>(Despite the commits being functionally the same)</em></p>
</blockquote>
<p><strong>TODO maybe actually explain it showcasing the issue more in detail</strong></p>
<ol start="4">
<li>Won’t really be carried out by you anymore unless well you have access to <code>tarballs.nixos.org</code>, it may still be interesting to see what would roughly take place though</li>
</ol>
<pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">nix-store</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>realize</span> /nix/store/ijkl5anf7mx1p3whdkxv4qs5crf6ic35-stdenv-bootstrap-tools-s390x-unknown-linux-gnu</span>
</span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">aws</span></span><span class="z-meta z-function-call z-arguments z-shell"> s3 cp<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>recursive</span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>acl</span> public-read /nix/store/ijkl5anf7mx1p3whdkxv4qs5crf6ic35-stdenv-bootstrap-tools-s390x-unknown-linux-gnu/on-server/ s3://nixpkgs-tarballs/stdenv/s390x-unknown-linux-gnu/0a7eaa55ccaa5103f44a9a4e3e0b06e5314a6401</span>
</span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">aws</span></span><span class="z-meta z-function-call z-arguments z-shell"> s3 cp<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>recursive</span> s3://nixpkgs-tarballs/stdenv/s390x-unknown-linux-gnu/0a7eaa55ccaa5103f44a9a4e3e0b06e5314a6401 ./</span>
</span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">sha256sum</span></span><span class="z-meta z-function-call z-arguments z-shell"> bootstrap-tools.tar.xz busybox</span>
</span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">sha256sum</span></span><span class="z-meta z-function-call z-arguments z-shell"> /nix/store/ijkl5anf7mx1p3whdkxv4qs5crf6ic35-stdenv-bootstrap-tools-s390x-unknown-linux-gnu/on-server/<span class="z-keyword z-operator z-regexp z-quantifier z-shell">*</span></span>
</span></code></pre>
<blockquote class="note">
<p class="alert-title">
<i class="icon"></i>Note</p>
<p>The previously mentioned security concerns resulted in
commands listed above now automatically being included in that pasta
like seen at <a href="https://github.com/NixOS/nixpkgs/pull/334334">pkgs/stdenv/linux: update s390x-unknown-linux-gnu bootstrap-files #334334</a></p>
<p><strong>TODO I very very much feel like this but on second thought I may have put them there myself
but im weirdly confident about
them already being there as I remember asking myself (is this a some github feature i don’t know of?!)</strong></p>
</blockquote>
<p>All of these changes should provide us with an stdenv sufficient to build some basic packages.
But of course a few more things require bootstrapping especially for languages that require
more involved forms of bootstrapping.</p>
<div class="crt scanlines" aria-hidden="true">
<pre class="z-code"><code><span class="z-text z-plain">TODO find a way to not make the term waste space
</span><span class="z-text z-plain">when using ` ticks and language highlihting there
</span></code></pre>
<pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> TODO show how now:</span><span class="z-comment z-line z-number-sign z-shell">
</span></span><span class="z-source z-shell z-bash">
</span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">nix</span></span><span class="z-meta z-function-call z-arguments z-shell"> build .#hello</span>
</span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">./result/bin/hello</span></span>
</span><span class="z-source z-shell z-bash">
</span><span class="z-source z-shell z-bash"><span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> appears to be working</span><span class="z-comment z-line z-number-sign z-shell">
</span></span></code></pre>
</div>
<h3 id="bootstrapping-rust">bootstrapping rust</h3>
<p>Its 2024, rust is growing more and more popular and in fact
in no time you get stuck when building NixOS systems that aren’t
extremely minimal. As <em>“something”</em> will end up depending on it
in some way. <small>(originally I wanted to add it much later on but
it kept showing up as a dependency … luckily its rather trivial to add)</small></p>
<p><strong>TODO: factcheck I think this even happens with the default profiles (rust being a dep)</strong></p>
<p><a href="https://github.com/NixOS/nixpkgs/pull/337908">rust: S390x add native support (bootstrap files) #337908</a> and <a href="https://github.com/NixOS/nixpkgs/pull/337907">openblas: Enable s390x-linux #337907</a> take care of that.</p>
<p><small>Assuming we forget about haskell since they refuse to offer bootstrap packages for s390x or riscv or some other cpus and cross compiling these is a pain and there are issues specifically on s390x (riscv sorta works).
<strong>TODO maybe link the issues</strong></small></p>
<h3 id="gcc-arch-march-matters">gcc.arch (march) matters …</h3>
<p>While for now most work focussed on bootstrapping things
and it should be <em>“enough”</em> to build most pkgs and even
a basic nixos system natively. This is with the assumption
that <code>gcc.march</code> is set adequately.</p>
<p><strong>TODO: show the actual errors</strong></p>
<blockquote>
<p>Honestly as I don’t feel like touching llvm for now and it
may not be trivial to fix this there is a way to prevent
running in various <code>gcc.arch</code> related issues during various builds:</p>
</blockquote>
<blockquote class="caution">
<p class="alert-title">
<i class="icon"></i>Caution</p>
<p><strong>NOT specifying <code>gcc.arch = "z10";</code> OR NEWER
WILL LIKELY CAUSE BUILD FAILURES</strong><small>(working on it though)</small></p>
<p>It should be possible to avoid these issues by
essentially just setting <code>gcc.arch=z10;</code> or <code>gcc.arch=TODO_FURTHER_ARCHES_NAMES</code> like seen below.</p>
<p>Until the decision is made on whether to set this within nixpkgs to <code>"z10"</code> or newer, you can
use something like this in your configuration.nix / flake.nix</p>
<pre data-lang="nix" class="language-nix z-code"><code class="language-nix" data-lang="nix"><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">nixpkgs</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">hostPlatform</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">system</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">"</span>s390x-linux<span class="z-punctuation z-definition z-string z-double z-end z-nix">"</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"> <span class="z-comment z-line z-number-sign z-nix"># linux-kernel = { ... };</span>
</span><span class="z-source z-nix"> <span class="z-comment z-line z-number-sign z-nix"># ... other settings</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">gcc</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">arch</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">"</span>z10<span class="z-punctuation z-definition z-string z-double z-end z-nix">"</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset z-nix">}</span>
</span></code></pre>
</blockquote>
<p><small>Personally I think there is little point to
supporting anything older than the Z10 (2008)
because it seems to cause significantly more issues
and there is another problem:
<del>even IBM’s bootloader appears to assume
that the march is no older than Z10</del>
<strong>(TODO start of 2025 IBM fixed this
after I pointed it out to them)!!!!</strong>
</small></p>
<h3 id="obscure-errors-during-testphase">obscure errors during <code>testPhase</code>?</h3>
<p><small><strong>TODO talk about failing tests
and stuff while they should be mostly fixed by
now mention that at times esp to move forward
its ok to temporarily ignore some tests maybe
for the sake of getting things to build. Crude
but I wanted a PoC ASAP</strong>
</small></p>
<hr />
<p><small>getting bored? Don’t worry
now that we have established the required
basis to build a proper nixos sytem
…
<strong>things may get more interesting now</strong>
</small></p>
<hr />
<h1 id="making-our-system-bootable">making our system bootable</h1>
<p>With the fixes/commits/settings previously mentioned applied,
the only NixOS systems <small>(as build in via: <code>nixpkgs.nixosSystem {}</code>)</small>
are essentially everything that does not require:</p>
<ul>
<li>the bootloader (zipl)</li>
<li>the kernel image</li>
<li>the initrd</li>
</ul>
<p>that effectively means:</p>
<ul>
<li>using a tarball (without kernel/initrd/bootloader) as the output format</li>
<li>using container images <small>(essentially a system rootfs tarball + metadata)</small></li>
</ul>
<p>a little bit further down we take care of the absence of these 3 things. Though
for those interested this is how to construct such container/tarballs:</p>
<p><strong>(TODO maybe show how build a tarball / container)</strong></p>
<h2 id="enable-nixos-to-build-kernels-initrds">enable NixOS to build kernels / initrds</h2>
<p>Like implied previously boot methods such as:</p>
<ul>
<li><a href="https://en.wikipedia.org/wiki/ISO_9660">ISO 966</a><small>(also known as simply “.iso” files)</small></li>
<li>kexec</li>
<li>zipl</li>
</ul>
<p>all require some sort of kernel image + initrd. This part describes how enable NixOS to build these:</p>
<h3 id="building-the-kernel">building the kernel</h3>
<p>To get NixOS to attempt to build a kernel we first
need to give it some details it will use when attempting to build the kernel.</p>
<p>In this case telling it that the <code>baseConfig</code> shall be taken from the platform (s390x)
specific <small>(kernel)</small> defconfig. And that the target to be expected is <code>"bzImage"</code> is enough.</p>
<pre data-lang="nix" class="language-nix z-code"><code class="language-nix" data-lang="nix"><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">nixpkgs</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">hostPlatform</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">system</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">"</span>s390x-linux<span class="z-punctuation z-definition z-string z-double z-end z-nix">"</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">linux-kernel</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">target</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">"</span>bzImage<span class="z-punctuation z-definition z-string z-double z-end z-nix">"</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">name</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">"</span>s390x-defconfig<span class="z-punctuation z-definition z-string z-double z-end z-nix">"</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">autoModules</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-language z-nix">true</span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">baseConfig</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">"</span>defconfig<span class="z-punctuation z-definition z-string z-double z-end z-nix">"</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset z-nix">}</span>
</span><span class="z-source z-nix">
</span></code></pre>
<blockquote class="tip">
<p class="alert-title">
<i class="icon"></i>Tip</p>
<p>While if you actually end up building this
specifically for some (IBM Z) mainframe
the linux kernel does in fact have varoius arch/target
specific specifc optimisation flags. Feel free to use
them or even better see what they do first.</p>
<p><strong>TODO link the options I’m talking about here</strong></p>
</blockquote>
<blockquote>
<p><small> This will probably make it into nixpkgs platform
specifications (there are still things that need to be discussed).
In that case you wouldn’t need to do this anymore but until then
or just as a general <em>“method”</em> thats how it can be done </small></p>
</blockquote>
<p><strong>TODO: give some insight in how nix actually builds the kernel
and how these variables make it there</strong></p>
<h3 id="building-the-initrd">building the initrd</h3>
<p><strong>TODO reword this part</strong></p>
<p>By default the <code>linuxArch</code> is set to <code>"s390"</code>
not <code>"s390x"</code> <small>(just like it would select <code>x86</code> instead of <code>x86_64</code>)</small>. As that compiles just fine one may expect it to
work just fine as well but sadly thats actually not the case as for <code>"s390"</code>.</p>
<p><strong>TODO link where this happens</strong></p>
<p>For example the <code>ifconfig</code> utility found within the initrd
which is invoked when passing <code>"ip="</code> type parameters trough the kernel commandline
will actually thorw an obscure <strong>“parsing error”</strong></p>
<div class="crt scanlines" aria-hidden="true">
<pre class="z-code"><code><span class="z-text z-plain">TODO <paste the actual error here>
</span></code></pre>
</div>
<p>For some reason this doesen’t happen on the 64bit 390x counterpart.</p>
<p>While klibc does actually claim to support s390 I couldn’t get this properly runnig with
ifconfig.</p>
<h3 id="can-we-just-use-64bit">can we just use 64bit?</h3>
<p>sort of … and honestly if using 64bit works without these
obscure issues. I belive that <small>(at least for now)</small> it should be fine.</p>
<blockquote class="note">
<p class="alert-title">
<i class="icon"></i>Note</p>
<p>The reasoning to use an <code>linuxArch</code> with a smaller bitsize <strong>TODO or how do you call this</strong>
exclusively within the initrd is for size binary reasons. Shorter Pointers and opcodes
naturally produce smaller executables than if the bitsize <strong>TODO how to call this</strong> was for example 2x as big</p>
</blockquote>
<p>to be honest I think that the size argument is much less relevant on mainframes when
booting via zipl. As the files are actually commonly found on the systems “/” partition
which shouldn’t lack storage like mainframes in general.</p>
<blockquote class="warning">
<p class="alert-title">
<i class="icon"></i>Warning</p>
<p><strong>TODO find it again: there was this one case (was it cdrom?) where either both combined
or maybe individuallly kernel and initrd had to be below a certain size</strong></p>
<p>In such cases maybe one actually benefits from a smaller <code>linuxArch</code>. I think this shouldn’t
matter though as long as one doesn’t build initrds with a lot of <em>“extra utilities”</em>.</p>
<p><small>In such cases though thats usually due to more involved crypto key
retrival operations. (ipsec/wg/tor/…) and in such cases I think using 32bit
may actually be a security concern.</small></p>
</blockquote>
<p>This may be a little more relevant if it was common to use some form
of a dedicated <code>"/boot/"</code> partition instead <small>(like see with EFI vfat32 based
boot partitions)</small>. But I guess as long as the zipl bootloader is used its fine to just boot from <code>"/"</code>.</p>
<p><strong>TODO include notice of</strong>: sth like: ( This of course assumes that your filesystem allows these old style bootloaders, ext4 does) < name how these sort of bootloaders that just sorta
remember the location of a file on the filesystem are called> blockdevice and load that … as afaik zipl only works with this kind of filesystem ></p>
<p><strong>TODO: talk more about how the bootloader works in detail maybe</strong></p>
<hr />
<p>so … are we finally good to go?</p>
<hr />
<h2 id="first-boots-and-debugging-them-with-limited-tools">first boots (AND debugging them with limited tools)</h2>
<p>I guess before things get a little ugly. I should point out that
of course a few more settings need to be done and of course
you need an actual nixos system configuration to build. Instead
of going trough every snippet especially as a lot of them
may change in the future I will just link <a href="https://github.com/bolives-hax/nixos-s390x-images/blob/master/flake.nix">my confgigs</a>.
As for now I’m more or less using <a href="https://github.com/bolives-hax/nixos-s390x-images/commit/cade49a7dfbd4bb48145f48456a38faa84550c9b">this commit</a>
which despite being a little crude does at least produce working</p>
<ul>
<li>lxc compatible images</li>
<li>kexec bundles (live systems)</li>
<li>rootfs tarballs</li>
<li>cdrom images <small>(will touch on that later on)</small></li>
</ul>
<p>Getting whatever I produced to actually boot up succesfully isn’t quite as straightforward
as I hoped it would be. Thats why the next section first points out the limitations I faced
and how I worked around them.</p>
<h3 id="the-restrictions">The restrictions</h3>
<p>Well honestly the biggest issue is the <strong>complete absence</strong> of a serial interface on the Z15 mainframe provided to me by IBM</p>
<blockquote class="note">
<p class="alert-title">
<i class="icon"></i>Note</p>
<p>I did contact IBM regarding the absence of a serial out of bounds console
and while they sadly couldn’t provide me with this please
keep in mind that this is not an issue people using the comercial platforms
would face.</p>
</blockquote>
<p>So since I brought it up before and even included a picture. You may wonder:</p>
<p><q>hey what about the Z10 why don’t you just use that?!</q></p>
<aside> pic related: <img class="no-hover"alt="power_plug_used_pic"src="https://files.catbox.moe/t3611w.jpg"/></aside>
<p>Well as you can probably tell by the picture, just using the closest ordinary poweroutlet
won’t really cut it. In fact there is 2 problems</p>
<ul>
<li>
<p>(<strong>TODO say a little about the connector, where its found, how it differs from ordinary
one and well trough this point out why we essentially need this) (and say the outlet for “this” is far away and the cable had to be made bla bla)</strong></p>
</li>
<li><blockquote class="important">
<p class="alert-title">
<i class="icon"></i>Important</p>
<p><strong>TODO explain the FI trip mechanism as well) Additionally turns out the (TODO translate FI schutzschalter) would actually trip with the default (TODO FI term). (TODO list what strengths were used I believe it was 30mA and eventually 300mA TODO confirm this).</strong></p>
<blockquote>
<p>((As an interesting sidenote up until the Z <strong>TODO which Z-series</strong> mainframe, having a relatively high leakage current apparently was the norm as it could be attributed to the way IBM
designed the <strong>?power distribution circuts TODO or how to call this?</strong>. So
actually having a <strong>TODO english term for FI</strong> trip is the expected behavior eventhough one may be
under the assumption that this is not the intended behavior.))</p>
</blockquote>
</blockquote>
</li>
</ul>
<h3 id="other-reasons">other reasons:</h3>
<ul>
<li>
<p>While being able to play around with a IBM Z15 mainframe is definitively something. Nobody
except me would really benefit from me getting to run NixOS on it. But in all honesty this
would be a waste of resources. IBM is paying the power either way and the idea after all
is to use this for developping/building open source community projects.</p>
</li>
<li>
<p>As IBM already agreed to provide me with additional resources on the Z15 to make it
part of the offcial NixOS CI. I decided to talk with some folks from the NixOS infra team who
gave their OK and the current plan is to use this server to offer
cached packages/drv builds for nixos on s390x stemming from an officially trusted source</p>
</li>
<li>
<p>The Z15 is also obviously much newer and performs better in quite a few scenarios</p>
</li>
</ul>
<p>so all in all it makes sense to use what they provided me with and the most desirable
outcome would be that its being used for hydra / cache.nixos.org</p>
<h3 id="what-are-the-implications-of-using-ibm-s-z15-instead-of-the-z10">what are the implications of using IBM’s Z15 instead of the Z10?</h3>
<p>Concretely this means: I am <strong>completely “blind”</strong> until … well … “something” boots up and responds to me</p>
<p>Of course I knew that this would cause me a lot of pain <del>but then I seem to get lucky at times</del>.</p>
<h3 id="what-about-using-emulators">What about using emulators?</h3>
<p>Luckily emulators while slow
(especially <a href="https://en.wikipedia.org/wiki/Hercules_(emulator)">Hercules</a>)
are still a viable choice.
Qemu in fact is fast enough for me to not really be bothered especially
when running <em>“minimal linux kernels/userland applications”</em>. As unlike the Z15 hosted by IBM
qemu does provide me with a serial console. The idea of using that until I confirmed that my initrd
was somewhat behaving like expected made sense to me at that time</p>
<p><mark>I must already give you a heads up here, things weren’t as simple
<br>…<br>
(I could have expected that)</mark></p>
<p>Just like with kexec to boot this in qemu we need a kernel+initrd image
to pass over the commandline.</p>
<p><small>(unless were ok with using iso images. But as of
<strong>numours issues</strong> usually of the <em>“I wish someone would have told me”</em>-kind
I didn’t manage to produce these yet at that time</small></p>
<h4 id="building-the-nixos-system-components">building the NixOS-system components</h4>
<p><strong>TODO either modify the snippets below to use qemu or mention
that I was just copying the paths from the kexec script as its
functionally almost the same</strong></p>
<p>Well while you can of course look at the config it does go somewhat like:</p>
<ol>
<li>import the <code><nixos/modules/installer/netboot/netboot-minimal.nix></code> module</li>
</ol>
<blockquote>
<p>while we aren’t going to netboot we still don’t want to split initrd
and whatever holds the root system. <small>You will see later why simplicity is key</small></p>
</blockquote>
<ol start="2">
<li>using some form of Mic92s <a href="https://gist.github.com/Mic92/4fdf9a55131a7452f97003f445294f97">kexec script</a>
<small>(while I don’t use it the concept is pretty much the same)</small>.
The attribute set spit out by <code>nixpkgs.lib.nixosSystem {}</code>
has the <code>config</code> attribute from which specifically <code>config.system.build</code> is of interest to us. Thus</li>
</ol>
<pre data-lang="nix" class="language-nix z-code"><code class="language-nix" data-lang="nix"><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">x</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-variable z-parameter z-name z-nix">nixpkgs</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">lib</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">nixosSystem</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span>
</span><span class="z-source z-nix"> <span class="z-comment z-line z-number-sign z-nix"># ... the config</span>
</span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset z-nix">}</span>
</span></code></pre>
<p>would make <code>x</code> expose <code>x.config.system.build</code> and consequently:</p>
<ul>
<li><code>x.config.system.build.toplevel</code></li>
<li><code>x.config.system.build.netbootRamdisk</code> (comes from the <code>netboot-minimal.nix</code> module we loaded before)</li>
<li><code>x.config.system.build.kernel</code></li>
</ul>
<p>As these are all derivations we can also build them individually using
nix build or use the <code>${}</code> operator on them. As all derivations
produce an <code>.outPath</code> <small>(/nix/store/<code><hash></code>-name)</small></p>
<p><code>.toplevel</code> Is the store path to something that qualifies as an
nixos system. Of particular interest here would be <code>${system.build.toplevel}/init</code> as we
need to pass that via the kernel commandline <small>(<code>init=</code>)</small>.</p>
<p><code>.netbootRamdisk</code> is essentially a <em>“fat initrd”</em>. As initrds usually just
contain a very minimal linux environment to mount the actual root drive.
We need to specifically build one that actually houses our entire system</p>
<p><code>.kernel</code> I believe needs no explaination. Though keep in mind that the
location may differ. Here it should be bzImage <strong>TODO confirm this</strong>
<strong>TODO maybe explain how nix chooses the kernel image format and
how to retrive it</strong></p>
<p>while not a derivation <code>.config.boot.kernelParams</code> is also integral
as it contains effectively our kernel commandline that we either
pass to kexec via <code>--commandline</code> or qemu via <code>--append</code>. Retriving
this can simply be done trough calling <code>toString</code> on it like
<code>${x.config.boot.kernelParams}</code> or using <code>nix eval</code> <strong>(TODO show an example for eval)</strong></p>
<p><strong>TODO check if this simplified snippet actually works like mine</strong></p>
<p>If we wanted to output this as a kexec script it would look something like this:</p>
<pre data-lang="nix" class="language-nix z-code"><code class="language-nix" data-lang="nix"><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">kexecScript</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-variable z-parameter z-name z-nix">pkgs</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">writeScriptBin</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">"</span>kexec-boot<span class="z-punctuation z-definition z-string z-double z-end z-nix">"</span></span> <span class="z-string z-quoted z-other z-nix"><span class="z-punctuation z-definition z-string z-other z-start z-nix">''</span>
</span></span><span class="z-source z-nix"><span class="z-string z-quoted z-other z-nix"> ./kexec --load <span class="z-markup z-italic"><span class="z-punctuation z-section z-embedded z-begin z-nix">${</span><span class="z-variable z-parameter z-name z-nix">x</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">config</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">system</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">build</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">kernel</span><span class="z-punctuation z-section z-embedded z-end z-nix">}</span></span>/bzImage \
</span></span><span class="z-source z-nix"><span class="z-string z-quoted z-other z-nix"> --initrd=<span class="z-markup z-italic"><span class="z-punctuation z-section z-embedded z-begin z-nix">${</span><span class="z-variable z-parameter z-name z-nix">x</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">config</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">system</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">build</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">netbootRamdisk</span><span class="z-punctuation z-section z-embedded z-end z-nix">}</span></span>/initrd \
</span></span><span class="z-source z-nix"><span class="z-string z-quoted z-other z-nix"> --command-line "init=<span class="z-markup z-italic"><span class="z-punctuation z-section z-embedded z-begin z-nix">${</span><span class="z-variable z-parameter z-name z-nix">x</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">config</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">system</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">build</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">toplevel</span><span class="z-punctuation z-section z-embedded z-end z-nix">}</span></span>/init <span class="z-markup z-italic"><span class="z-punctuation z-section z-embedded z-begin z-nix">${</span><span class="z-support z-function z-nix">toString</span> <span class="z-variable z-parameter z-name z-nix">x</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">config</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">boot</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">kernelParams</span><span class="z-punctuation z-section z-embedded z-end z-nix">}</span></span>"
</span></span><span class="z-source z-nix"><span class="z-string z-quoted z-other z-nix"> <span class="z-punctuation z-definition z-string z-other z-end z-nix">''</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset z-nix">}</span>
</span></code></pre>
<p><strong>TODO test the qemu one and put it on git as well and maybe make it
a little smarter than hardcoding mem like via nix run and accepting
dunno additionall params or sth like -m or -net instead of hardcoding them</strong></p>
<pre data-lang="nix" class="language-nix z-code"><code class="language-nix" data-lang="nix"><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">qemuScript</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-variable z-parameter z-name z-nix">pkgs</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">writeScriptBin</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">"</span>qemu-boot<span class="z-punctuation z-definition z-string z-double z-end z-nix">"</span></span> <span class="z-string z-quoted z-other z-nix"><span class="z-punctuation z-definition z-string z-other z-start z-nix">''</span>
</span></span><span class="z-source z-nix"><span class="z-string z-quoted z-other z-nix"> <span class="z-markup z-italic"><span class="z-punctuation z-section z-embedded z-begin z-nix">${</span><span class="z-variable z-parameter z-name z-nix">pkgs</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">qemu</span><span class="z-punctuation z-section z-embedded z-end z-nix">}</span></span>/bin/qemu-system-s390x -M s390-ccw-virtio \
</span></span><span class="z-source z-nix"><span class="z-string z-quoted z-other z-nix"> -m 2048 -smp 4 -nographic \
</span></span><span class="z-source z-nix"><span class="z-string z-quoted z-other z-nix"> -net nic -net user \
</span></span><span class="z-source z-nix"><span class="z-string z-quoted z-other z-nix"> -kernel <span class="z-markup z-italic"><span class="z-punctuation z-section z-embedded z-begin z-nix">${</span><span class="z-variable z-parameter z-name z-nix">x</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">config</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">system</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">build</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">kernel</span><span class="z-punctuation z-section z-embedded z-end z-nix">}</span></span>/bzImage \
</span></span><span class="z-source z-nix"><span class="z-string z-quoted z-other z-nix"> -initrd <span class="z-markup z-italic"><span class="z-punctuation z-section z-embedded z-begin z-nix">${</span><span class="z-variable z-parameter z-name z-nix">x</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">config</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">system</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">build</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">netbootRamdisk</span><span class="z-punctuation z-section z-embedded z-end z-nix">}</span></span>/initrd \
</span></span><span class="z-source z-nix"><span class="z-string z-quoted z-other z-nix"> -append "init=<span class="z-markup z-italic"><span class="z-punctuation z-section z-embedded z-begin z-nix">${</span><span class="z-variable z-parameter z-name z-nix">x</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">config</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">system</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">build</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">toplevel</span><span class="z-punctuation z-section z-embedded z-end z-nix">}</span></span>/init <span class="z-markup z-italic"><span class="z-punctuation z-section z-embedded z-begin z-nix">${</span><span class="z-support z-function z-nix">toString</span> <span class="z-variable z-parameter z-name z-nix">x</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">config</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">boot</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">kernelParams</span><span class="z-punctuation z-section z-embedded z-end z-nix">}</span></span>"
</span></span><span class="z-source z-nix"><span class="z-string z-quoted z-other z-nix">}
</span></span></code></pre>
<p><strong>TODO mention how to use nix repl to elegantly retrive
these</strong></p>
<p><strong>TODO talk about the limited set of drivers, that dasd’s can’t be emulated here (TODO are you 1000% sure)
and youd need hercules (maybe also show how to do that)</strong></p>
<h2 id="booting-nixos-qemu">booting NixOS (qemu)</h2>
<p>if we either extract the variables from above or use the kexec script we build we will end up
with</p>
<pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">qemu-system-s390x</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>M</span> s390-ccw-virtio <span class="z-punctuation z-separator z-continuation z-line z-shell">\
</span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>m</span> 2048<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>smp</span> 4<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>nographic</span> <span class="z-punctuation z-separator z-continuation z-line z-shell">\
</span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>net</span> nic<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>net</span> user <span class="z-punctuation z-separator z-continuation z-line z-shell">\
</span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>kernel</span> /TODO/NIX/STORE/aaaaaaaaaaaaaaaaaa/bzImage <span class="z-punctuation z-separator z-continuation z-line z-shell">\
</span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>initrd</span> /TODO/NIX/STORE/aaaaaaaaaaaaaaaaaa/initrd <span class="z-punctuation z-separator z-continuation z-line z-shell">\
</span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>append</span> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">"</span>init=<span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-punctuation z-section z-expansion z-parameter z-begin z-shell">{</span></span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-variable z-other z-readwrite z-shell">x.config.system.build.toplevel</span></span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-section z-expansion z-parameter z-end z-shell">}</span></span>/init <span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-punctuation z-section z-expansion z-parameter z-begin z-shell">{</span></span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-variable z-other z-readwrite z-shell">toString x.config.boot.kernelParams</span></span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-section z-expansion z-parameter z-end z-shell">}</span></span><span class="z-punctuation z-definition z-string z-end z-shell">"</span></span></span>
</span></code></pre>
<blockquote class="tip">
<p class="alert-title">
<i class="icon"></i>Tip</p>
<p>For a variety of reasons when doing these sort of things its <mark>very useful</mark> to
be able to drop a shell in the early initrd stages upon failure. Like when
adding <code>boot.shell_on_fail</code> to the kernel commandline. But also deliberately dropping it.
It is quite likely that you will
need to drop it specifically during various stages of the initrd. There are a few
kernel commandline parameters that can be used (besides just patching the init template scripts of course):</p>
<ul>
<li>
<p><code>boot.debug1</code> <strong>TODO im not sure anymore if the serial console
provided by qemu is already usable/configured in this stage of the init</strong></p>
</li>
<li>
<p><code>boot.debug1devices</code></p>
</li>
<li>
<p><code>boot.debug1mounts</code></p>
</li>
<li>
<p><code>boot.trace</code> (I almost kept this on until the very end. <small>May slow down the boot process slightly though</small>)</p>
</li>
</ul>
</blockquote>
<blockquote class="important">
<p class="alert-title">
<i class="icon"></i>Important</p>
<p><strong>serial console ignoring keystrokes?!</strong></p>
<p>Keep in mind that <strong>(if present)</strong> on s390/s390x the native
console type is SCLP. Linux thus assigns the console a name
such as: <code>/dev/ttysclp0</code> <strong>(in my case as the first serial console)</strong></p>
<p>As of the way the NixOS’s init scripts are designed you can only configure one
terminal interface to function in a bidirectional fashion. This is the first terminal
specified over the kernel commandline. If left unspecified <strong>(in qemu but maybe also
in other places)</strong> the console chosen may not actually be exposed to you
in a way where you can actual issue any inputs. Which makes dropping a shell
impossible.</p>
<p>When systemd takes over later on into the bootprocess and launches
actual services to set up the various consoles for the second time it shouldn’t matter. But
if for any reason you require a functional console earlier on. Be it to
provide your disk unlocking mechanism or to provide you with
a functional debug+rescue shell environment. You have to set this correctly
or you simply won’t be able to interface with your host</p>
</blockquote>
<p><strong>TODO mention that -cdrom isn’t actually scsi or whatever but virtio
and the implications of that in terms of cdrom boot not working
https://www.ibm.com/docs/en/linux-on-systems?topic=virtio-ccw-device-specifications
essentially SCSI Host Bus Adapter (virtio-scsi) and Block (virtio-block) aint the same</strong></p>
<p><strong>TODO terminal abstract of this actually sort of working</strong></p>
<h3 id="booting-alpine-linux">booting Alpine Linux ?!</h3>
<blockquote>
<p><strong>wasn’t this an article about NixOS why would one talk about alpine linux now?!</strong></p>
</blockquote>
<p>While my build would actually come up <strong>“when directly booting into the kernel/initrd”</strong> . There
were still some things that would not work. (most importantly networking).</p>
<p>As alpine was tested to at least run in qemu
I settled with attempting to actually boot
something thats known to work as well. Having a working setup to use as a reference
or just to query certain hardware information proved quite useful. Also quite a few tools are pre-packaged
so I’d not have to bother as much fixing NixOS package builds(drv’s) before I got a host natively running NixOS</p>
<p>Alpine linux ended up being my distro of choice as it provided
the most straightforward way to boot into a linux system
by just offering kernel/initrd/etc images at:</p>
<p><a href="https://dl-cdn.alpinelinux.org/alpine/v3.19/releases/s390x/netboot/">https://dl-cdn.alpinelinux.org/alpine/v3.19/releases/s390x/netboot/</a></p>
<p>Having direct access to the files making up the kernel + initrd does not only mean
I can run this in qemu. But also that I could make use of kexec later on like on the Z15.</p>
<p>Which is quite neat as opposed to
having to bake an .iso image or such things which tends to involve much more things
that can go wrong. <small>(Sometimes netboot is an option as well but I didn’t look into that)</small></p>
<p>Thus after downloading the files in question from there one could attempt something like:</p>
<pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">qemu-system-s390x</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>M</span> s390-ccw-virtio <span class="z-punctuation z-separator z-continuation z-line z-shell">\
</span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>m</span> 2048<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>smp</span> 4<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>nographic</span> <span class="z-punctuation z-separator z-continuation z-line z-shell">\
</span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>net</span> nic<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>net</span> user <span class="z-punctuation z-separator z-continuation z-line z-shell">\
</span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>kernel</span> vmlinuz-lts <span class="z-punctuation z-separator z-continuation z-line z-shell">\
</span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>initrd</span> initramfs-lts <span class="z-punctuation z-separator z-continuation z-line z-shell">\
</span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>append</span> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">"</span>ip=dhcp alpine_repo=https://dl-cdn.alpinelinux.org/alpine/v3.19/main modloop=https://dl-cdn.alpinelinux.org/alpine/v3.19/releases/s390x/netboot/modloop-lts<span class="z-punctuation z-definition z-string z-end z-shell">"</span></span></span>
</span></code></pre>
<blockquote class="important">
<p class="alert-title">
<i class="icon"></i>Important</p>
<p>(make sure that the files provided via the “-kernel” and “-initrd” flags match the version
of whats specified via alpine_repo AND modloop!</p>
</blockquote>
<blockquote class="note">
<p class="alert-title">
<i class="icon"></i>Note</p>
<p>this relies on a properly working WAN internet connection (including dns ofc).
It also takes quite a while when using emulation (instead of kvm accel) and most
importantly if emulation is VEEERY slow or if your network is unstable this may fail
and drop you off into an emergency shell.</p>
<p>Definitively suboptimal but as long as you
are aware of these things it shouldn’t be a huge issue. If you get errors suggesting some remote
resources can’t be found (<strong>TODO example</strong>) you may have a version mismatch.
<mark> Please make sure all the files come from the same release.
You can also use the emergency shell to debug the network configuration if you suspect something
is off. </mark></p>
</blockquote>
<p>After running the command above pretty much instantly im presented with
<div class="crt scanlines" aria-hidden="true">
<pre class="z-code"><code><span class="z-text z-plain">ASLR disabled: CPU has no PRNG
</span><span class="z-text z-plain">KASLR disabled: CPU has no PRNG
</span><span class="z-text z-plain">[ 0.539393] Linux version 6.6.32-0-lts (buildozer@build-edge-s390x) (gcc (Alpine 13.2.1_git20240309) 13.2.1 20240309, GNU ld (GNU Binutils) 2.42) #1-Alpine SMP Fri, 24 May 2024 08:38:05 +0000
</span><span class="z-text z-plain">[ 0.541762] setup: Linux is running under KVM in 64-bit mode
</span></code></pre>
</div>
indicating that its about to boot, further down I saw:</p>
<div class="crt scanlines" aria-hidden="true">
<pre class="z-code"><code><span class="z-text z-plain"> * Mounting boot media: [ 13.341232] Mounting boot media: ok.
</span><span class="z-text z-plain">ok.
</span><span class="z-text z-plain">[ 13.604847] Obtaining IP via DHCP (eth0)...
</span><span class="z-text z-plain"> * Obtaining IP via DHCP (eth0): udhcpc: started, v1.36.1
</span><span class="z-text z-plain">[ 13.900690] NET: Registered PF_PACKET protocol family
</span><span class="z-text z-plain">udhcpc: broadcasting discover
</span><span class="z-text z-plain">udhcpc: broadcasting select for 10.0.2.15, server 10.0.2.2
</span><span class="z-text z-plain">udhcpc: lease of 10.0.2.15 obtained from 10.0.2.2, lease time 86400
</span><span class="z-text z-plain">[ 14.903964] Obtaining IP via DHCP (eth0): ok.
</span><span class="z-text z-plain">ok.
</span><span class="z-text z-plain">[ 16.258689] Installing packages to root filesystem...
</span><span class="z-text z-plain"> * Installing packages to root filesystem: fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/s390x/APKINDEX.tar.gz
</span><span class="z-text z-plain">(1/26) Installing alpine-baselayout-data (3.4.3-r2)
</span></code></pre>
</div>
<p>and then finally</p>
<div class="crt scanlines" aria-hidden="true">
<pre class="z-code"><code><span class="z-text z-plain"> OpenRC 0.52.1 is starting up Linux 6.6.32-0-lts (s390x)
</span><span class="z-text z-plain">
</span><span class="z-text z-plain"> * /proc is already mounted
</span><span class="z-text z-plain"> * Mounting /run ... [ ok ]
</span><span class="z-text z-plain"> * /run/openrc: creating directory
</span><span class="z-text z-plain"> * /run/lock: creating directory
</span><span class="z-text z-plain"> * /run/lock: correcting owner
</span><span class="z-text z-plain"> * Caching service dependencies ... [ ok ]
</span><span class="z-text z-plain"> * Remounting devtmpfs on /dev ... [ ok ]
</span><span class="z-text z-plain"> * Mounting /dev/mqueue ... [ ok ]
</span><span class="z-text z-plain">Connecting to dl-cdn.alpinelinux.org (151.101.130.132:443)
</span><span class="z-text z-plain">saving to '/lib/modloop-lts'
</span><span class="z-text z-plain">modloop-lts 100% |********************************| 17.5M 0:00:00 ETA
</span><span class="z-text z-plain">'/lib/modloop-lts' saved
</span><span class="z-text z-plain"> * Mounting modloop /lib/modloop-lts ... * Verifying modloop
</span><span class="z-text z-plain"> * Failed to verify signature of !
</span><span class="z-text z-plain"> [ ok ]
</span><span class="z-text z-plain"> * Mounting security filesystem ... [ ok ]
</span><span class="z-text z-plain"> * Mounting debug filesystem ... [ ok ]
</span><span class="z-text z-plain"> * Starting busybox mdev ... [ ok ]
</span><span class="z-text z-plain"> * Scanning hardware for mdev ... [ ok ]
</span><span class="z-text z-plain"> * Loading hardware drivers ... [ ok ]
</span><span class="z-text z-plain"> * Loading modules ...modprobe: can't change directory to '6.6.32-0-lts': No such file or directory
</span><span class="z-text z-plain">modprobe: can't change directory to '6.6.32-0-lts': No such file or directory
</span><span class="z-text z-plain"> [ ok ]
</span><span class="z-text z-plain"> * Not setting clock for s390 system
</span><span class="z-text z-plain"> * Checking local filesystems ... [ ok ]
</span><span class="z-text z-plain"> * Remounting filesystems ... [ ok ]
</span><span class="z-text z-plain"> * Mounting local filesystems ... [ ok ]
</span><span class="z-text z-plain"> * Configuring kernel parameters ... [ ok ]
</span><span class="z-text z-plain"> * Migrating /var/lock to /run/lock ... [ ok ]
</span><span class="z-text z-plain"> * Creating user login records ... [ ok ]
</span><span class="z-text z-plain"> * Cleaning /tmp directory ... [ ok ]
</span><span class="z-text z-plain"> * Setting hostname ... [ ok ]
</span><span class="z-text z-plain"> * Starting busybox syslog ... [ ok ]
</span><span class="z-text z-plain"> * Starting firstboot ... [ ok ]
</span><span class="z-text z-plain">
</span><span class="z-text z-plain">Welcome to Alpine Linux 3.19
</span><span class="z-text z-plain">Kernel 6.6.32-0-lts on an s390x (/dev/ttysclp0)
</span><span class="z-text z-plain">
</span><span class="z-text z-plain">localhost login:
</span></code></pre>
</div>
<p><strong>TODO there seems to be a can’t change directory issue here … fix it</strong></p>
<blockquote class="tip">
<p class="alert-title">
<i class="icon"></i>Tip</p>
<p>logging in as root requires no password and drops a shell</p>
</blockquote>
<p>While emulated now we at least have some solid ground under our feet to
test our initrd or pretty much all software on. I extensively made use of this
to test out things before I could get them to run on real hardware from executables
to kexec-ing into other things to just checking network connectivity. <small>(of course packages
can be installed via <code>apk add</code> as for example kexec is not present by default)</small></p>
<blockquote class="note">
<p class="alert-title">
<i class="icon"></i>Note</p>
<p>Please keep in mind that the virtio hardware thats being emulated
is <strong>quite different</strong> from what you may find when trying to pull of any of these things
on an actual s390x host. Interface names may be different, the drivers used are most
definitively different and as you will see later setting up disk/network appears to be
much more involved than loading the kernel module …</p>
</blockquote>
<p><strong>TODO explain how I tested nix before testing NixOS by essentially putting /nix into the alpine and starting the daemon manually …</strong></p>
<h2 id="booting-nixos-ibm-z15">booting NixOS (IBM Z15)</h2>
<p>By the point everything seemed fine in qemu. I decided to run it natively
on the Z15 I mentioned earlier. The initial idea was quite simple:</p>
<p><strong>TODO compare this with the actual command this is simplified</strong></p>
<p><strong>TODO make sure to mention that no root disk is needed
in this case as it uses a modified version of Mic92s kexec thingy</strong></p>
<pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">kexec</span></span><span class="z-meta z-function-call z-arguments z-shell"> ./bl0v3s-s390x-nixos-kernel <span class="z-punctuation z-separator z-continuation z-line z-shell">\
</span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>initrd</span><span class="z-keyword z-operator z-assignment z-option z-shell">=</span>./bl0v3s-s390x-nixos-initrd <span class="z-punctuation z-separator z-continuation z-line z-shell">\
</span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>append</span><span class="z-keyword z-operator z-assignment z-option z-shell">=</span><span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">"</span>ip=dhcp<span class="z-punctuation z-definition z-string z-end z-shell">"</span></span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>l</span></span>
</span><span class="z-source z-shell z-bash">
</span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">kexec</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>e <span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> to execute the loaded os</span><span class="z-comment z-line z-number-sign z-shell">
</span></span></span></span></code></pre>
<blockquote class="note">
<p class="alert-title">
<i class="icon"></i>Note</p>
<p>running the command above or any kexec call won’t give you
any feedback unless it fails before jumping to the new
kernel. This means that until whatever you booted manages
to notify you in some way <small>like trough the serial console
which isn’t present in my case …</small> you have no
way of knowing if its still loading. Crashed or came up
but failed to notify you <small>(for example if you rely on
ssh/icmp like I do)</small>. Turning you essentially blind</p>
</blockquote>
<blockquote class="note">
<p class="alert-title">
<i class="icon"></i>Note</p>
<p>Well, while this is specific to the network environment IBM uses on the
machine given to me. There already is an issue: <code>ip=dhcp</code> won’t cut it
because they actually don’t offer dhcp. <mark>which is perfect fine as
under normal conditions you’d never have or rather want to interface
with the network in that way but just sort of let them administer the
system for you</mark></p>
</blockquote>
<p><strong>TODO show how I modified stage 1 to drop me an ssh shell (if it would have worked)</strong></p>
<p>Which meant changing the append parameter found in the kexec call above to something like:</p>
<p><code>--append="ip=148.100.85.113::148.100.84.1:255.255.254.0:nixos:eth0::9.9.9.9:8.8.8.8"</code></p>
<p>with:</p>
<ul>
<li><code>148.100.85.113</code> being the ip statically assigned to eth0</li>
<li><code>148.100.84.1</code> being the gateway</li>
<li><code>255.255.254.0</code> being the subnetmask (a <code>/23</code> essentially)</li>
<li><code>nixos</code> being the hostname</li>
<li><code>eth0</code> being the target interface to assign these values to</li>
<li>
<blockquote>
<p><code>9.9.9.9</code> and <code>8.8.8.8</code> are dns servers run by cloud9 and google as fallback.
<small> <blockquote class="note">
<p class="alert-title">
<i class="icon"></i>Note</p>
<p>(not too relevant for our nix host but running alpine like described above would
actually require this when resolving domains as it sort of is a “minimal netboot” image)</small></p>
</blockquote>
</p>
</blockquote>
</li>
</ul>
<p>For reference here is the network configuration (excluding v6) directly
taken from the Z15 when it was running something that offered me a shell:</p>
<pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> show the ip address configuration</span><span class="z-comment z-line z-number-sign z-shell">
</span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">ip</span></span><span class="z-meta z-function-call z-arguments z-shell"> a show eth0</span>
</span><span class="z-source z-shell z-bash">
</span><span class="z-source z-shell z-bash"><span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> show the routes (focus on "default")</span><span class="z-comment z-line z-number-sign z-shell">
</span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">ip</span></span><span class="z-meta z-function-call z-arguments z-shell"> r</span>
</span></code></pre>
<div class="crt scanlines" aria-hidden="true">
<pre class="z-code"><code><span class="z-text z-plain">2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
</span><span class="z-text z-plain"> link/ether 02:02:12:f4:34:1a brd ff:ff:ff:ff:ff:ff
</span><span class="z-text z-plain"> inet 148.100.85.113/23 scope global eth0
</span><span class="z-text z-plain"> valid_lft forever preferred_lft forever
</span></code></pre>
</div>
<p>and for the route</p>
<div class="crt scanlines" aria-hidden="true">
<pre class="z-code"><code><span class="z-text z-plain">default via 148.100.84.1 dev eth0 proto static
</span></code></pre>
</div>
<p>seems correct. Especially with the initrd userland using 64bit ifconfig
should be able to make sense out of the kernel commandline, right? Well actually <strong>it does</strong>
only issue is that <mark><strong>it really aint that simple</strong></mark></p>
<p><strong>TODO add how I confirmed that networking this stuff worked in qemu and even initrd sshd stage1</strong></p>
<p><strong>TODO mention how and why I turned off predictable interface foo</strong></p>
<blockquote class="important">
<p class="alert-title">
<i class="icon"></i>Important</p>
<p><strong>TODO explain <a href="https://www.ibm.com/docs/en/linux-on-systems?topic=resources-persistent-configuration">Persistent device configuration</a>.
And why this is a thing. And most importantly why this is not the sort of approach one
would want to use in NixOS (declarative distros). As it heavily relies on udev
and rebuilding the initrd to contain the “persistent” effectively udev rule changes</strong></p>
</blockquote>
<p>So at this point in time my assumption was that it probably is lacking drivers.</p>
<p>A quick way to check without using any
specialized commands that may not be present is going
via sysfs. So in this case with our interface being named <code>eth0</code>
running:</p>
<pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">ls</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>l</span> /sys/class/net/eth0/device/driver/</span>
</span></code></pre>
<p>reveals:</p>
<div class="crt scanlines" aria-hidden="true">
<pre class="z-code"><code><span class="z-text z-plain">...
</span><span class="z-text z-plain">lrwxrwxrwx 1 root root 0 Feb 3 01:18 0.0.1000 -> ../../../../devices/qeth/0.0.1000
</span><span class="z-text z-plain">--w------- 1 root root 4096 Feb 3 01:20 bind
</span><span class="z-text z-plain">--w------- 1 root root 4096 Nov 10 12:04 group
</span><span class="z-text z-plain">lrwxrwxrwx 1 root root 0 Feb 3 01:20 module -> ../../../../module/qeth
</span><span class="z-text z-plain">--w------- 1 root root 4096 Feb 3 01:20 uevent
</span><span class="z-text z-plain">--w------- 1 root root 4096 Feb 3 01:20 unbind
</span><span class="z-text z-plain">...
</span></code></pre>
</div>
<p>so <code>module -> ../../../../module/qeth</code> is suggesting the <code>qeth</code> driver is responsible
for this interface. Now looking at <code>lsmod | grep qe</code> shows:</p>
<div class="crt scanlines" aria-hidden="true">
<pre class="z-code"><code><span class="z-text z-plain">...
</span><span class="z-text z-plain">qeth_l3 65536 0
</span><span class="z-text z-plain">qeth_l2 57344 1
</span><span class="z-text z-plain">bridge 348160 1 qeth_l2
</span><span class="z-text z-plain">qeth 163840 2 qeth_l3,qeth_l2
</span><span class="z-text z-plain">qdio 61440 2 qeth,qeth_l3
</span><span class="z-text z-plain">ccwgroup 20480 1 qeth
</span><span class="z-text z-plain">...
</span></code></pre>
</div>
<p><strong>TODO provide an more elegant way how one would have come
to the conclusion thaht qeth_l2 is needed instead of seeing
it and just loading it</strong></p>
<p><strong>TODO qeth_l3 isn’t needed right. I forgot sort of need to test via kexec</strong></p>
<p>The only thing that caught my attention was <code>qeth_l2</code> and <code>qeth_l3</code> which both depend on <code>qeth</code>.
But there also is <code>ccwgroup</code> which I have never heard of but <strong>made the fatal flaw of believing
I won’t need this …</strong></p>
<p>Well I tried and tried but it just wouldn’t work. At one point loading pretty
much any module there is.
<mark>
Always keep the previously described
restrictions of actually not having a serial
console in mind
</mark></p>
<h3 id="running-alpine-z15">running alpine (Z15)</h3>
<p>Maybe alpine linux would load the proper modules or do things
I don’t know about yet. So why not give alpine a shot again then:</p>
<p><strong>TODO test V</strong></p>
<pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> TODO test this actually please</span><span class="z-comment z-line z-number-sign z-shell">
</span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">kexec</span></span><span class="z-meta z-function-call z-arguments z-shell"> vmlinuz-lts <span class="z-punctuation z-separator z-continuation z-line z-shell">\
</span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>initrd</span><span class="z-keyword z-operator z-assignment z-option z-shell">=</span>initramfs-lts <span class="z-punctuation z-separator z-continuation z-line z-shell">\
</span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>cmdline</span><span class="z-keyword z-operator z-assignment z-option z-shell">=</span><span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">"</span>ip=148.100.85.113::148.100.84.1:255.255.254.0:nixos:eth0::9.9.9.9:8.8.8.8 alpine_repo=https://dl-cdn.alpinelinux.org/alpine/v3.19/main modloop=https://dl-cdn.alpinelinux.org/alpine/v3.19/releases/s390x/netboot/modloop-lts<span class="z-punctuation z-definition z-string z-end z-shell">"</span></span></span>
</span></code></pre>
<h4 id="what-if-i-kexec-d-whats-already-running">what if I kexec’d whats already running?!</h4>
<p>It wasn’t long until I suspected that kexec simply
doesn’t work. While s390x <a href="https://github.com/torvalds/linux/blob/2014c95afecee3e76ca4a56956a936e23283f05b/arch/s390/Kconfig#L283">has kexec support</a> maybe the hardware
or maybe rather hardware configuration somehow prevents it. You never know
its a mainframe after all. Only one way to find out:</p>
<p><strong>TODO show abstracts of the original zipl config from IBMs default OS choices</strong></p>
<p>So assuming these details are correct lets just call <code>kexec</code> in the same way
the pre-installed zipl installation called kernel+initrd+cmdline:</p>
<pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">kexec</span></span><span class="z-meta z-function-call z-arguments z-shell"> ./TODO show the command</span>
</span></code></pre>
<p>as I was anxiously awaiting an icmp reply after running the previous command
made it stop replying to pings for obvious reasons <br>…<br></p>
<p><mark><strong>Its up!</strong> and responding to icmp!!</mark>
<small>(followed by
a weirdly long period of silence that got me slightly worried but
eventually it was allowing me to log in via ssh)
</small></p>
<p>Ok so its safe to say that my fear of <strong>kexec NOT working</strong> was <strong>proven wrong</strong>,
great!</p>
<blockquote class="note">
<p class="alert-title">
<i class="icon"></i>Note</p>
<p>In such a situation it may be smart to append a tiny exta line to your
cmdline to make sure your machine actually booted trough kexec instead of
somehow having just rebooted which in this case would have had the same result
but appending a tiny piece to the cmdline allows us to check <code>/proc/cmdline</code>
and compare it to the default invoked by a reboot</p>
</blockquote>
<p>So what could be making not just my initrd but also alpine fail? In alpines
case I can’t even argue that a driver may be missing as I’d be rather sure
that they thought of that. Its most definitely should be possible to run this
places other than just qemu I figured.</p>
<h3 id="alpines-s390x-net-option">alpines <code>s390x_net=</code> option</h3>
<p>I ended up asking IBM but before they got a chance to reply, a friend pointed
out to me that there is a kernel cmdline option called <code>s390x_net=</code>. I never heard of
this option though. So I started digging. Since I couldn’t find it listed in any
official kernel resources <a href="https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html">documenting the kernel parameters</a>
the only option left would be that the init takes care of it.</p>
<p>As previously pointed out <code>/sys/class/net/eth0/device</code> in my case linked to <code>0.0.1000</code>
like we can see here:</p>
<p><code>lrwxrwxrwx 1 root root 0 Feb 3 01:19 device -> ../../../0.0.1000</code></p>
<p>and running <code>ls -la /sys/devices/qeth</code> reveals:</p>
<div class="crt scanlines" aria-hidden="true">
<pre class="z-code"><code><span class="z-text z-plain">drwxr-xr-x 3 root root 0 Feb 3 17:05 .
</span><span class="z-text z-plain">drwxr-xr-x 14 root root 0 Nov 10 12:04 ..
</span><span class="z-text z-plain">drwxr-xr-x 5 root root 0 Jan 14 13:02 0.0.1000
</span><span class="z-text z-plain">lrwxrwxrwx 1 root root 0 Feb 3 17:05 module -> ../../module/qeth
</span><span class="z-text z-plain">-rw-r--r-- 1 root root 4096 Feb 3 17:05 uevent
</span></code></pre>
</div>
<p>it made sense to assume this should be set to <code>0.0.1000</code>. So I gave it a shot setting
<code>s390x_net=0.0.1000</code> but nope. Googling the option dind’t really seem to help and
I don’t know of many alpine linux resources for s390x.</p>
<p>By now I realized that its probably not the best idea to continue digging here if I don’t even
know if NixOS or alpine actually boot. As in that case even if I set the correct option here
but some other part of the boot process would fail. I’d not even be able to tell <small>(as
pointed out earlier) serial isn’t an option in this specific case</small></p>
<p><strong>TODO point out that for qeth its always 3 id’s VV</strong></p>
<ul>
<li>read</li>
<li>write</li>
<li>data bus ID of your virtual network card</li>
</ul>
<h3 id="unconventional-logging-methods">unconventional logging methods</h3>
<p>I’d need some way of knowing the state the system is in post kexec. With network/serial
based methods being out of the question, one of the things that remained was the disk.</p>
<h4 id="dasd-direct-access-storage-devices">dasd - (Direct Access Storage Devices)</h4>
<p><strong>TODO reformat and remove the ></strong></p>
<blockquote>
<p>While configuring storage in my case (dasd)(<strong>TODO explain it a lil and link</strong>)
was still a little simpler than just loading the right modules and having a <code>/dev</code> node
for the blockdevice show up. It had one advantage over <code>s390x_net=</code> which is that
the <code>dasd</code> kernel commandline parameter is much more common to encounter.</p>
</blockquote>
<p>For example the <a href="https://wiki.gentoo.org/wiki/S390/Install">gentoo S390 install guide</a>
provides a sample zipl configuration <code>/etc/zipl.conf</code>:</p>
<pre class="z-code"><code><span class="z-text z-plain">[defaultboot]
</span><span class="z-text z-plain">defaultmenu = menu
</span><span class="z-text z-plain">
</span><span class="z-text z-plain">[Gentoo]
</span><span class="z-text z-plain"> image = /boot/image
</span><span class="z-text z-plain"> target = /boot/zipl
</span><span class="z-text z-plain"> parameters = "dasd=0150 root=/dev/dasda1 rootfstype=ext4 TERM=dumb net.ifnames=0"
</span><span class="z-text z-plain">
</span><span class="z-text z-plain">:menu
</span><span class="z-text z-plain"> default = 1
</span><span class="z-text z-plain"> prompt = 1
</span><span class="z-text z-plain"> target = /boot/zipl
</span><span class="z-text z-plain"> timeout = 10
</span><span class="z-text z-plain"> 1 = Gentoo
</span></code></pre>
<p>with <code>dasd=0150</code> being what I’m after. Of course this wouldn’t be <code>0150</code> but looking in
<code>/sys/class/block/dasda/device -> ../../../0.0.0100</code> suggestes giving <code>0.0.0100</code> a shot.</p>
<blockquote class="note">
<p class="alert-title">
<i class="icon"></i>Note</p>
<p>In the hardware configurations encountered in this article
both <code>0.0.0100</code> and <code>0.0.1000</code> are brought up.
Make sure to not confuse them by accident</p>
<p><small>(yes this happened to me)</small></p>
</blockquote>
<p>So lets get our kexec’s <code>--append</code>’s (kernel commandline) statement to include <code>dasd=0.0.0100</code></p>
<pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">kexec</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>l</span> ./bl0v3s-s390x-nixos-kernel <span class="z-punctuation z-separator z-continuation z-line z-shell">\
</span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>initrd</span><span class="z-keyword z-operator z-assignment z-option z-shell">=</span>./bl0v3s-s390x-nixos-initrd <span class="z-punctuation z-separator z-continuation z-line z-shell">\
</span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>append</span><span class="z-keyword z-operator z-assignment z-option z-shell">=</span><span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">"</span>dasd=0.0.0100<span class="z-punctuation z-definition z-string z-end z-shell">"</span></span></span>
</span></code></pre>
<blockquote>
<p>I left out the network configuration because it didn’t work for now either way and
I wanted to reduce the amount of things that could somehow affect the boot process
(<small>not just slowing it down</small>) . But it probably would have been fine to leave
it there its simply good practice.</p>
</blockquote>
<blockquote class="tip">
<p class="alert-title">
<i class="icon"></i>Tip</p>
<p>Unlike <code>s390x_net=</code> the <code>dasd=</code> kernel commandline option is actually handled by the kernel.
Thus you can make the kernel online it without the initrd/initramfs being involved yet</p>
<p><strong>TODO factcheck this once more (memory sucks)</strong></p>
</blockquote>
<p>in the linux kernel source tree <code>/drivers/s390/block/dasd.c</code> <strong>TODO permalink</strong> we can
see a notice regarding the <code>dasd=</code> kernel commandline parameter:</p>
<pre data-lang="C" class="language-C z-code"><code class="language-C" data-lang="C"><span class="z-source z-c"><span class="z-comment z-block z-c"><span class="z-punctuation z-definition z-comment z-c">/*</span>
</span></span><span class="z-source z-c"><span class="z-comment z-block z-c"> <span class="z-punctuation z-definition z-comment z-c">*</span> Initial attempt at a probe function. this can be simplified once
</span></span><span class="z-source z-c"><span class="z-comment z-block z-c"> <span class="z-punctuation z-definition z-comment z-c">*</span> the other detection code is gone.
</span></span><span class="z-source z-c"><span class="z-comment z-block z-c"> <span class="z-punctuation z-definition z-comment z-c">*/</span></span>
</span><span class="z-source z-c"><span class="z-storage z-type z-c">int</span> <span class="z-meta z-function z-c"><span class="z-entity z-name z-function z-c">dasd_generic_probe</span></span><span class="z-meta z-function z-parameters z-c"><span class="z-meta z-group z-c"><span class="z-punctuation z-section z-group z-begin z-c">(</span></span></span><span class="z-meta z-function z-parameters z-c"><span class="z-meta z-group z-c"><span class="z-storage z-type z-c">struct</span> ccw_device <span class="z-keyword z-operator z-c">*</span><span class="z-variable z-parameter z-c">cdev</span><span class="z-punctuation z-section z-group z-end z-c">)</span></span></span><span class="z-meta z-function z-c">
</span></span><span class="z-source z-c"><span class="z-meta z-function z-c"></span><span class="z-meta z-function z-c"><span class="z-meta z-block z-c"><span class="z-punctuation z-section z-block z-begin z-c">{</span></span></span><span class="z-meta z-function z-c"><span class="z-meta z-block z-c">
</span></span></span><span class="z-source z-c"><span class="z-meta z-function z-c"><span class="z-meta z-block z-c"> cdev<span class="z-punctuation z-accessor z-c">-></span>handler <span class="z-keyword z-operator z-assignment z-c">=</span> <span class="z-keyword z-operator z-c">&</span>dasd_int_handler<span class="z-punctuation z-terminator z-c">;</span>
</span></span></span><span class="z-source z-c"><span class="z-meta z-function z-c"><span class="z-meta z-block z-c">
</span></span></span><span class="z-source z-c"><span class="z-meta z-function z-c"><span class="z-meta z-block z-c"> <span class="z-comment z-block z-c"><span class="z-punctuation z-definition z-comment z-c">/*</span>
</span></span></span></span><span class="z-source z-c"><span class="z-meta z-function z-c"><span class="z-meta z-block z-c"><span class="z-comment z-block z-c"> <span class="z-punctuation z-definition z-comment z-c">*</span> Automatically online either all dasd devices (dasd_autodetect)
</span></span></span></span><span class="z-source z-c"><span class="z-meta z-function z-c"><span class="z-meta z-block z-c"><span class="z-comment z-block z-c"> <span class="z-punctuation z-definition z-comment z-c">*</span> or all devices specified with dasd= parameters during
</span></span></span></span><span class="z-source z-c"><span class="z-meta z-function z-c"><span class="z-meta z-block z-c"><span class="z-comment z-block z-c"> <span class="z-punctuation z-definition z-comment z-c">*</span> initial probe.
</span></span></span></span><span class="z-source z-c"><span class="z-meta z-function z-c"><span class="z-meta z-block z-c"><span class="z-comment z-block z-c"> <span class="z-punctuation z-definition z-comment z-c">*/</span></span>
</span></span></span><span class="z-source z-c"><span class="z-meta z-function z-c"><span class="z-meta z-block z-c"> <span class="z-keyword z-control z-c">if</span> <span class="z-meta z-group z-c"><span class="z-punctuation z-section z-group z-begin z-c">(</span><span class="z-meta z-group z-c"><span class="z-punctuation z-section z-group z-begin z-c">(</span><span class="z-meta z-function-call z-c"><span class="z-variable z-function z-c">dasd_get_feature</span><span class="z-meta z-group z-c"><span class="z-punctuation z-section z-group z-begin z-c">(</span></span></span><span class="z-meta z-function-call z-c"><span class="z-meta z-group z-c">cdev<span class="z-punctuation z-separator z-c">,</span> DASD_FEATURE_INITIAL_ONLINE</span></span><span class="z-meta z-function-call z-c"><span class="z-meta z-group z-c"><span class="z-punctuation z-section z-group z-end z-c">)</span></span></span> <span class="z-keyword z-operator z-comparison z-c">></span> <span class="z-constant z-numeric z-integer z-decimal z-c">0</span> <span class="z-punctuation z-section z-group z-end z-c">)</span></span> <span class="z-keyword z-operator z-arithmetic z-c">||</span>
</span></span></span></span><span class="z-source z-c"><span class="z-meta z-function z-c"><span class="z-meta z-block z-c"><span class="z-meta z-group z-c"> <span class="z-meta z-group z-c"><span class="z-punctuation z-section z-group z-begin z-c">(</span>dasd_autodetect <span class="z-keyword z-operator z-arithmetic z-c">&&</span> <span class="z-meta z-function-call z-c"><span class="z-variable z-function z-c">dasd_busid_known</span><span class="z-meta z-group z-c"><span class="z-punctuation z-section z-group z-begin z-c">(</span></span></span><span class="z-meta z-function-call z-c"><span class="z-meta z-group z-c"><span class="z-meta z-function-call z-c"><span class="z-variable z-function z-c">dev_name</span><span class="z-meta z-group z-c"><span class="z-punctuation z-section z-group z-begin z-c">(</span></span></span><span class="z-meta z-function-call z-c"><span class="z-meta z-group z-c"><span class="z-keyword z-operator z-c">&</span>cdev<span class="z-punctuation z-accessor z-c">-></span>dev</span></span><span class="z-meta z-function-call z-c"><span class="z-meta z-group z-c"><span class="z-punctuation z-section z-group z-end z-c">)</span></span></span></span></span><span class="z-meta z-function-call z-c"><span class="z-meta z-group z-c"><span class="z-punctuation z-section z-group z-end z-c">)</span></span></span> <span class="z-keyword z-operator z-comparison z-c">!=</span> <span class="z-constant z-numeric z-integer z-decimal z-c">0</span><span class="z-punctuation z-section z-group z-end z-c">)</span></span><span class="z-punctuation z-section z-group z-end z-c">)</span></span>
</span></span></span><span class="z-source z-c"><span class="z-meta z-function z-c"><span class="z-meta z-block z-c"> <span class="z-meta z-function-call z-c"><span class="z-variable z-function z-c">async_schedule</span><span class="z-meta z-group z-c"><span class="z-punctuation z-section z-group z-begin z-c">(</span></span></span><span class="z-meta z-function-call z-c"><span class="z-meta z-group z-c">dasd_generic_auto_online<span class="z-punctuation z-separator z-c">,</span> cdev</span></span><span class="z-meta z-function-call z-c"><span class="z-meta z-group z-c"><span class="z-punctuation z-section z-group z-end z-c">)</span></span></span><span class="z-punctuation z-terminator z-c">;</span>
</span></span></span><span class="z-source z-c"><span class="z-meta z-function z-c"><span class="z-meta z-block z-c"> <span class="z-keyword z-control z-flow z-return z-c">return</span> <span class="z-constant z-numeric z-integer z-decimal z-c">0</span><span class="z-punctuation z-terminator z-c">;</span>
</span></span></span><span class="z-source z-c"><span class="z-meta z-function z-c"><span class="z-meta z-block z-c"></span></span><span class="z-meta z-function z-c"><span class="z-meta z-block z-c"><span class="z-punctuation z-section z-block z-end z-c">}</span></span></span>
</span></code></pre>
<blockquote class="important">
<p class="alert-title">
<i class="icon"></i>Important</p>
<p><strong>TODO explain autodetect and ccw and whatever to point out that you
have to use dasd= (or udev effectively trough chzdev from s390-tools
or what it was again)</strong></p>
<p>(<strong>TODO figure out how the autodetect works and maybe explain it</strong>)
<strong>online all devices“</strong></p>
<p>/sys/bus/ccw/drivers/ <a href="https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/installation_guide/chap-post-installation-configuration-s390#sect-post-installation-adding-dasds-s390">this page</a></p>
<p><strong>TODO point out how I initially onlined it using echo 1> …. /online in my initrd init script</strong></p>
</blockquote>
<p><strong>TODO show how I modified my initrd to mount the dasd and first write hello world and later on some state.
Mention why I first checked if the dasd works before instantly dumping network crap</strong></p>
<p>Well remember alpines <code>s390x_net=</code> that I didn’t want to play around with until I
found some (admittedly crude!) way of getting some form of log output. Now that retriving the logs
works I guess a sane question would be:</p>
<p><q>“why don’t you just look at the code?”<small>(its probably going to take less time than finding a resource explaining this)</small></q></p>
<p><strong>TODO use permalinks</strong></p>
<p>so looking into how alpine creates their intiramfs’s I came to the conclusion that
<a href="https://gitlab.alpinelinux.org/alpine/mkinitfs">alpines mkinitfs repo</a> would be the place to look.</p>
<p>The <a href="https://gitlab.alpinelinux.org/alpine/mkinitfs/-/blob/master/initramfs-init.in">initramfs-init.in</a> file contains what I was
looking for. At <a href="https://gitlab.alpinelinux.org/alpine/mkinitfs/-/blob/master/initramfs-init.in#L471">line 471</a> the <code>myopts=</code>
variable is being introduced. Containing various options some commonly known like
<code>ip</code> <small>(remember the <code>ip=</code> statement)</small>. Other than that for example <code>ssh_key</code> one can use to make for example the netboot images
we used earlier retrive that from a remote resource so we don’t have to bake our own kernels/initrd’s. <mark>Line 530 to 536</mark>
would then make sense of <code>myopts</code> by effectively doing the following:</p>
<p><strong>TODO explain that its breaking these down and assigning this to the KOPT shell variables</strong></p>
<pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-keyword z-control z-loop z-for z-shell">for</span><span class="z-meta z-group z-for z-shell"> i <span class="z-keyword z-control z-in z-shell">in</span> <span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">myopts</span></span></span><span class="z-keyword z-operator z-logical z-continue z-shell">;</span> <span class="z-keyword z-control z-loop z-do z-shell">do</span>
</span><span class="z-source z-shell z-bash"> <span class="z-meta z-conditional z-case z-shell"><span class="z-keyword z-control z-conditional z-case z-shell">case</span> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">"</span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">opt</span></span><span class="z-punctuation z-definition z-string z-end z-shell">"</span></span> <span class="z-keyword z-control z-in z-shell">in</span>
</span></span><span class="z-source z-shell z-bash"><span class="z-meta z-conditional z-case z-shell"> </span><span class="z-meta z-conditional z-case z-clause z-patterns z-shell"><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">i</span></span>=<span class="z-keyword z-operator z-regexp z-quantifier z-shell">*</span><span class="z-meta z-conditional z-case z-shell"><span class="z-keyword z-control z-conditional z-patterns z-end z-shell">)</span></span></span><span class="z-meta z-conditional z-case z-clause z-commands z-shell"> <span class="z-meta z-function-call z-shell"><span class="z-support z-function z-eval z-shell">eval</span></span><span class="z-meta z-function-call z-arguments z-shell"> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">"</span>KOPT_<span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-punctuation z-section z-expansion z-parameter z-begin z-shell">{</span></span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-variable z-other z-readwrite z-shell">i</span></span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-section z-expansion z-parameter z-end z-shell">}</span></span><span class="z-punctuation z-definition z-string z-end z-shell">"</span></span>=<span class="z-string z-quoted z-single z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">'</span>${opt#*=}<span class="z-punctuation z-definition z-string z-end z-shell">'</span></span></span></span><span class="z-meta z-conditional z-case z-clause z-commands z-shell"><span class="z-punctuation z-terminator z-case z-clause z-shell">;;</span></span><span class="z-meta z-conditional z-case z-clause z-patterns z-shell">
</span></span><span class="z-source z-shell z-bash"><span class="z-meta z-conditional z-case z-clause z-patterns z-shell"> </span><span class="z-meta z-conditional z-case z-clause z-patterns z-shell"><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">i</span></span><span class="z-meta z-conditional z-case z-clause z-patterns z-shell"><span class="z-keyword z-control z-conditional z-patterns z-end z-shell">)</span></span></span><span class="z-meta z-conditional z-case z-clause z-commands z-shell"> <span class="z-meta z-function-call z-shell"><span class="z-support z-function z-eval z-shell">eval</span></span><span class="z-meta z-function-call z-arguments z-shell"> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">"</span>KOPT_<span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-punctuation z-section z-expansion z-parameter z-begin z-shell">{</span></span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-variable z-other z-readwrite z-shell">i</span></span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-section z-expansion z-parameter z-end z-shell">}</span></span>=yes<span class="z-punctuation z-definition z-string z-end z-shell">"</span></span></span></span><span class="z-meta z-conditional z-case z-clause z-commands z-shell"><span class="z-punctuation z-terminator z-case z-clause z-shell">;;</span></span><span class="z-meta z-conditional z-case z-clause z-patterns z-shell">
</span></span><span class="z-source z-shell z-bash"><span class="z-meta z-conditional z-case z-clause z-patterns z-shell"> </span><span class="z-meta z-conditional z-case z-clause z-patterns z-shell">no<span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">i</span></span><span class="z-meta z-conditional z-case z-clause z-patterns z-shell"><span class="z-keyword z-control z-conditional z-patterns z-end z-shell">)</span></span></span><span class="z-meta z-conditional z-case z-clause z-commands z-shell"> <span class="z-meta z-function-call z-shell"><span class="z-support z-function z-eval z-shell">eval</span></span><span class="z-meta z-function-call z-arguments z-shell"> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">"</span>KOPT_<span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-punctuation z-section z-expansion z-parameter z-begin z-shell">{</span></span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-variable z-other z-readwrite z-shell">i</span></span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-section z-expansion z-parameter z-end z-shell">}</span></span>=no<span class="z-punctuation z-definition z-string z-end z-shell">"</span></span></span></span><span class="z-meta z-conditional z-case z-clause z-commands z-shell"><span class="z-punctuation z-terminator z-case z-clause z-shell">;;</span></span><span class="z-meta z-conditional z-case z-clause z-patterns z-shell">
</span></span><span class="z-source z-shell z-bash"><span class="z-meta z-conditional z-case z-clause z-patterns z-shell"> <span class="z-keyword z-control z-conditional z-end z-shell">esac</span></span>
</span><span class="z-source z-shell z-bash"><span class="z-keyword z-control z-loop z-end z-shell">done</span>
</span></code></pre>
<p>Further down starting <a href="https://gitlab.alpinelinux.org/alpine/mkinitfs/-/blob/master/initramfs-init.in#L607">at line 607</a>
we can see the following:</p>
<pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-keyword z-control z-conditional z-if z-shell">if</span> <span class="z-support z-function z-test z-begin z-shell">[</span><span class="z-meta z-function-call z-arguments z-shell"> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">"</span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-punctuation z-section z-expansion z-parameter z-begin z-shell">{</span></span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-variable z-other z-readwrite z-shell">KOPT_s390x_net<span class="z-keyword z-operator z-expansion z-shell">%%</span></span></span><span class="z-meta z-group z-expansion z-parameter z-shell">,<span class="z-keyword z-operator z-regexp z-quantifier z-shell">*</span></span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-section z-expansion z-parameter z-end z-shell">}</span></span><span class="z-punctuation z-definition z-string z-end z-shell">"</span></span> <span class="z-keyword z-operator z-logical z-shell">=</span> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">"</span>qeth_l2<span class="z-punctuation z-definition z-string z-end z-shell">"</span></span> <span class="z-support z-function z-test z-end z-shell">]</span></span><span class="z-keyword z-operator z-logical z-continue z-shell">;</span> <span class="z-keyword z-control z-conditional z-then z-shell">then</span>
</span><span class="z-source z-shell z-bash"> <span class="z-keyword z-control z-loop z-for z-shell">for</span><span class="z-meta z-group z-for z-shell"> mod <span class="z-keyword z-control z-in z-shell">in</span> qeth qeth_l2 qeth_l3</span><span class="z-keyword z-operator z-logical z-continue z-shell">;</span> <span class="z-keyword z-control z-loop z-do z-shell">do</span>
</span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell"><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">MOCK</span></span></span></span><span class="z-meta z-function-call z-arguments z-shell"> modprobe <span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">mod</span></span></span>
</span><span class="z-source z-shell z-bash"> <span class="z-keyword z-control z-loop z-end z-shell">done</span>
</span><span class="z-source z-shell z-bash"> <span class="z-variable z-other z-readwrite z-assignment z-shell">_channel</span><span class="z-keyword z-operator z-assignment z-shell">=</span><span class="z-string z-unquoted z-shell"><span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">"</span><span class="z-meta z-group z-expansion z-command z-parens z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-punctuation z-section z-parens z-begin z-shell">(</span><span class="z-meta z-function-call z-shell"><span class="z-support z-function z-echo z-shell">echo</span></span><span class="z-meta z-function-call z-arguments z-shell"> <span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-punctuation z-section z-expansion z-parameter z-begin z-shell">{</span></span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-variable z-other z-readwrite z-shell">KOPT_s390x_net<span class="z-keyword z-operator z-expansion z-shell">#</span></span></span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-keyword z-operator z-regexp z-quantifier z-shell">*</span>,</span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-section z-expansion z-parameter z-end z-shell">}</span></span></span> <span class="z-keyword z-operator z-logical z-pipe z-shell">|</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">tr</span></span><span class="z-meta z-function-call z-arguments z-shell"> <span class="z-keyword z-control z-regexp z-set z-begin z-shell">[</span>A<span class="z-keyword z-operator z-word z-shell">-</span>Z<span class="z-keyword z-control z-regexp z-set z-end z-shell">]</span> <span class="z-keyword z-control z-regexp z-set z-begin z-shell">[</span>a<span class="z-keyword z-operator z-word z-shell">-</span>z<span class="z-keyword z-control z-regexp z-set z-end z-shell">]</span></span><span class="z-punctuation z-section z-parens z-end z-shell">)</span></span><span class="z-punctuation z-definition z-string z-end z-shell">"</span></span></span>
</span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-support z-function z-echo z-shell">echo</span></span><span class="z-meta z-function-call z-arguments z-shell"> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">"</span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">_channel</span></span><span class="z-punctuation z-definition z-string z-end z-shell">"</span></span> <span class="z-keyword z-operator z-assignment z-redirection z-shell">></span> /sys/bus/ccwgroup/drivers/qeth/group</span>
</span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-support z-function z-echo z-shell">echo</span></span><span class="z-meta z-function-call z-arguments z-shell"> 1 <span class="z-keyword z-operator z-assignment z-redirection z-shell">></span> /sys/bus/ccwgroup/drivers/qeth/<span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">"</span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-punctuation z-section z-expansion z-parameter z-begin z-shell">{</span></span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-variable z-other z-readwrite z-shell">_channel<span class="z-keyword z-operator z-expansion z-shell">%%</span></span></span><span class="z-meta z-group z-expansion z-parameter z-shell">,<span class="z-keyword z-operator z-regexp z-quantifier z-shell">*</span></span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-section z-expansion z-parameter z-end z-shell">}</span></span><span class="z-punctuation z-definition z-string z-end z-shell">"</span></span>/layer2</span>
</span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-support z-function z-echo z-shell">echo</span></span><span class="z-meta z-function-call z-arguments z-shell"> 1 <span class="z-keyword z-operator z-assignment z-redirection z-shell">></span> /sys/bus/ccwgroup/drivers/qeth/<span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">"</span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-punctuation z-section z-expansion z-parameter z-begin z-shell">{</span></span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-variable z-other z-readwrite z-shell">_channel<span class="z-keyword z-operator z-expansion z-shell">%%</span></span></span><span class="z-meta z-group z-expansion z-parameter z-shell">,<span class="z-keyword z-operator z-regexp z-quantifier z-shell">*</span></span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-section z-expansion z-parameter z-end z-shell">}</span></span><span class="z-punctuation z-definition z-string z-end z-shell">"</span></span>/online</span>
</span><span class="z-source z-shell z-bash"><span class="z-keyword z-control z-conditional z-end z-shell">fi</span>
</span></code></pre>
<p>So looking at this step by step
the first thing to take out of this is that
the condition defined by:</p>
<p><code>"${KOPT_s390x_net%%,*}" = "qeth_l2"</code></p>
<p>Has to be met in order to load the <code>qeth</code>, <code>qeth_l2</code>, <code>qeth_l3</code>
modules. This means that in alpine at least in any case we’d
have to specify <code>s390x_net=qeth_l2</code>. Before specifying any
ids like <code>0.0.1000</code>. Thus writing: <code>s390x_net=0.0.1000</code> can’t
actually have any effect, even if <code>qeth_l2</code> missing wouldn’t be the only issue.</p>
<p>the next line is:</p>
<p><strong>TODO include kernel source code that made me discover it</strong></p>
<p><strong>TODO I think when you echo anything other than 3 ids in there
this failed in the nixos initrd logs dumped to dasd but
confirm it</strong></p>
<p><code>_channel="$(echo ${KOPT_s390x_net#*,} | tr [A-Z] [a-z])"</code></p>
<p>which is essentially being stripped from <code>qeth_l2</code> before
being written to the ccwgroup+qeth drivers sysfs path like</p>
<p><code>echo "$_channel" > /sys/bus/ccwgroup/drivers/qeth/group</code></p>
<p>Sadly you can’t cat this file to see what the distros provided
by IBM put in there. While the kernel doc actually gives clues
on what this driver is for, before I managed to come across that
a friend once again pointed out that I should give
<code>s390x_net=qeth_l2,0.0.1000,0.0.1001,0.0.1002</code> a try.</p>
<p><strong>TODO <a href="https://www.kernel.org/doc/html/v5.5/s390/driver-model.html#ccwgroup-devices">link this</a>
or maybe rater this https://docs.kernel.org/driver-api/s390-drivers.html#the-ccwgroup-bus</strong></p>
<p>So it turns out that unlike seen before with the <code>dasd=0.0.0100</code> driver option.
The qeth driver along other networking devices actually consists out of multiple (3x) ccw devices.
One for each channel:</p>
<ul>
<li>read</li>
<li>write</li>
<li>data</li>
</ul>
<p><strong>(^ TODO explain these more indepth)</strong></p>
<p>IDs are typically handed out in the pattern like seen above.</p>
<p><strong>TODO show how youd obtain these using tools from s390-tools or
maybe sysfs</strong></p>
<p>Now after having placed a few values into <code>/sys/bus/ccwgroup/drivers/qeth/group</code>
to no success. <code>echo "0.0.1000,0.0.1001,0.0.1002" > /sys/bus/ccwgroup/drivers/qeth/group</code>
did finally work.</p>
<p>The last 2 lines left in alpines init are</p>
<pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-support z-function z-echo z-shell">echo</span></span><span class="z-meta z-function-call z-arguments z-shell"> 1 <span class="z-keyword z-operator z-assignment z-redirection z-shell">></span> /sys/bus/ccwgroup/drivers/qeth/<span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">"</span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-punctuation z-section z-expansion z-parameter z-begin z-shell">{</span></span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-variable z-other z-readwrite z-shell">_channel<span class="z-keyword z-operator z-expansion z-shell">%%</span></span></span><span class="z-meta z-group z-expansion z-parameter z-shell">,<span class="z-keyword z-operator z-regexp z-quantifier z-shell">*</span></span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-section z-expansion z-parameter z-end z-shell">}</span></span><span class="z-punctuation z-definition z-string z-end z-shell">"</span></span>/layer2</span>
</span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-support z-function z-echo z-shell">echo</span></span><span class="z-meta z-function-call z-arguments z-shell"> 1 <span class="z-keyword z-operator z-assignment z-redirection z-shell">></span> /sys/bus/ccwgroup/drivers/qeth/<span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">"</span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-punctuation z-section z-expansion z-parameter z-begin z-shell">{</span></span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-variable z-other z-readwrite z-shell">_channel<span class="z-keyword z-operator z-expansion z-shell">%%</span></span></span><span class="z-meta z-group z-expansion z-parameter z-shell">,<span class="z-keyword z-operator z-regexp z-quantifier z-shell">*</span></span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-section z-expansion z-parameter z-end z-shell">}</span></span><span class="z-punctuation z-definition z-string z-end z-shell">"</span></span>/online</span>
</span></code></pre>
<p>which are pretty self explainatory if you keep in mind that
<code>"${_channel%%,*}"</code> is just gonna take the first bus id from the 3 ids
passd. Thus <code>0.0.1000,0.0.1001,0.0.1002</code> becomes <code>0.0.1000</code> and
the resulting path would be just <code>/sys/bus/ccwgroup/devices/0.0.1000/</code>.</p>
<blockquote class="note">
<p class="alert-title">
<i class="icon"></i>Note</p>
<p><code>/sys/bus/ccwgroup/devices/$ID/</code> doesn’t exist unless you setup
the ccwgroup like explained before. The directory only appears if this
has been done previously. Also as the path only contains one ID in
our case the first one from the 3 setup via qeth’s ccwgroup you
can get the impression that there is just <code>0.0.1000</code> but actually
<code>0.0.1001</code> and <code>0.0.1002</code> are involved too.</p>
</blockquote>
<p><strong>TODO also somewhere in here mention that given how on hercules
we didn’t have to do the ccwgroup stuff I belived it was fine
to just write one address but confirm this claim</strong></p>
<blockquote class="note">
<p class="alert-title">
<i class="icon"></i>Note</p>
<p>The Z15 relies on qeth as determined earlier. If networking
was configured on hercules | z/VM’s I think you will also end
up having to use qeth. <strong>TODO confirm if z/VM only offers this</strong></p>
</blockquote>
<p><strong>TODO V strip paths, give credit and explain</strong></p>
<pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">kexec</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>l</span> /home/host/TestAndDevel/kexec390/bzImage <span class="z-punctuation z-separator z-continuation z-line z-shell">\
</span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell">--</span>initrd</span><span class="z-keyword z-operator z-assignment z-option z-shell">=</span>/home/host/TestAndDevel/kexec390/initramfs-lts <span class="z-punctuation z-separator z-continuation z-line z-shell">\
</span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell">--</span>append</span><span class="z-keyword z-operator z-assignment z-option z-shell">=</span><span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">"</span>ip=192.168.10.106:none:192.168.10.1:255.255.255.0:alpikexec:eth0:none:1.1.1.1:8.8.8.8 alpine_repo=https://dl-cdn.alpinelinux.org/alpine/edge/main modloop=https://dl-cdn.alpinelinux.org/alpine/edge/releases/s390x/netboot/modloop-lts ssh_key=https://raw.githubusercontent.com/cuzrawr/kexec390/refs/heads/main/randssh.key.pub dasd=0.0.0100 s390x_net=qeth_l2,0.0.1000,0.0.1001,0.0.1002
</span></span></span></code></pre>
<blockquote>
<p>So this should mark the last issue to overcome before we can proceed booting something on the Z15.
Though as booting isn’t installing:</p>
</blockquote>
<h2 id="installing-nixos">installing NixOS</h2>
<p>As we can now boot NixOS in qemu or even on the Z15. The next rational step would be
actually installing it. But to do that we first need a bootloader:</p>
<h3 id="supporting-the-zipl-booloader-on-nixos">supporting the zipl booloader on NixOS</h3>
<p>Unlike seen with <code>x86</code> where there are more bootloaders than let say linux distro families.
S390x / IBM Z doesn’t give us a lot to choose from. We got <a href="https://bl0v3.com/Blog/porting-nixos-to-an-ibm-mainframe/%22https://github.com/ibm-s390-linux/zipl%22">s390-tools</a>’s zipl
(Z initial program load)er. Thats it</p>
<blockquote class="note">
<p class="alert-title">
<i class="icon"></i>Note</p>
<p>GRUB2 also supports s390x but as it needs to be loaded from zipl. Thus usually one
does not gain too much by using it.</p>
<p>(Yes some boot setups benefit from it and I think
suse actually uses that. <small>but then you can also boot into a linux kernel and use
kexec to chainload stuff.</p>
<p>Effectively building your own linux-based bootloader</small>)
<strong>TODO link the article I wrote on linux based bootloaders when I finished it</strong></p>
</blockquote>
<p>Well unlike arch where you would call lets say <code>grub-install</code> yourself. Or even
on distros supporting mainframes where you’d call <code>zipl</code> in a likewise fashion.</p>
<p>When using NixOS things are a little different. You will not just be required to write a <a href="https://bl0v3.com/Blog/porting-nixos-to-an-ibm-mainframe/%22x%22">derivation TODO LINK</a> but also
a module to go with it. Which is meant to allow NixOS to track the options exposed and thus define
things in an declarative fashion like we are used to when using Nix(OS).</p>
<p>While packaging zipl wasn’t the last thing I did, I still put it down here.
I chose this in order to reduce the amount of sections the reader would have to go trough
before getting to see NixOS boot up for the first time.</p>
<p><small>(I already got it to compile in lxc before even bothering doing
anything beyond what was needed to boot zipl as I really wouldn’t
want to put in all this effort to realize I couldn’t get zipl working
under nix <strong>which almost was the case</strong> as you will see :o )</small></p>
<h3 id="writing-the-zipl-derivation-compiling-it">writing the zipl derivation (compiling it)</h3>
<p><a href="https://bl0v3.com/Blog/porting-nixos-to-an-ibm-mainframe/%22https://github.com/ibm-s390-linux/zipl%22">S390-tools</a> contains a lot of utilities/tools
other than just zipl itself. Some are more or less useless on a NixOS host, some you
can’t get around. But as I wanted to initially just focus on zipl I decided to
strip down the makefile <small>(it appears as if there is no way to be particularly selective about the
components you actually want to build)</small>. So I outfitted <a href="https://github.com/bolives-hax/nixos-s390x/blob/dbg-fm/pkgs/zipl-package.nix">my drvs</a> (TODO link to updated one) <code>patchPhase</code>
with:</p>
<pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">substituteInPlace</span></span><span class="z-meta z-function-call z-arguments z-shell"> Makefile <span class="z-punctuation z-separator z-continuation z-line z-shell">\
</span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>replace-fail</span> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">"</span>LIB_DIRS = libvtoc libzds libdasd libccw libvmcp libekmfweb <span class="z-constant z-character z-escape z-shell">\\</span><span class="z-punctuation z-definition z-string z-end z-shell">"</span></span> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">"</span>LIB_DIRS = #<span class="z-constant z-character z-escape z-shell">\\</span><span class="z-punctuation z-definition z-string z-end z-shell">"</span></span> <span class="z-punctuation z-separator z-continuation z-line z-shell">\
</span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>replace-fail</span> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">"</span>TOOL_DIRS = zipl zdump fdasd dasdfmt dasdview tunedasd <span class="z-constant z-character z-escape z-shell">\\</span><span class="z-punctuation z-definition z-string z-end z-shell">"</span></span> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">"</span>TOOL_DIRS = zipl dasdfmt netboot zdev#<span class="z-constant z-character z-escape z-shell">\\</span><span class="z-punctuation z-definition z-string z-end z-shell">"</span></span></span>
</span></code></pre>
<p><small>for a quick and hacky yet effective temporary solution</small></p>
<p>Then I ended up fighting the buildsystem for quite some time to get the dependency checks to
not fail anymore. To hit the first real issue …</p>
<p><strong>TODO mention how nix wraps compilers, binutils, and the libc.
Why it does that (like how it injects a bunch of flags)
and why it injecting flags here is a huge problem and not desired</strong></p>
<p>I didn’t expect that id run into any issues that can’t aren’t essentially
to blame on nix. But here we go: <a href="https://github.com/ibm-s390-linux/s390-tools/issues/171">linker section overlap #171</a>.</p>
<blockquote class="note">
<p class="alert-title">
<i class="icon"></i>Note</p>
<p>By now IBM actually went out of their way to fix this. So any claims made after
this notice no longer apply. But I will still include the parts I wrote before that was the case.
As actually many months passed until <a href="https://github.com/ibm-s390-linux/s390-tools/commit/605680d6fdf49807d90b6036d925232ffabaa2c9">zipl/boot: Increase section size for eckd_mv dumper</a> fixed it.</p>
</blockquote>
<p>While you can read my observations from the issue meant for the folks at IBM. I will still attempt to
break down and simplify the issue a little here as well.</p>
<h4 id="linkerscripts">linkerscripts</h4>
<p>Whats relevant to know about linkerscripts in this context is that
among other things they most importantly tell our compiler(linker) the layout it should
confirm to when “linking” our input files. Its not that common to come across them
unless you work relatively close to hardware and need a higher degree of control
over the binary outputs your compiler produces.</p>
<p>For a bootloader this is of great importance as without an operating system or similar
things around <strong>something</strong> needs to tell it where exactly to place things. For example
lets assume that the hardware loads a certain segment of code into a hardcoded memory
address and then moves execution to it. We’d have to link our executable accordingly.</p>
<p>Of course our program also needs to know the locations of resources required at runtime
like where to find the content of constant variables. Where the stack should be kept
and of corse where the executable sections are. You may even include things such as
headers/metadata which of course also need to be in the right locations.</p>
<p>Linker scripts are meant help with that. Here is an abstract from (s390-tools/zipl’s) linker script in question:</p>
<pre data-lang="lds" class="language-lds z-code"><code class="language-lds" data-lang="lds"><span class="z-text z-plain">SECTIONS
</span><span class="z-text z-plain">{
</span><span class="z-text z-plain"> . = STAGE2_DESC;
</span><span class="z-text z-plain"> __stage2_desc = .;
</span><span class="z-text z-plain">
</span><span class="z-text z-plain"> . = STAGE2_LOAD_ADDRESS;
</span><span class="z-text z-plain"> .stage2.head : { *(.stage2.head) }
</span><span class="z-text z-plain"> . = STAGE2_ENTRY;
</span><span class="z-text z-plain"> .text.start : {
</span><span class="z-text z-plain"> *(.text.start)
</span><span class="z-text z-plain"> }
</span><span class="z-text z-plain"> .text : { *(.text) }
</span><span class="z-text z-plain"> __ex_table_start = .;
</span><span class="z-text z-plain"> .ex_table : { *(.ex_table) }
</span><span class="z-text z-plain"> __ex_table_stop = .;
</span><span class="z-text z-plain"> .rodata : {*(.rodata) }
</span><span class="z-text z-plain"> .data : { *(.data) }
</span><span class="z-text z-plain"> __stage2_params = .;
</span><span class="z-text z-plain">
</span><span class="z-text z-plain"> . = 0x4ff0;
</span><span class="z-text z-plain"> .stage2dump.tail : { *(.stage2dump.tail) }
</span><span class="z-text z-plain"> . = 0x5000;
</span><span class="z-text z-plain"> .eckd2dump_mv.tail : { *(.eckd2dump_mv.tail) }
</span><span class="z-text z-plain">
</span><span class="z-text z-plain"> . = 0x5200;
</span><span class="z-text z-plain"> __bss_start = .;
</span><span class="z-text z-plain"> .bss : { *(.bss) }
</span><span class="z-text z-plain"> __bss_stop = .;
</span><span class="z-text z-plain">
</span><span class="z-text z-plain"> . = STAGE2_HEAP_ADDRESS;
</span><span class="z-text z-plain">
</span><span class="z-text z-plain"> ...
</span><span class="z-text z-plain"> ...
</span><span class="z-text z-plain"> ...
</span><span class="z-text z-plain">}
</span></code></pre>
<p>for example the following sections have the following functions:</p>
<ul>
<li>
<p><code>.text</code> <small>yes .text (this has nothing to do with ascii text)</small>
traditionally contains out executable code. As in the actual opcodes/machine code.</p>
</li>
<li>
<p><code>.rodata</code> contains static variables, be it global or local if they are static they
are “ro”=“read only” and are put here</p>
</li>
<li>
<p><code>.bss</code> (block statring symbol) contains statically allocated variables. But unlike
<code>.rodata</code> these are merely allocated meaning they haven’t been assigned values yet
as that happens at runtime.</p>
</li>
<li>
<p><code>.data</code> think <code>.rodata</code> except that its rw as in “read/write”-able</p>
</li>
</ul>
<p>The actual layout (that was causing me issues) looks like this:</p>
<pre class="z-code"><code><span class="z-text z-plain">/*
</span><span class="z-text z-plain"> * Memory layout of stage 2 for ECKD DASD dump tool
</span><span class="z-text z-plain"> * (single volume and multi volume)
</span><span class="z-text z-plain"> * ===============================================
</span><span class="z-text z-plain"> *
</span><span class="z-text z-plain"> * General memory layout
</span><span class="z-text z-plain"> * ---------------------
</span><span class="z-text z-plain"> *
</span><span class="z-text z-plain"> * 0x0000-0x1fff Lowcore
</span><span class="z-text z-plain"> * 0x2000-0xafff Sections (load): head, text, data, rodata, rodata.str,
</span><span class="z-text z-plain"> * stage2dump.tail, eckd2dump_mv.tail, bss
</span><span class="z-text z-plain"> * 0xb000-0xdfff Memory allocation (heap)
</span><span class="z-text z-plain"> * 0xe000-0xffff Stack
</span><span class="z-text z-plain"> *
</span><span class="z-text z-plain"> * Special memory locations
</span><span class="z-text z-plain"> * ------------------------
</span><span class="z-text z-plain"> *
</span><span class="z-text z-plain"> * 0x78 Stage 2 description parameters
</span><span class="z-text z-plain"> * 0x2018 Stage 2 entry point
</span><span class="z-text z-plain"> * 0x4ff0 Stage 2 multi-volume dump parameters (eckd2dump_mv)
</span><span class="z-text z-plain"> * 0x5000 Multi-volume dump parameters table (eckd2dump_mv)
</span><span class="z-text z-plain"> * 0x9ff0 Stage 2 single volume dump parameters (eckd2dump_sv)
</span><span class="z-text z-plain"> */
</span></code></pre>
<p>In the linkerscript abstract shown above there is on line thats of particular interest
to us. And its <code>. = 0x4ff0;</code> followed by <code>.stage2dump.tail : { *(.stage2dump.tail) }</code>.</p>
<p>So as the comment describing the layout in a human readable fashion
states <code>0x2018 Stage 2 entry point</code> and <code>0x4ff0 Stage 2 multi-volume dump parameters (eckd2dump_mv)</code>.
We can say that at <code>0x2018</code> stage 2 pickups execution leaving space up until <code>0x4ff0</code> where
the multi volume dump parameters are located. Thus <code>0x4ff0 - 0x2018 = 0x2fd8</code> gives us
<code>11248 bytes</code> of space to place our executable code before another non executable section
follows.</p>
<h4 id="whats-the-problem-and-why-did-i-run-into-it">whats the problem (and why did I run into it)?</h4>
<p>What needs to be said here is as I wanted to run this on the Z10, I chose to
compile it with the Z10’s (cpu) set as the target. (As opposed to z13-z16 which are whats commonly
used at the time of writing this). The problem though is that quoting <a href="https://github.com/ibm-s390-linux/s390-tools/issues/171#issuecomment-2258137192">sharkcz’s comment</a></p>
<p><q>in Fedora we are now at z13 as the arch level, we were on zEC12 for a long time and on z10 before that“</q></p>
<p>with the last z10 being from 2008 its safe to assume that development is most likely done on
Z13 or even Z15 as IBM operates a Z15 for selected open source projects to use. Even I do so. But
unlike the other devs I guess what sets me apart from them is wanting to run this on the Z10
as that is the machine I actually have sitting around locally.</p>
<p><small>Running things on real hardware. Sitting right in front of you remains special in my eyes.
Cloud services lack this magical feeling real hardware provides :3</small></p>
<p>But how does this now play into the issue? Wouldn’t zipl compile just fine for the Z10? There shouldn’t
really anything explicitly preventing it. While many more features got added none of them
appear to explicitly be incompatible with the Z10 architecture.</p>
<p>The issue is much simpler. Its not a specific
feature rather its that there are <strong>“many features”</strong> now resulting in more code to be compiled
and consequently more machine code to be emitted.</p>
<p>Now remember the fact that we “only” got <code>11248 bytes</code>. But that still doesn’t answer why this works for the Z13-Z15.
Well the thing is while the size limit remains the same for every IBM Z series mainframe. The
compilers <code>-march=</code> optimisation flag does have a bigger and bigger effect when compiling with
<code>-Os</code> / <code>-Oz</code> optimizing for size. With s390x being a
<a href="https://github.com/ibm-s390-linux/s390-tools/issues/171#issuecomment-2258137192">cisc architecture</a>and more and more
instructions being added. The compiler can naturally generate smaller and smaller binaries.
Because imagine you can do some sort of cpu operation that would have required lets say 20 (or even 200) instructions in the
same space as 1 to 10.</p>
<p>Thats the issue at core here.</p>
<p><strong>TODO show diffs proving my point</strong></p>
<h3 id="well-what-now">well what now?</h3>
<p>While trying out various compiler flags, compiler versions and even clang/LLVM <small>(because why not)</small>
I got a little lucky. Turns out that when using gcc14 paired with the <code>gcc.arch = "z10";</code> flag. Things
would actually compile. Making the Z10 I guess the oldest mainframe one could build s390-tools zipl for
<small>(and luckily that was the one I was targetting)</small>.</p>
<p>Its of advantage that nix(pkgs) makes it somewhat trivial to swap out the version GCC used by <code>stdenv.mkDerivation {}</code>. It
does this by offering a bunch of packages
with the following namingscheme <q>gcc<code><MAJOR_VERSION_NUMBER></code>Stdenv</q>. As packages providing an <code>stdenv</code>
expose <code>.mkDerivation {}</code> which is what we want to build pretty much everything in nix.</p>
<p>Using the following did the trick:</p>
<pre data-lang="nix" class="language-nix z-code"><code class="language-nix" data-lang="nix"><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">packages</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">s390x_linux</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">default</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-variable z-parameter z-name z-nix">pkgs</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">gcc14Stdenv</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">mkDerivation</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">name</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">"</span>s390-tools<span class="z-punctuation z-definition z-string z-double z-end z-nix">"</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"> <span class="z-comment z-line z-number-sign z-nix"># ...</span>
</span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset z-nix">}</span>
</span></code></pre>
<p><strong>(TODO after fixing the actual zipl drv include a proper snippet though it more or less says this either way)</strong></p>
<blockquote class="tip">
<p class="alert-title">
<i class="icon"></i>Tip</p>
<p>Using multiple stdenv packages <code>mkDerivation {}</code> function will result
in having to compile it from scratch if you don’t have binary caches. This is often the case on
niche architectures or when using custom compiler flags with your nixpkgs.</p>
<p>Among other reasons, this is why using multiple <code>stdenv</code>’s is not exactly favorable. Especially if
you will only do that for a single package <small>(unlike using a certain gcc version is unavoidable like in my case until IBM’s fix</small>)</p>
</blockquote>
<p>So with my initial crude attempts at writing the “s390-tools => (zipl)” derivation
now actually building succesfully for the first time. Its time for the next stage:</p>
<p><small>(gotta come back here some day to bring this derivation out of this hacky state its in but let that be my problem)</small></p>
<h3 id="going-from-zipl-to-boot-loader-zipl">going from zipl to <code>boot.loader.zipl</code>:</h3>
<p>If you ever succesfully installed nixos before you probably remember having used an option starting with <code>boot.loader</code> like
<code>boot.loader.grub</code> or <code>boot.loader.systemd-boot</code> or even <code>boot.loader.generic-extlinux-compatible</code> (<small>assuming of course
you did not use the installer which seems to exist these days</small>)</p>
<h4 id="though-what-would-need-to-be-done-to-get-boot-loader-zipl-to-show-up-in-here-as-well">Though what would need to be done to get <code>boot.loader.zipl</code> to show up in here as well?</h4>
<p>In order for the module to work like the other bootloaders it needs to declare a single setting: <code>system.build.installBootLoader</code>
<small>(and maybe <code>system.boot.loader.id = "zipl";</code> but that one is boring)</small> <strong>TODO confirm if loader.id is needed I forgot</strong></p>
<p>This is as when <code>nixos-install</code> is being run (or <code>nixos-rebuild</code> for this matter) it will evaluate this specific option
and at some point during the rebuild/install process execute whatever this option points to. All bootloaders I’ve seen
packaged so far bring some sort of wrapper script.</p>
<p>Grub for example has: <a href="https://github.com/NixOS/nixpkgs/blob/5135c59491985879812717f4c9fea69604e7f26f/nixos/modules/system/boot/loader/grub/install-grub.pl">nixos/modules/system/boot/loader/grub/install-grub.pl</a>.</p>
<p>And extlinux has: <a href="https://github.com/NixOS/nixpkgs/blob/5135c59491985879812717f4c9fea69604e7f26f/nixos/modules/system/boot/loader/generic-extlinux-compatible/extlinux-conf-builder.sh">nixos/modules/system/boot/loader/generic-extlinux-compatible/extlinux-conf-builder.sh</a>.</p>
<p>Besides the fact that I don’t know the perl language the <code>extlinux-conf-builder.sh</code> helper script is also much simpler. So lets look at this
one as opposed to <code>install-grub.pl</code> .</p>
<p>From simply looking at the way its invoked its pretty clear what we will have to do something like:</p>
<pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-storage z-modifier z-shell">export</span> <span class="z-variable z-other z-readwrite z-assignment z-shell">PATH</span><span class="z-keyword z-operator z-assignment z-shell">=</span><span class="z-string z-unquoted z-shell">/empty:@path@</span></span>
</span><span class="z-source z-shell z-bash">
</span><span class="z-source z-shell z-bash"><span class="z-meta z-function z-shell"><span class="z-entity z-name z-function z-shell">usage</span><span class="z-punctuation z-section z-parens z-begin z-shell">(</span><span class="z-punctuation z-section z-parens z-end z-shell">)</span> <span class="z-punctuation z-section z-braces z-begin z-shell">{</span>
</span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function z-shell"> <span class="z-meta z-function-call z-shell"><span class="z-support z-function z-echo z-shell">echo</span></span><span class="z-meta z-function-call z-arguments z-shell"> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">"</span>usage: <span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">0</span></span> -t <timeout> -c <path-to-default-configuration> [-d <boot-dir>] [-g <num-generations>] [-n <dtbName>] [-r]<span class="z-punctuation z-definition z-string z-end z-shell">"</span></span> <span class="z-keyword z-operator z-assignment z-redirection z-shell">>&</span><span class="z-constant z-numeric z-integer z-decimal z-file-descriptor z-shell">2</span></span>
</span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function z-shell"> <span class="z-meta z-function-call z-shell"><span class="z-support z-function z-exit z-shell">exit</span></span><span class="z-meta z-function-call z-arguments z-shell"> 1</span>
</span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function z-shell"><span class="z-punctuation z-section z-braces z-end z-shell">}</span></span>
</span><span class="z-source z-shell z-bash">
</span><span class="z-source z-shell z-bash"><span class="z-variable z-other z-readwrite z-assignment z-shell">timeout</span><span class="z-keyword z-operator z-assignment z-shell">=</span> <span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> Timeout in centiseconds</span><span class="z-comment z-line z-number-sign z-shell">
</span></span><span class="z-source z-shell z-bash"><span class="z-variable z-other z-readwrite z-assignment z-shell">menu</span><span class="z-keyword z-operator z-assignment z-shell">=</span><span class="z-string z-unquoted z-shell">1</span> <span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> Enable menu by default</span><span class="z-comment z-line z-number-sign z-shell">
</span></span><span class="z-source z-shell z-bash"><span class="z-variable z-other z-readwrite z-assignment z-shell">default</span><span class="z-keyword z-operator z-assignment z-shell">=</span> <span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> Default configuration</span><span class="z-comment z-line z-number-sign z-shell">
</span></span><span class="z-source z-shell z-bash"><span class="z-variable z-other z-readwrite z-assignment z-shell">target</span><span class="z-keyword z-operator z-assignment z-shell">=</span><span class="z-string z-unquoted z-shell">/boot</span> <span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> Target directory</span><span class="z-comment z-line z-number-sign z-shell">
</span></span><span class="z-source z-shell z-bash"><span class="z-variable z-other z-readwrite z-assignment z-shell">numGenerations</span><span class="z-keyword z-operator z-assignment z-shell">=</span><span class="z-string z-unquoted z-shell">0</span> <span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> Number of other generations to include in the menu</span><span class="z-comment z-line z-number-sign z-shell">
</span></span><span class="z-source z-shell z-bash">
</span><span class="z-source z-shell z-bash"><span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> ...</span><span class="z-comment z-line z-number-sign z-shell">
</span></span><span class="z-source z-shell z-bash"><span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> actual implementation</span><span class="z-comment z-line z-number-sign z-shell">
</span></span><span class="z-source z-shell z-bash"><span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> ...</span><span class="z-comment z-line z-number-sign z-shell">
</span></span></code></pre>
<p><small>For now just assume <code>$PATH</code> is properly populated with all the executables we need</small></p>
<p>You can see that the helper script is merely being called with a bunch of parameters. Just two of these <strong>have to</strong> be provided:
namely <code>-t timeout</code> and <code>-c path-to-default-configuration</code>. But all the other ones are optional | may have a default
value assigned if unspecified.</p>
<p>While most of these settings are quite useful and should be supported <strong>in theory</strong> <code>-c</code>
would be the only option you to respect. The <code>-c</code> doesn’t come from the nixos activation scripts though
as they wrapped it a tiny bit in reality it looks
like this</p>
<pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell"><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-punctuation z-section z-expansion z-parameter z-begin z-shell">{</span></span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-variable z-other z-readwrite z-shell">builder</span></span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-section z-expansion z-parameter z-end z-shell">}</span></span></span></span><span class="z-meta z-function-call z-arguments z-shell"> <span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-punctuation z-section z-expansion z-parameter z-begin z-shell">{</span></span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-variable z-other z-readwrite z-shell">builderArgs</span></span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-section z-expansion z-parameter z-end z-shell">}</span></span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>d</span> <span class="z-string z-quoted z-single z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">'</span>${args.path}<span class="z-punctuation z-definition z-string z-end z-shell">'</span></span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>c</span> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">"</span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-language z-shell">@</span></span><span class="z-punctuation z-definition z-string z-end z-shell">"</span></span></span>
</span><span class="z-source z-shell z-bash">
</span></code></pre>
<p>So <code>$@</code> is actually whats being passed rather than <code>-c <value></code></p>
<p>Thus its “the <strong>first</strong> commandline argument” passed to the executable declared by <code>system.build.installBootLoader</code> that
defines the path this script should install the bootloader files to.</p>
<p>The way one would want to wrap this and maybe processes further parameters,
depends on the features your bootloader actually supports and consequently the the module options you exposed
<small>(the menu bootentry selection timeout is a good example)</small>.</p>
<small>
<blockquote class="note">
<p class="alert-title">
<i class="icon"></i>Note</p>
<p>grub supports <strong>A LOT</strong> of features which is probably why it needs a huge perl script as opposed to the shell based extlinux-compat shown above</p>
</blockquote>
</small>
<p>So to write a bootloader module one essentially only has to respect the installation path and ideally expose every
functionality/feature the bootloader configs offer trough the nixos option system. And consequently parse the flags
you fed to your helper script trough the commandline parameters/env given.</p>
<p>So the skeleton I ended up using looks something like this:</p>
<pre data-lang="nix" class="language-nix z-code"><code class="language-nix" data-lang="nix"><span class="z-source z-nix"><span class="z-punctuation z-definition z-entity z-function z-2 z-nix">{</span><span class="z-variable z-parameter z-function z-1 z-nix">config</span><span class="z-keyword z-operator z-nix">,</span><span class="z-variable z-parameter z-function z-1 z-nix">pkgs</span><span class="z-keyword z-operator z-nix">,</span><span class="z-variable z-parameter z-function z-1 z-nix">lib</span><span class="z-keyword z-operator z-nix">,</span><span class="z-keyword z-operator z-nix">...</span><span class="z-punctuation z-definition z-entity z-function z-nix">}</span><span class="z-punctuation z-definition z-function z-nix">:</span> <span class="z-keyword z-other z-nix">let</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">builder</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-variable z-parameter z-name z-nix">pkgs</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">substituteAll</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">src</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-unquoted z-path z-nix">./zipl-conf-builder.sh</span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">isExecutable</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-language z-nix">true</span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">path</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-keyword z-other z-nix">with</span> <span class="z-variable z-parameter z-name z-nix">pkgs</span>; <span class="z-punctuation z-definition z-list z-nix">[</span>
</span><span class="z-source z-nix"> <span class="z-variable z-parameter z-name z-nix">coreutils</span>
</span><span class="z-source z-nix"> <span class="z-variable z-parameter z-name z-nix">gnused</span>
</span><span class="z-source z-nix"> <span class="z-variable z-parameter z-name z-nix">gnugrep</span>
</span><span class="z-source z-nix"> <span class="z-variable z-parameter z-name z-nix">s390-tools</span> <span class="z-comment z-line z-number-sign z-nix"># zipl is found here</span>
</span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-list z-nix">]</span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"> <span class="z-keyword z-other z-inherit z-nix">inherit</span> <span class="z-punctuation z-section z-function z-arguments z-nix">(</span><span class="z-variable z-parameter z-name z-nix">pkgs</span><span class="z-punctuation z-section z-function z-arguments z-nix">)</span> <span class="z-entity z-other z-attribute-name z-single z-nix">bash</span><span class="z-punctuation z-terminator z-inherit z-nix">;</span>
</span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"><span class="z-keyword z-other z-nix">in</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">options</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">boot</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">loader</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">zipl</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">enable</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-variable z-parameter z-name z-nix">mkOption</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">default</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-language z-nix">false</span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">type</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-variable z-parameter z-name z-nix">types</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">bool</span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">description</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">"</span>wether to enable s390-tools zipl bootloader<span class="z-punctuation z-definition z-string z-double z-end z-nix">"</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix">
</span><span class="z-source z-nix"> <span class="z-comment z-block z-nix">/* other options here like:
</span></span><span class="z-source z-nix"><span class="z-comment z-block z-nix"> timeout = mkOption { ... };
</span></span><span class="z-source z-nix"><span class="z-comment z-block z-nix">
</span></span><span class="z-source z-nix"><span class="z-comment z-block z-nix"> to then append below in between ${builder} and -c
</span></span><span class="z-source z-nix"><span class="z-comment z-block z-nix"> */</span>
</span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix">
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">config</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-variable z-parameter z-name z-nix">mkIf</span> <span class="z-variable z-parameter z-name z-nix">config</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">boot</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">loader</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">zipl</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">enable</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">system</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">build</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">installBootLoader</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">"</span><span class="z-markup z-italic"><span class="z-punctuation z-section z-embedded z-begin z-nix">${</span><span class="z-variable z-parameter z-name z-nix">builder</span><span class="z-punctuation z-section z-embedded z-end z-nix">}</span></span> -c<span class="z-punctuation z-definition z-string z-double z-end z-nix">"</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">system</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">boot</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">loader</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">id</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">"</span>zipl<span class="z-punctuation z-definition z-string z-double z-end z-nix">"</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span>
</span><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset z-nix">}</span>
</span></code></pre>
<p>other than <code>pkgs.substituteAll</code> wich takes care of the conf-builders <code>$PATH</code> variable being properly
populated. Everything seen above should have been covered by now. The script itself won’t
structurally and conceptually differ <small>(toooo much)</small> from the extlinux one. Except
that in my case boot loader menu entries and the actual
entries are being organized into 2 different blocks that are later on merged together.</p>
<p>Because one of nixos’s greatest feature is being able to switch between system-generations
(ideally directly from the bootloader) to temporarily roll back broken systems and such without
relying on CoW filesystems or such things. One most likely wants to add support fot that in ones
script as well. Thus a little funky shell magic is needed:</p>
<pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"> <span class="z-keyword z-control z-loop z-for z-shell">for</span><span class="z-meta z-group z-for z-shell"> generation <span class="z-keyword z-control z-in z-shell">in</span> <span class="z-meta z-group z-expansion z-command z-parens z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-punctuation z-section z-parens z-begin z-shell">(</span>
</span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-group z-for z-shell"><span class="z-meta z-group z-expansion z-command z-parens z-shell"> <span class="z-punctuation z-definition z-compound z-begin z-shell">(</span><span class="z-meta z-function-call z-shell"><span class="z-support z-function z-cd z-shell">cd</span></span><span class="z-meta z-function-call z-arguments z-shell"> /nix/var/nix/profiles</span> <span class="z-keyword z-operator z-logical z-and z-shell">&&</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">ls</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>d</span> system-<span class="z-keyword z-operator z-regexp z-quantifier z-shell">*</span>-link</span><span class="z-punctuation z-definition z-compound z-end z-shell">)</span><span class="z-meta z-function-call z-arguments z-shell"> <span class="z-punctuation z-separator z-continuation z-line z-shell">\
</span></span></span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-group z-for z-shell"><span class="z-meta z-group z-expansion z-command z-parens z-shell"><span class="z-meta z-function-call z-arguments z-shell"></span> <span class="z-keyword z-operator z-logical z-pipe z-shell">|</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">sed</span></span><span class="z-meta z-function-call z-arguments z-shell"> <span class="z-string z-quoted z-single z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">'</span>s/system-\([0-9]\+\)-link/\1/<span class="z-punctuation z-definition z-string z-end z-shell">'</span></span> <span class="z-punctuation z-separator z-continuation z-line z-shell">\
</span></span></span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-group z-for z-shell"><span class="z-meta z-group z-expansion z-command z-parens z-shell"><span class="z-meta z-function-call z-arguments z-shell"></span> <span class="z-keyword z-operator z-logical z-pipe z-shell">|</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">sort</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>n</span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>r</span> <span class="z-punctuation z-separator z-continuation z-line z-shell">\
</span></span></span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-group z-for z-shell"><span class="z-meta z-group z-expansion z-command z-parens z-shell"><span class="z-meta z-function-call z-arguments z-shell"></span> <span class="z-keyword z-operator z-logical z-pipe z-shell">|</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">head</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>n</span> <span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">numGenerations</span></span></span><span class="z-punctuation z-section z-parens z-end z-shell">)</span></span></span><span class="z-keyword z-operator z-logical z-continue z-shell">;</span> <span class="z-keyword z-control z-loop z-do z-shell">do</span>
</span><span class="z-source z-shell z-bash"> <span class="z-variable z-other z-readwrite z-assignment z-shell">link</span><span class="z-keyword z-operator z-assignment z-shell">=</span><span class="z-string z-unquoted z-shell">/nix/var/nix/profiles/system-<span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">generation</span></span>-link</span>
</span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">addZiplEntry</span></span><span class="z-meta z-function-call z-arguments z-shell"> <span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">link</span></span> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">"</span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-punctuation z-section z-expansion z-parameter z-begin z-shell">{</span></span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-variable z-other z-readwrite z-shell">generation</span></span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-section z-expansion z-parameter z-end z-shell">}</span></span><span class="z-punctuation z-definition z-string z-end z-shell">"</span></span> <span class="z-keyword z-operator z-assignment z-redirection z-shell">>></span> <span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">tmpFile</span></span></span>
</span><span class="z-source z-shell z-bash">
</span><span class="z-source z-shell z-bash"> <span class="z-keyword z-control z-loop z-for z-shell">for</span><span class="z-meta z-group z-for z-shell"> specialisation <span class="z-keyword z-control z-in z-shell">in</span> <span class="z-meta z-group z-expansion z-command z-parens z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-punctuation z-section z-parens z-begin z-shell">(</span>
</span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-group z-for z-shell"><span class="z-meta z-group z-expansion z-command z-parens z-shell"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">ls</span></span><span class="z-meta z-function-call z-arguments z-shell"> /nix/var/nix/profiles/system-<span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">generation</span></span>-link/specialisation <span class="z-punctuation z-separator z-continuation z-line z-shell">\
</span></span></span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-group z-for z-shell"><span class="z-meta z-group z-expansion z-command z-parens z-shell"><span class="z-meta z-function-call z-arguments z-shell"></span> <span class="z-keyword z-operator z-logical z-pipe z-shell">|</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">sort</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>n</span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>r</span></span><span class="z-punctuation z-section z-parens z-end z-shell">)</span></span></span><span class="z-keyword z-operator z-logical z-continue z-shell">;</span> <span class="z-keyword z-control z-loop z-do z-shell">do</span>
</span><span class="z-source z-shell z-bash"> <span class="z-variable z-other z-readwrite z-assignment z-shell">link</span><span class="z-keyword z-operator z-assignment z-shell">=</span><span class="z-string z-unquoted z-shell">/nix/var/nix/profiles/system-<span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">generation</span></span>-link/specialisation/<span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">specialisation</span></span></span>
</span><span class="z-source z-shell z-bash"> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">addZiplEntry</span></span><span class="z-meta z-function-call z-arguments z-shell"> <span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">link</span></span> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">"</span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-punctuation z-section z-expansion z-parameter z-begin z-shell">{</span></span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-variable z-other z-readwrite z-shell">generation</span></span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-section z-expansion z-parameter z-end z-shell">}</span></span>-<span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-punctuation z-section z-expansion z-parameter z-begin z-shell">{</span></span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-variable z-other z-readwrite z-shell">specialisation</span></span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-section z-expansion z-parameter z-end z-shell">}</span></span><span class="z-punctuation z-definition z-string z-end z-shell">"</span></span> <span class="z-keyword z-operator z-assignment z-redirection z-shell">>></span> <span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">tmpFile</span></span></span>
</span><span class="z-source z-shell z-bash"> <span class="z-keyword z-control z-loop z-end z-shell">done</span>
</span><span class="z-source z-shell z-bash"> <span class="z-keyword z-control z-loop z-end z-shell">done</span>
</span></code></pre>
<p>while a little simplified and still WIP / subjected to change. addZiplEntry effectively</p>
<ul>
<li>add a menu option to the zipl config file</li>
<li>extracts the profiles init= and other kernel parameters</li>
<li>copies over the profiles kernel and initrd to the bootloader install dir</li>
<li>is able to also perform cleanup on kernels/initrds if the generation was to be removed</li>
<li>actually installs zipl <code>zipl --config=</code> with the newly generated config</li>
</ul>
<h2 id="booting-nixos-ibm-z10">booting NixOS (IBM Z10)</h2>
<p><strong>TODO link to secondary article that explains how
to even set up a mainframe (as in sitting in front of it)</strong></p>
<hr />
<h2 id="etc">etc :</h2>
<p>I also needed to address issues in pythons psutil,numpy1,numpy2 when building a more common NixOS installation, it is also required to do some modifications to libfuse,libuv,klibc,spdlog,luajit,libopus,tmp2-tss,aws-c-sdk,aws-c-common.</p>
<p>Also to have a more or less proper nixos system one may not get around the fact that nsncd has some serialization issues and netcat needs to be
taken from openssl not libressl as libressl completely dropped s390x support in recent years.</p>
<p>Also the tpm2 systemd service being broken and some kernel flags being absent is something to keep in mind.
<strong>TODO maybe elaborate on all of these points</strong></p>
<p>TODO talk about https://github.com/aws/aws-sdk-cpp</p>