Skip to content

Commit

Permalink
catching up to present situation
Browse files Browse the repository at this point in the history
  • Loading branch information
@HankB
HankB committed Oct 3, 2019
1 parent 0d4763d commit 3216d48
Show file tree
Hide file tree
Showing 2 changed files with 22 additions and 104 deletions.
120 changes: 19 additions & 101 deletions Debian/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,11 @@ Script to facilitate installing Debian Buster on a ZFS root.

# Note

Debian 10.1 Live Gnome is recommended.
Debian 10.1 Live Gnome install media is used for all testing and is recommended.

This commit has been fairly thoroughly tested including on real H/W. During the last planned test, ZFS 0.8.2 was released to Buster backports and broke the script. The patch will be removed form the script and testing repeated.
This commit has been fairly thoroughly tested including on real H/W with the latest release available in backports (0.8.2-2~bpo10+1).

As of 2019-09-21 the instructions no longer support the EXPERIMENTAL branch and have been simplified to use backports for all installs. LUKS encryption was removed and has been restored.
As of 2019-09-21 the instructions no longer support the EXPERIMENTAL branch and have been simplified to use backports for all installs. LUKS encryption was removed and has since been restored.

## Inspiration

Expand All @@ -17,33 +17,28 @@ Any references to "instructions" below refer to the contents of this links.

## Roadmap

The intent of this script is to automate the instructions linked above. Alternate pool configurations (e.g. RAIDZ5, mirror etc.) are left to the user. At present all functionality passes the tests listed at [Google Sheets](https://docs.google.com/spreadsheets/d/1aqDocC9FZhQqJpilyDI7LxOcShHNU8znhwk0IFEm-gQ/edit?usp=sharing) for commit 4c3fa56 and using Debian Live 10.0. On the same day the tests were completed, Debian 10.1 was released. Future testing will use 10.1 when additional features/fizes are to be tested.
The intent of this script is to automate the instructions linked above. Alternate pool configurations (e.g. RAIDZ5, mirror etc.) are left to the user mto prepare and the script can be used to install to them. At present all functionality passes the tests listed at [Google Sheets](https://docs.google.com/spreadsheets/d/1F_enjjrheYRwfxnIVbyzsWkdxQesY6dfqt8ElmtgOL8/edit?usp=sharing) for commit 9e27688 and using Debian Live 10.1.

The need for testing can come from several external sources.

* Bug reports, pull requests and/or feature requests.
* Upgrade of backports to a new version of ZFS (presently 8.1, could be 8.2 soon.)
* Upgrade of backports to a new version of ZFS.
* Changes to the isntructions.

When the script changes to accommodate either of these, all tests will be repeated.

The next effort will be to bring the script into compliance with the instructions (eliminating the BACKPORTS and EXPERIMENTAL options. BACKPORTS will be the default and
EXPERIMENTAL has been eliminated.) Note: this work has been completed but not thoroughly tested.

The next effort after that will be to revise the test scripts. They have grown in an ad-hoc fashion as testing has been performed on different machines. They can be configured such that the same script will work on all test enviroments (presently Virtualbox VMs.) And each of the 18 test cases will get its own script.
When the script changes to accommodate either of these, all tests are repeated.

## Deviations from Debian Buster Root on ZFS

The intent is to follow the instructions closely, however occasional problems cropped up that required further changes not included in the original instructions.
The intent is to follow the instructions closely, however occasional problems cropped up that required changes to the original instructions.

* The script supports the capability to install dual boot with Windows or other Linux distros. It is entirely possible that it will not work with all distros. At present any problems encountered have resulted in failure to install and have not caused a problem with existing installations. Nevertheless is is highly recommended to back up the drive before proceeding. (I can backup a 120GB drive to my local file server in about 15 minutes.)
* The script supports the capability to install dual boot with Windows or other Linux distros. It is entirely possible that it will not work with all distros. At present any problems encountered have resulted in failure to install and have not caused a problem with existing installations. Nevertheless is is highly recommended to back up the drive before proceeding.
* The `-f` (force) flag is included in the `zpool create` commands because on too many occasions the command exited with a warning and indicated it could be overridden with this flag.
* The device is wiped using `wipefs` of all previous filesystem signatures. This was added because a previous ZFS pool would cause `zpool create` to fail, even with the `-f` option. In the case of using existing partitions, this is applied to the partitions selected for the `bpool` and `rpool`.
* The EFI and boot partitions are increased to 1024MB to reduce the chance that they could fill up. (The boot partition in an existing installation filled due to snapshots.)
* The device is wiped using `wipefs` of all previous filesystem signatures. This was added because a previous ZFS pool would cause `zpool create` to fail, even with the `-f` option. In the case of using preconfigured partitions, this is applied to the partitions selected for the boot pool and root pool.
* Specify the URL http://deb.debian.org/debian on the `debootstrap` command line. It is not clear to me what the default is.

## Limitations

* UEFI support only. All of my PCs on which I would use this support UEFI and I have found advantages to using that. In the case of dual boot support it will use the existing UEFI partition (unless specific partitions are specified.)
* UEFI support only. All of my PCs on which I would use this support UEFI and I have found advantages to using that. In the case of dual boot support it can use the existing UEFI partition or a diffewrent UEFI partition can be created and used.
* The script requires interaction. Some commands could probably be fully automated but at present it is necessary to acknowledge a popup regarding the ZFS license.

## Motivation
Expand All @@ -62,11 +57,9 @@ There are other scripts that may suit your needs better than this.
## Status

* Script has been checked with `shellcheck` and all reported issues resolved.
* Script is current with the intructions listed at https://github.com/zfsonlinux/zfs/wiki/Debian-Buster-Root-on-ZFS (Note: The release of 0.8.2 breaks the instructions and script.)
* All test cases have passed using Debian Live 10.1 which is now recommended. Many tests do not perform well on Virtualbox, resulting in no rpool pool on reboot. Repeating the identical test often succeeds. Fiddling with rpool before booting sometimes helps. This has not been seen on real H/W.
* Script is current with the intructions listed at https://github.com/zfsonlinux/zfs/wiki/Debian-Buster-Root-on-ZFS except for the `sources.list` additions.
* All test cases have passed using Debian Live 10.1. (see Errata)
* Testing scripts have been revised and all pass `shellcheck`.
* Script has been broken due to release of ZFS 0.8.2 to backports. The fix is simple and testing will commence shortly.


## WARNING WILL ROBINSON

Expand Down Expand Up @@ -94,7 +87,7 @@ Uses simple host name - not FQDN (step 4.1)

### Suggested

Pick a settings file from the `.../testing` directoy and tailor it to fit your specifics. Run the script using that. If necessary,modify the script itself. (Note that some operations such as building the modules is done twice - once in the live environment and again in the `chroot`.)
Pick a settings file from the `.../testing` directoy and tailor it to fit your specifics. See also the files in `.../testing/obsolete`. Run the script using that. If necessary,modify the script itself.

### ENV variables

Expand All @@ -106,86 +99,11 @@ Environment variables are provided to the script from a text file. If a text fil
install_Debian_to_ZFS_root.sh env.sh
```

Examples of settings are shown.

#### Using an existing partition (e.g. dual or multi-boot.)

```shell
export ETHERNET=enp3s0
export NEW_HOSTNAME=rocinante
export YOURUSERNAME=hbarta
export ROOT_POOL_NAME=rpool
export BOOT_POOL_NAME=bpool
export ENCRYPT=no|yes

export INSTALL_TYPE=use_partitions

export EFI_PART=/dev/disk/by-id/nvme-eui.000000000000001000080d02003e9b51-part2
export ROOT_PART=/dev/disk/by-id/nvme-eui.000000000000001000080d02003e9b51-part7
export BOOT_PART=/dev/disk/by-id/nvme-eui.000000000000001000080d02003e9b51-part8
```

It is the responsibility of the user to prepare all three partitions. In addition, it is required that the EFI partition be formatted. It can be an existing EFI partition from e.g. a Windows install or a new partition formatted as described in the instructions.

#### INSTALL_TYPE=whole_disk

```shell
export ETHERNET=enp3s0
export NEW_HOSTNAME=rocinante
export YOURUSERNAME=hbarta
export ROOT_POOL_NAME=rpool
export BOOT_POOL_NAME=bpool
export ENCRYPT=no|yes

export INSTALL_TYPE=whole_disk

export DRIVE_ID=nvme-eui.000000000000001000080d02003e9b51
```

The script will clear the drive, create and format all required partitions. This option requires the least preparation.

#### INSTALL_TYPE=use_pools

```shell
export ETHERNET=enp3s0
export NEW_HOSTNAME=rocinante
export YOURUSERNAME=hbarta
export ROOT_POOL_NAME=rpool
export BOOT_POOL_NAME=bpool
export ENCRYPT=no|yes

export INSTALL_TYPE=use_pools

export EFI_PART=/dev/disk/by-id/nvme-eui.000000000000001000080d02003e9b51-part2
```

This option requires that the EFI partition be present as for `INSTALL_TYPE=use_partitions`. In addition the root and boot pools must be created.
It would probably be betrter to describe these files as configuration files. Early on, the script did get these variables from the environment but that was early in development.

#### Other options
Please see the test settings files to see what environment variables are used for the various cases.

```shell
export USE_BACKPORTS=yes
```
Use backports as described at https://github.com/zfsonlinux/zfs/wiki/Debian. This will result in newer versions of ZFS to be used. Use with caution. I do not know what happens when a version change (e.g. 0.7.13 -> 0.8.1) happens. At present (2019-08-15) this will result in installation of version 0.7.13 vs. 0.7.12 in stable. In the relatively near future 0.8.x will make it to backports.

#### Tips

`ls -l /dev/disk/by-id` will identify disks and partitions. `ip addr` will show the Ethernet device. (This process requires Ethernet.) The partitions are conveniently created using `sgdisk` as shown in the instructions.

The file `env.sh` looks like a shell script but is intended to be copied/pasted into a terminal window. It is convenient to edit it to set the desired variables for the target system.

### Process

Boot the Debian Live USB. Note that to install using EFI it may be necessary to boot the USB using EFI.

1. Prepare any required partitions and/or pools as listed above.
1. Copy the scripts to a convenient directory in the live environment. (Or put on a handy USB drive.)
1. Run `initial.sh` to install `ssh` so rest of the process can be run by `ssh`-ing in from another host. (This can be skipped if all work is done at the console.)
1. Identify the required environment variables and edit `env.sh` accordingly.
1. `sudo` to `root` user.
1. Source `env.sh` to populate the environment variables.
1. Execute `install_Debian_to_ZFS_root.sh`.
1. Ponder the need for a post-install script and copy/paste commands from the linked instructions and enjoy!
Examples of settings are shown.

## Testing

Expand All @@ -203,6 +121,6 @@ Feel free to submit pull requests for features or other improvements. My inclina

I have submitted issues for things that I anticipate as possible next steps. If you wish to help with one of these, please reply to the issue so we can coordinate efforts (before you start work.) If you see the need for something additional or identify a problem, please file an issue.

## TODO
## Errata

* Fully automate the script (Eliminate three interactive commands.)
Many tests do not perform well resulting in a no rpool condition on reboot. Repeating the identical test often succeeded. Fiddling with rpool before booting sometimes helps. This has been seen on real H/W as well. A fixup was added at the end of the script which will report `root pool fixup applied` if needed and otherwise simply import and then export the pool. During testing the fixup was never seen. Final version during testing was 0.8.2-2~bpo10+1 and perhaps a fix was issued for this. ZFS versiojn will be tracked more closely during subsequent testing.
6 changes: 3 additions & 3 deletions Debian/testing/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,10 @@ My procedure is as follows.

* ticking the "Enable EFI (special OSes only)" box in Settings -> System -> Motherboard
* mounting the Live ISO on the SATA bus in Settings -> Storage. (Linux is a special OS! ;) )
* Configure Networking - Settings -> Network -> Attached to: Bridged Adapter. (Allows the host to connect with the guest.)
* Configure Networking - Settings -> Network -> Attached to: Bridged Adapter. (Allows the host to connect with the guest.) (Fairly early on I switched to using NAT for the network connection and initiating file transfers from the guest)

2) Boot from the ISO and confirm that the system booted in EFI mode. Check for existence of `/sys/firmware/efi'. (Shutdown the VM and clone from it for all subsequent tests.)
0) Clone the VM and boot. Once in the live environment, install and start `openssh-server`.
2) Boot from the ISO and confirm that the system booted in EFI mode. Check for existence of `/sys/firmware/efi`. (Shutdown the VM and clone from it for all subsequent tests.)
0) Once in the live environment, install and start `openssh-server`. (Not needed when initiating the connection from the guest.)

```shell
sudo -s
Expand Down

0 comments on commit 3216d48

Please sign in to comment.