OpenZFS on OS X - User contributions [en]

Windows Registry

2024-10-18T02:01:45Z

210.172.128.230: /* Vectorized Raidz */

== Registry ==

Open ZFS uses the Registry for default tunable values which is read in on startup.

'''\HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\ZFSin''' old

'''\HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\OpenZFS''' new

=== hostid ===

The Registry key '''hostid''' is generally handled by OpenZFS internally, ie, generated randomly if it is not set the first time Open ZFS is loaded.

You can change '''hostid''' if that is so desired. For example, if you share the pool between multiple Operating Systems on the same Hardware.

=== Arc Max ===

To set the ARC maximum memory usage to 2GB, create an entry;

"zfs_arc_max" REG_QWORD "0x80000000"

and reboot. If you wish to change it on a running system without rebooting, see the [[Windows_kstat|kstat]] example.

=== Vectorized Raidz ===

There are a few vectorized implementations to chose from, due to a lack of
kstat porting, we can not display the method used or available. It is
printed on the debug console.

To disable all assembler implementations, create a registry setting:
"zfs_sha256_impl" REG_SZ "generic"
"zfs_sha512_impl" REG_SZ "generic"
"icp_aes_impl" REG_SZ "generic"
"icp_gcm_impl" REG_SZ "generic"
"zfs_blake3_impl" REG_SZ "generic"
"zfs_vdev_raidz_impl" REG_SZ "original"
"zfs_fletcher_4_impl" REG_SZ "scalar"

Which would set the implementations to use the standard C versions. The
default value is '''fastest''' which picks the highest available. Examples are
zfs_vdev_raidz_impl: [original] scalar sse2 ssse3
icp_gcm_impl_get: cycle fastest [generic]
icp_aes_impl_get: cycle fastest [generic] x86_64 aesni
zfs_fletcher_4_impl_get: fastest [scalar] superscalar superscalar4 sse2 ssse3

=== Available tunables ===

As of 2022, the tunables are now written to the Registry when you install OpenZFS, so all
available tunables are visible there. This list below is now obsolete.
<hr>
The ZFS tunables are many and constantly changing. The easiest way to display them all is to run the [[Windows_kstat|kstat]] program. <- Not true

Obsolete <s>Complete</s> list as of 2019-06-03:

active_vnodes
aggregation_limit
arc_lotsfree_percent
arc_reduce_dnlc_percent
async_read_max_active
async_read_min_active
async_write_max_active
async_write_max_dirty_pct
async_write_min_active
async_write_min_dirty_pct
create_negatives
crtime
dbuf_cache_max_bytes
force_formd_normalized
fzap_default_block_shift
hostid
ignore_negatives
ignore_positives
l2arc_feed_again
l2arc_feed_min_ms
l2arc_feed_secs
l2arc_headroom
l2arc_headroom_boost
l2arc_max_block_size
l2arc_noprefetch
l2arc_norw
l2arc_write_boost
l2arc_write_max
max_active
metaslab_aliquot
metaslab_df_alloc_threshold
metaslab_df_free_pct
read_gap_limit
reclaim_nodes
scrub_max_active
scrub_min_active
skip_unlinked_drain
snaptime
spa_asize_inflation
spa_max_replication_override
spa_mode_global
spa_version
sync_read_max_active
sync_read_min_active
sync_write_max_active
sync_write_min_active
use_system_sync
vdev_mirror_shift
vnop_debug
write_gap_limit
zfetch_array_rd_sz
zfetch_max_streams
zfetch_min_sec_reap
zfs_arc_average_blocksize
zfs_arc_grow_retry
zfs_arc_max
zfs_arc_meta_limit
zfs_arc_meta_min
zfs_arc_min
zfs_arc_p_min_shift
zfs_arc_shrink_shift
zfs_default_bs
zfs_default_ibs
zfs_delay_max_ns
zfs_delay_min_dirty_percent
zfs_delay_scale
zfs_dirty_data_max
zfs_dirty_data_sync
zfs_flags
zfs_free_bpobj_enabled
zfs_immediate_write_sz
zfs_lua_max_instrlimit
zfs_lua_max_memlimit
zfs_mdcomp_disable
zfs_no_scrub_io
zfs_no_scrub_prefetch
zfs_nocacheflush
zfs_prefetch_disable
zfs_read_chunk_size
zfs_recover
zfs_recv_queue_length
zfs_scrub_limit
zfs_send_corrupt_data
zfs_send_holes_without_brth_tm
zfs_send_queue_length
zfs_send_set_freerecords_bit
zfs_send_unmodified_spill_bloc
zfs_special_class_metadata_res
zfs_trim_extent_bytes_max
zfs_trim_extent_bytes_min
zfs_trim_metaslab_skip
zfs_trim_queue_limit
zfs_trim_txg_batch
zfs_txg_timeout
zfs_vdev_cache_bshift
zfs_vdev_cache_max
zfs_vdev_cache_size
zfs_vdev_file_size_mismatch_cn
zfs_vdev_queue_depth_pct
zfs_write_implies_delete_child
zil_replay_disable
zio_dva_throttle_enabled
zio_injection_enabled
zpl_version
zvol_immediate_write_sz
zvol_inhibit_dev

Windows cbuf

2024-09-06T06:07:59Z

210.172.128.230: /* Saving the debug print buffer */

== Saving the debug print buffer ==

Starting from version zfswin-2.2.6rc2 the cbuf can be more easily saved from
a running system, to assist with debugging.

Open up regedit.exe, and navigate to:
\HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\OpenZFS

Here there is now a entry called
zfs_cbuf_save

Modify it by writing a "1" (one) to it, which will trigger a save.
Use F5 to refresh Regedit in case there is an error message.

The file will be written to:

C:\Program Files\OpenZFS On Windows\cbuf.txt

If the installer was used, this directory should already exist. Otherwise,
create it by hand.

Windows cbuf

2024-09-06T06:07:41Z

210.172.128.230: /* Saving the debug print buffer */

== Saving the debug print buffer ==

Starting from version zfswin-2.2.6rc2 the cbuf can be more easily saved from
a running system, to assist with debugging.

Open up regedit.exe, and navigate to:
\HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\OpenZFS

Here there is now a entry called '''zfs_cbuf_save'''.

Modify it by writing a "1" (one) to it, which will trigger a save.
Use F5 to refresh Regedit in case there is an error message.

The file will be written to:

C:\Program Files\OpenZFS On Windows\cbuf.txt

If the installer was used, this directory should already exist. Otherwise,
create it by hand.

Windows cbuf

2024-09-06T06:07:18Z

210.172.128.230: /* Saving the debug print buffer */

== Saving the debug print buffer ==

Starting from version zfswin-2.2.6rc2 the cbuf can be more easily saved from
a running system, to assist with debugging.

Open up regedit.exe, and navigate to:
\HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\OpenZFS

Here there is now a entry called **zfs_cbuf_save**.

Modify it by writing a "1" (one) to it, which will trigger a save.
Use F5 to refresh Regedit in case there is an error message.

The file will be written to:

C:\Program Files\OpenZFS On Windows\cbuf.txt

If the installer was used, this directory should already exist. Otherwise,
create it by hand.

Windows cbuf

2024-09-06T06:06:58Z

210.172.128.230: Created page with "== Saving the debug print buffer == Starting from version zfswin-2.2.6rc2 the cbuf can be more easily saved from a running system, to assist with debugging. Open up regedit...."

== Saving the debug print buffer ==

Starting from version zfswin-2.2.6rc2 the cbuf can be more easily saved from
a running system, to assist with debugging.

Open up regedit.exe, and navigate to:
\HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\OpenZFS

Here there is now a entry called ***zfs_cbuf_save***.

Modify it by writing a "1" (one) to it, which will trigger a save.
Use F5 to refresh Regedit in case there is an error message.

The file will be written to:

C:\Program Files\OpenZFS On Windows\cbuf.txt

If the installer was used, this directory should already exist. Otherwise,
create it by hand.

Windows

2024-09-06T06:03:17Z

210.172.128.230: /* Documentation */

== OpenZFS on Windows ==

The old port '''ZFSin''' is slowly being replaced by a more proper '''OpenZFS''' port,
as the first was a fast proof-of-concept demo and now we are taking the time to
port it properly with upstream integration in mind.

If you are here to test it, get latest OpenZFS from nightly.

The OpenZFS port on Windows.

Current status: ''beta release candidate''
Latest Download: https://github.com/openzfsonwindows/openzfs/releases
Sources: https://github.com/openzfsonwindows/openzfs.git

=== Documentation ===

* Creating a [[windows_pool|Windows pool]]
* Handling [[windows_driveletter|Driveletters]]
* [[Windows_Registry|Registry]]
* Using [[windows_kstat|kstat]]
* Automatic pool [[windows_import|import]]
* Running zfs-tests [[windows_zfstest|zfstest]]
* Saving cbuf (debug print buffer) [[windows_cbuf|cbuf]]

=== Downloads ===

Currently we are doing releases on github.com, so use:

* [https://github.com/openzfsonwindows/ZFSin/releases Downloads]
* [[windows_builds|Nightly builds]]

=== Crash dump analysis ===

* [[Windows_BSOD|Windows minidumps]]
* [[Windows_bootloop|Windows boot-loop]]

=== Porting notes ===

* Read the notes on porting to [[Windows_port|Windows]]

=== Codesigning ===

* [[Codesign]]

Windows

2024-09-06T06:03:05Z

210.172.128.230: /* Documentation */

== OpenZFS on Windows ==

The old port '''ZFSin''' is slowly being replaced by a more proper '''OpenZFS''' port,
as the first was a fast proof-of-concept demo and now we are taking the time to
port it properly with upstream integration in mind.

If you are here to test it, get latest OpenZFS from nightly.

The OpenZFS port on Windows.

Current status: ''beta release candidate''
Latest Download: https://github.com/openzfsonwindows/openzfs/releases
Sources: https://github.com/openzfsonwindows/openzfs.git

=== Documentation ===

* Creating a [[windows_pool|Windows pool]]
* Handling [[windows_driveletter|Driveletters]]
* [[Windows_Registry|Registry]]
* Using [[windows_kstat|kstat]]
* Automatic pool [[windows_import|import]]
* Running zfs-tests [[windows_zfstest|zfstest]]
* Saving cbuf (debug print buffer) [[windows_cbuf|zbuf]]

=== Downloads ===

Currently we are doing releases on github.com, so use:

* [https://github.com/openzfsonwindows/ZFSin/releases Downloads]
* [[windows_builds|Nightly builds]]

=== Crash dump analysis ===

* [[Windows_BSOD|Windows minidumps]]
* [[Windows_bootloop|Windows boot-loop]]

=== Porting notes ===

* Read the notes on porting to [[Windows_port|Windows]]

=== Codesigning ===

* [[Codesign]]

Windows builds

2021-12-16T03:05:04Z

210.172.128.230:

== Nightly builds ==

We do not have nightly builds.

https://www.lundman.net/OpenZFSOnWindows-debug-2.1.0-2041-gfbc23d71b-dirty.exe 2021/12/16 First signed installer?

older:
https://www.lundman.net/OpenZFSOnWindows-debug-0.8.0-2020-ga5ead8ecd-dirty.exe 2021/09/30 2.1.0-rc1

Since I am still fighting Microsoft on the partner network, you will have to follow this procedure
to allow the driver to load:

https://support.viewsonic.com/en/support/solutions/articles/33000252416-how-to-install-unsigned-drivers-in-windows-10

Or enable developer mode:

https://www.howtogeek.com/292914/what-is-developer-mode-in-windows-10/

Windows builds

2020-07-03T06:50:01Z

210.172.128.230:

== Nightly builds ==

We do not have nightly builds.

However, from time to time, we do quick builds for people to test a fix.

http://www.lundman.net/OpenZFSOnWindows-debug-20200703.exe target: Win8.1 x64
http://www.lundman.net/OpenZFSOnWindows-debug-20200624.exe trying out exclusive disk
http://www.lundman.net/OpenZFSOnWindows-debug-20200615.exe keylocation=file:// fix
http://www.lundman.net/OpenZFSOnWindows-debug-20200410.exe New ZVOL code
http://www.lundman.net/OpenZFSOnWindows-debug-20200406.exe Icons
http://www.lundman.net/OpenZFSOnWindows-debug-20200402.exe Hagliga release.
http://www.lundman.net/OpenZFSOnWindows-release-20200402.exe Hagliga release.

Windows builds

2020-06-24T05:20:17Z

210.172.128.230: /* Nightly builds */

== Nightly builds ==

We do not have nightly builds.

However, from time to time, we do quick builds for people to test a fix.

http://www.lundman.net/OpenZFSOnWindows-debug-20200624.exe trying out exclusive disk
http://www.lundman.net/OpenZFSOnWindows-debug-20200615.exe keylocation=file:// fix
http://www.lundman.net/OpenZFSOnWindows-debug-20200410.exe New ZVOL code
http://www.lundman.net/OpenZFSOnWindows-debug-20200406.exe Icons
http://www.lundman.net/OpenZFSOnWindows-debug-20200402.exe Hagliga release.
http://www.lundman.net/OpenZFSOnWindows-release-20200402.exe Hagliga release.

ZFS on Boot

2019-10-17T01:20:13Z

210.172.128.230:

== ZFS as Boot ==

Status: Beta
Most of the commands listed below must be run as root or with '''sudo'''.

'''Updated for macOS Catalina '''

This guide assumes you have installed the latest Open ZFS on OS X on the system, both
to create and populate the '''rpool''' with data, but that the data copied will have the
kexts and prelinked kernels containing OpenZFS kexts.

=== Install OpenZFS ===

Download latest OpenZFSonOsX pkg and install.

=== Partitioning disk ===

You need generally a 200MB EFI partition, your ZFS partition, then finally a 200MB Apple boot partition.

''In this example, $NEWDISK is the blank disk to use; set it now''

(Change /dev/disk1 to your blank disk!)
# export NEWDISK=/dev/disk1
# echo $NEWDISK
/dev/disk1

Default '''diskutil''' partitions appear to work:

# diskutil partitionDisk $NEWDISK GPT ZFS rpool 0b
(You can ignore this error: )
Could not mount disk1s2 after erase
Error: -69832: File system formatter failed

/dev/disk1 (internal, physical):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *64.4 GB disk1
1: EFI EFI 209.7 MB disk1s1
2: FFFFFFFF-FFFF-FFFF-FFFF-FFFFFFFFFFFF 64.1 GB disk1s2
3: Apple_Boot Boot OS X 134.2 MB disk1s3

=== Setting up ZFS slice 2 ===

Create the actual pool:

# sudo zpool create -f -o ashift=12 -O casesensitivity=insensitive \
-O normalization=formD -O atime=off -O compression=lz4 \
-O mountpoint=none -O canmount=off rpool ${NEWDISK}s2

Create minimal datasets for ROOT

# zfs create -o mountpoint=none -o canmount=off rpool/ROOT
# zfs create -o mountpoint=legacy -o com.apple.mimic_hfs=on rpool/ROOT/Catalina
# zpool set bootfs=rpool/ROOT/Catalina rpool

Optional, create further datasets, perhaps for /Users, and/or, individual users...
# sudo zfs create -o mountpoint=/Users -o canmount=off rpool/HOME
# sudo zfs create rpool/HOME/username

Populate the rpool from the running system:

# mkdir /Volumes/ZFSBoot
# /Library/Filesystems/zfs.fs/Contents/Resources/mount_zfs rpool/ROOT/Catalina /Volumes/ZFSBoot
# rsync -axH --progress --exclude=".Spotlight-V100" --exclude=".fseventsd" --exclude=".vol" / /Volumes/ZFSBoot/

.. and wait.

(It would be neat if we could (ab)use the installer for the OS: ie some version of
installer -pkg [path to]/Install macOS Catalina.app/Contents/SharedSupport/InstallInfo.plist -target /Volumes/ZFSBoot -verbose -dumplog
but possibly it always insists on partitioning.)

Update the Boot.plist:

# vim /Volumes/ZFSBoot/Library/Preferences/SystemConfiguration/com.apple.Boot.plist

<dict>
<key>Kernel Flags</key>
<string>-v keepsyms=y zfs_boot=rpool</string>
<key>Root UUID</key>
<string>5A465320-626F-6F74-2064-657669636500</string>
</dict>

=== Setting up boot slice 3 ===

These steps require slice 2 copy to have finished.

# newfs_hfs -J -v "boot" ${NEWDISK}s3
# diskutil mount ${NEWDISK}s3
# mkdir -p /Volumes/boot/System/Library/CoreServices
# mkdir -p /Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels
# mkdir -p /Volumes/boot/com.apple.boot.R/Library/Preferences/SystemConfiguration
# mkdir -p /Volumes/boot/com.apple.boot.R/usr/standalone/i386

Populate boot:

# rsync -a /Volumes/ZFSBoot/System/Library/CoreServices/PlatformSupport.plist \
/Volumes/boot/System/Library/CoreServices/PlatformSupport.plist
# rsync -a /Volumes/ZFSBoot/System/Library/CoreServices/SystemVersion.plist \
/Volumes/boot/System/Library/CoreServices/SystemVersion.plist
# rsync -a /Volumes/ZFSBoot/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels/prelinkedkernel
# rsync -a /Volumes/ZFSBoot/Library/Preferences/SystemConfiguration/com.apple.Boot.plist \
/Volumes/boot/com.apple.boot.R/Library/Preferences/SystemConfiguration/com.apple.Boot.plist
# rsync -a /Volumes/ZFSBoot/usr/standalone/i386/ \
/Volumes/boot/com.apple.boot.R/usr/standalone/i386/

Bless the volume:

# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
--file /Volumes/boot/System/Library/CoreServices/boot.efi \
--bootefi /Volumes/ZFSBoot/System/Library/CoreServices/boot.efi \
--label "ZFS Boot"

This step does not change the current boot device, unless you add ''--setBoot''. See blessing the boot device to change the startup device.
Another useful argument is ''--nextonly'' to just affect one boot, which is great for testing.
To get out of trouble, remember that you can
hold `Alt` during boot to select disk to boot from.

=== Booting ZFS ===

Finally, tell your Mac to use the boot helper on boot. You may want to try the ''--nextonly'' option to avoid permanent changes.

# sudo bless --device ${NEWDISK}s3 --setBoot
# reboot

Additional tricks. If updating the kexts, and you want to update the boot volume as well:

Mount and update ZFSBoot
# mkdir /Volumes/ZFSBoot
# /Library/Filesystems/zfs.fs/Contents/Resources/mount_zfs rpool/ROOT/Catalina /Volumes/ZFSBoot

If you prefer to update the kexts in /Volumes/ZFSBoot/Library/Extensions/
# kextcache -c "/Volumes/ZFSBoot/System/Library/PrelinkedKernels/prelinkedkernel" -K "/Volumes/ZFSBoot/System/Library/Kernels/kernel" -l -- "/Volumes/ZFSBoot/System/Library/Extensions" "/Volumes/ZFSBoot/Library/Extensions/"

Or if you install new kexts on machine, and then update /Volumes/ZFSBoot/
# rsync -ar /Library/Extensions/ /Volumes/ZFSBoot/Library/Extensions/
# kextcache -c "/Volumes/ZFSBoot/System/Library/PrelinkedKernels/prelinkedkernel" -K "/System/Library/Kernels/kernel" -l -- "/System/Library/Extensions" "/Library/Extensions/"

Also update the tiny boot volume - find the disk it is on, in example, disk1s3. Pick kernel either on "/" or "/Volumes/ZFSBoot/"
# diskutil mount disk1s3
# mkdir -p /Volumes/boot/com.apple.boot.R/System/Library/Caches/com.apple.kext.caches/Startup
# kextcache -c "/Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels/prelinkedkernel" -K "/System/Library/Kernels/kernel" -l -- "/System/Library/Extensions" "/Library/Extensions/"

----

''' The remaining text is the old guide. '''

----

Because you have to compile your own KEXTs, they will not be signed. [https://support.apple.com/en-us/HT204899 System Integrity Protection] (SIP) needs to be disabled for unsigned KEXTs to work.
A future version of OpenZFSOnOSX will come with --enable-boot compiled KEXTs and be signed.

=== Disk Layout ===

Play disk: '''disk1'''

I used '''gptfdisk''', but '''gpt''' and '''diskutil''' can be used also.
Create the following partitions on the test boot disk:

Disk /dev/rdisk1: 83886080 sectors, 40.0 GiB
Number Start (sector) End (sector) Size Code Name
1 2048 411647 200.0 MiB EF00 EFI System
2 411648 82200575 39.0 GiB BF01 rpool
3 82200576 83886046 823.0 MiB AB00 Apple boot

* '''s1''' is the system EFI partition, which is used for software updates. You can also add rEFInd boot loader here if desired (optional).
* '''s2''' is the ZFS pool used for booting. Note: You are not limited to using the name '''rpool'''.
* '''s3''' is the small HFS boot helper which will carry the prelinkedkernel to load the kernel with ZFS.

The layout need not be exactly this, but:

* The EFI partition (type ef00) should always exist at the beginning of the disk. It is usually 200 MB.
* The Apple Boot (type ab00) partition should directly follow the ZFS partition. It should be at least 128 MB but may be as large as you want.

=== Populating the ZFS Root '''s2''' ===

==== Root pool ====
Any pool name may be used - '''rpool''' is used here for example.
Additional research is needed on atime, casesensitivity, and normalization (and probably more).
# sudo zpool create -f -o ashift=12 -O casesensitivity=insensitive \
-O normalization=formD -O atime=off -O compression=lz4 \
-O mountpoint=none -O canmount=off rpool disk1s2

==== Root dataset ====

Create '''ROOT''' and lower just like IllumOS so that you can easy swap OS snapshots and reboot into something else (for example to run softwareupdate).
# sudo zfs create -o mountpoint=none -o canmount=off rpool/ROOT
# sudo zfs create -o mountpoint=legacy rpool/ROOT/Capitan
# sudo zpool set bootfs=rpool/ROOT/Capitan rpool

Note: com.apple.mimic_hfs may be set on the root mountpoint. Testing is still underway for ZFS Root both with or without mimic set.

You probably want to relocate your home folder outside of the root dataset, for example rpool/HOME/username, or on ZVOL for compatibility.
# sudo zfs create -o mountpoint=/Users -o canmount=off rpool/HOME
# sudo zfs create rpool/HOME/username

Mount it:
# sudo mkdir /Volumes/Capitan
# sudo mount_zfs rpool/ROOT/Capitan /Volumes/Capitan

==== Install macOS ====
Copy over the OS as you see fit, rsync will generally do. (Remember that etc, tmp and var are symlinks into /private)
# sudo rsync -axH --exclude=".Spotlight-V100" --exclude=".fseventsd" \
--exclude=".vol" / /Volumes/Capitan/

==== Install ZFS ====

Boot support is now included with the binary install versions, so you don't have to compile it yourself.

In you haven't done so already, install homebrew (or macports, but in the example we will stick to homebrew as it's quicker to set-up):
# /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Install automake and libtool (required to compile spl and zfs):
# brew install automake libtool

Create target directory for installation:
# sudo mkdir -p /Volumes/Capitan/usr/local

Use your own repository:
# cd $your-zfs-source-area
Or clone it one from github:
# git clone https://github.com/openzfsonosx/spl.git
# git clone https://github.com/openzfsonosx/zfs.git

Build and install spl:
# cd spl
# git fetch --all
# ./autogen.sh
# ./configure --enable-boot
# make
# sudo make install DESTDIR=/Volumes/Capitan

Build and install zfs:

# cd ../zfs
# git fetch --all
# ./autogen.sh
# ./configure --enable-boot
# make
# sudo make install DESTDIR=/Volumes/Capitan

==== Edit Boot.plist ====

You should set Kernel Flags and Root UUID in the root volume (not the boot helper) as it will be copied to the boot helper automatically. It is located at /Library/Preferences/SystemConfiguration/com.apple.Boot.plist

More examples of the Boot.plist can be found at [[Boot.plist|com.apple.Boot.plist]]

# sudo vim /Volumes/Capitan/Library/Preferences/SystemConfiguration/com.apple.Boot.plist

Two keys must be set: '''Kernel Flags''' and '''Root UUID''':

Look for
<key>Kernel Flags</key>
<string></string>

Replace with:
<key>Kernel Flags</key>
<string>-v keepsyms=y zfs_boot=rpool</string>
<key>Root UUID</key>
<string>00000000-0000-0000-0000-000000000000</string>

Any UUID may be used here, I used 5A465320-626F-6F74-2064-657669636500, (which is literally the string "ZFS boot device\0").

See [[Boot.plist]] for more examples.

[[ZVOL boot]] from HFS+ can be accomplished by using the UUID of the partition containing a standard Mac OS install, for compatibility.

=== Setting up boot slice '''s3''' ===

==== Format ====
Journaled HFS+ filesystem, and populated thusly; (Volume named '''boot''' here)
These can be rsynced from the (normal) booted OS, or from the '''Capitan''' mount.

# newfs_hfs -J -v "boot" /dev/disk1s3
# diskutil mount disk1s3
# sudo mdutil -i off /Volumes/boot

==== Directories ====
# sudo mkdir -p /Volumes/boot/System/Library/CoreServices
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/Library/Preferences/SystemConfiguration
# sudo mkdir -p /Volumes/boot/com.apple.boot.R/usr/standalone/i386

==== Required Files ====
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/PlatformSupport.plist \
/Volumes/boot/System/Library/CoreServices/PlatformSupport.plist
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/SystemVersion.plist \
/Volumes/boot/System/Library/CoreServices/SystemVersion.plist
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels/prelinkedkernel
# sudo rsync -a /Volumes/Capitan/Library/Preferences/SystemConfiguration/com.apple.Boot.plist \
/Volumes/boot/com.apple.boot.R/Library/Preferences/SystemConfiguration/com.apple.Boot.plist
# sudo rsync -a /Volumes/Capitan/usr/standalone/i386/ \
/Volumes/boot/com.apple.boot.R/usr/standalone/i386/

==== Bless the volume ====
The boot helper needs to know where to load boot.efi from.

You may use any label in place of '''ZFS Boot''' here, it will appear at the option-boot screen.
# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
--file /Volumes/boot/System/Library/CoreServices/boot.efi \
--bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi \
--label "ZFS Boot"

This step does not change the current boot device, unless you add --setBoot. See [[#Blessing the boot device|blessing the boot device]] to change the startup device.

==== Generate prelinkedkernel ====
Generate caches for '''kernel'''

# sudo kextcache -arch x86_64 -local-root -volume-root /Volumes/Capitan \
-kernel /Volumes/Capitan/System/Library/Kernels/kernel \
-prelinked-kernel /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
-- /Volumes/Capitan/System/Library/Extensions /Volumes/Capitan/Library/Extensions /Volumes/boot/Library/Extensions

If you get a symlink error here like;
symlink("/Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel", "/Volumes/Capitan/System/Library/Caches/com.apple.kext.caches/Startup/kernelcache") failed 17 (File exists) <createPrelinkedKernel 2795>

Just delete the symlink, and run again, ie;
rm /Volumes/Capitan/System/Library/Caches/com.apple.kext.caches/Startup/kernelcache

For developers who also have '''development''', or '''debug''' kernels, do the same for each one. For example, caches for '''kernel.development'''

# kextcache -arch x86_64 -local-root -volume-root /Volumes/Capitan \
-kernel /Volumes/Capitan/System/Library/Kernels/kernel.development \
-prelinked-kernel /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel.development \
-- /Volumes/Capitan/System/Library/Extensions /Volumes/boot/Library/Extensions

==== Copy prelinkedkernel to boot helper ====

# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/com.apple.boot.R/System/Library/PrelinkedKernels/prelinkedkernel

==== Set the boot device ====
Finally, tell your Mac to use the boot helper on boot. You may want to try the '''--nextonly''' option to avoid permanent changes.
# sudo bless --device /dev/disk1s3 --setBoot

=== Booting ===
==== Booting on ESX ====

This is only required if you prefer not to bless the device. If you bless it, it will boot automatically. However, if you skip the blessing part you can easily chose which HDD to boot
by using the EFI Boot Shell.

Booting on ESX, hit F2 to get the EFI menu up. One of the options is the "EFI Shell (not supported)". Once the EFI shell has been entered

# map

will list all available maps. For me,

'''fs0''' disk0 EFI partition
'''fs1''' disk0 OSX boot
'''fs2''' disk0 OSX Recovery
'''fs3''' disk1 EFI
'''fs4''' disk1 '''boot HFS'''
'''fs5''' disk1 recovery
'''fs6''' disk1 ZFS

So in my case
EFI> '''fs4:'''
fs4: ls
usr System Library
fs4: cd System
fs4: cd Library
fs4: cd CoreServices
fs4: boot.efi

Which runs (boots) boot.efi, which then locates and loads the prelinkedkernel.
You can specify boot options here too:

fs4: boot.efi zfs_boot=rpool -s -v

To ask it to boot from pool '''rpool''' (using bootfs= property) into single-user mode, with verbose output.

==== Booting on VirtualBox ====

SIP can be disabled in VirtualBox, but must be disabled before every boot.
One way to do this is to install '''rEFInd''' into the EFI partition.
It is possible to write a startup.nsh script to do so.

[Please provide details on how to disable SIP in startup.nsh]

VirtualBox does not like the boot helper as created above. There are two options to get a bootable setup:

===== Minimal boot partition for VirtualBox (tested with macOS Sierra) =====
# sudo mkdir -p /Volumes/boot//Library/Preferences/SystemConfiguration
# sudo mkdir -p /Volumes/boot/System/Library/CoreServices/com.apple.recovery.boot
# sudo rsync -a /Volumes/Capitan/Library/Preferences/SystemConfiguration/autodiskmount.plist /Volumes/boot/Library/Preferences/SystemConfiguration/autodiskmount.plist
# sudo rsync -a /Volumes/Capitan/Library/Preferences/SystemConfiguration/com.apple.Boot.plist /Volumes/boot/Library/Preferences/SystemConfiguration/com.apple.Boot.plist
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/PlatformSupport.plist /Volumes/boot/System/Library/CoreServices/PlatformSupport.plist
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/SystemVersion.plist /Volumes/boot/System/Library/CoreServices/SystemVersion.plist
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/boot.efi /Volumes/boot/System/Library/CoreServices/boot.efi
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/bootbase.efi /Volumes/boot/System/Library/CoreServices/bootbase.efi
# sudo rsync -a /Volumes/Capitan/System/Library/CoreServices/com.apple.recovery.boot/PlatformSupport.plist /Volumes/boot/System/Library/CoreServices/com.apple.recovery.boot/PlatformSupport.plist
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel /Volumes/boot/System/Library/PrelinkedKernels/prelinkedkernel

===== Alternative solution using recovery boot base system image =====
# diskutil mount disk0s3
# hdiutil attach -readonly "/Volumes/Recovery HD/com.apple.recovery.boot/BaseSystem.dmg"

Then
# sudo rsync -axH "/Volumes/OS X Base System/" "/Volumes/boot/"
Or if you prefer '''asr''':
# sudo asr restore -source "/Volumes/OS X Base System/" -target "/Volumes/boot" -erase
# diskutil rename "/Volumes/OS X Base System 1" "boot"

And the paths used above will change, as we are using a “real” disk layout with '''no com.apple.boot.R''':
# sudo rsync -a /Volumes/Capitan/System/Library/PrelinkedKernels/prelinkedkernel \
/Volumes/boot/System/Library/PrelinkedKernels/prelinkedkernel
# sudo vim /Volumes/boot/Library/Preferences/SystemConfiguration/com.apple.Boot.plist
(See [[#com.apple.Boot.plist|Boot.plist]])

==== Booting on real Macs ====

Always make extra backups!
Test this out using flash drives first!

Create partitions as above. If you already created a boot helper on one disk, you can clone it with Disk Utility or '''asr''' and edit the [com.apple.Boot.plist] to put a different pool name in '''zfs_boot''' in the '''Kernel flags'''.

''I was unable to boot Sierra on 2016 MacBookPro with "com.apple.boot.R" setup, but the minimal VirtualBox boot partition solution above has worked just fine.''

Hold '''option''' while powering on or restarting your Mac.

Look for '''ZFS Boot''' as set by bless --label above, and double click.

ZFS will load and search available disks for the pool specified in '''zfs_boot''' above. If the pool is not found, or not enough disks are present, ZFS will wait and check additional disks as they appear.

If all is well, the pool will import and MacOS will boot from the '''Capitan''' dataset.

After a certain number of attempts, the MacOS loader will show a 'prohibitory symbol' (circle with a line through it). If you see this, hold the power button until your Mac powers off.

If you are stuck in a boot loop, power the machine off, then power on while holding the '''option''' key. Select the partition with standard MacOS to reboot normally. Once booted, check and set '''System Preferences->Startup Disk'''.

=== Additional ===

==== Mounting other boot environments ====

# sudo zfs snapshot rpool/ROOT/Capitan1@updating
# sudo zfs clone rpool/ROOT/Capitan1@updating rpool/ROOT/Capitan2
# mkdir /Volumes/Capitan2
# sudo mount_zfs rpool/ROOT/Capitan2 /Volumes/Capitan2

Later you can promote
# sudo zfs promote rpool/ROOT/Capitan2
Then if you want
# sudo zfs destroy -r rpool/ROOT/Capitan

==== Blessing the boot device ====

Need only be run once, to setup the '''boot''' volume:
# sudo bless --folder /Volumes/boot/System/Library/CoreServices \
--file /Volumes/boot/System/Library/CoreServices/boot.efi \
--bootefi /Volumes/Capitan/System/Library/CoreServices/boot.efi

This command actually sets the boot helper to be used on reboot:
# sudo bless --device /dev/disk1s3 --setBoot --verbose

This will also work, as it will locate the boot helper following the ZFS partition:
# sudo bless --device /dev/disk1s2 --setBoot --verbose

Currently --folder and --mount options will work (partially) but may select the wrong boot helper if there are multiple bootable ZFS pools.
(For example, if you have ZFS boot setup on both the internal disk and an external disk.)
# sudo bless --folder /Volumes/Capitan --setBoot --verbose
# sudo bless --mount /Volumes/Capitan --setBoot --verbose

Versioning

2019-05-28T01:43:16Z

210.172.128.230:

<div class="noautonum">__TOC__</div>
{| class="wikitable"
|-
! O3X Version !! Tag || ZOL !! Product !! Module !! CFBundleVersion !! ZFS pkg !! SPL kernel pkg !! ZFS kernel pkg !! ZFS user pkg
|-
| 1.9.0 || 1.9.0 || 0.8.0 || 1.9.0 || 17.0.8/9/10/11/12/13 || 1.9.0 || 17.0 || - || - || -
|-
| 1.8.1 || 1.8.1 || 0.8.0 || 1.8.1 || 16.0.8/9/10/11/12/13 || 1.8.1 || 16.0 || - || - || -
|-
| 1.7.4 || 1.7.4 || 0.7.9 || 1.7.4 || 15.0.8/9/10/11/12/13 || 1.7.4 || 15.0 || - || - || -
|-
| 1.7.3 || 1.7.3 || 0.7.7 || 1.7.3 || 14.0.8/9/10/11/12/13 || 1.7.3 || 14.0 || - || - || -
|-
| 1.7.2 || 1.7.2 || 0.7.7 || 1.7.2 || 13.0.8/9/10/11/12/13 || 1.7.2 || 13.0 || - || - || -
|-
| 1.7.1 || 1.7.1 || 0.7.6 || 1.7.1 || 12.0.8/9/10/11/12/13 || 1.7.1 || 12.0 || - || - || -
|-
| 1.7.0 || 1.7.0 || 0.7.3 || 1.7.0 || 11.0.8/9/10/11/12/13 || 1.7.0 || 11.0 || - || - || -
|-
| 1.6.2 || 1.6.2 || 0.6.5.10 || 1.6.2 || 10.0.8/9/10/11/12/13 || 1.6.2 || 10.0 || - || - || -
|-
| 1.6.1 || 1.6.1 || 0.6.5.8 || 1.6.1 || 9.0.8/9/10/11/12 || 1.6.1 || 9.0 || - || - || -
|-
| 1.5.2 || 1.5.2 || 0.6.5 || 1.5.2 || 8.0.8/9/10/11 || 1.5.2 || 8.0 || - || - || -
|-
| 1.4.5 || 1.4.5 || 0.6.4 || 1.4.5 || 7.0.8/9/10/11 || 1.4.5 || 7.0 || - || - || -
|-
| 1.4.2 || 1.4.2 || 0.6.4 || 1.4.2 || 6.0.8/9/10 || 1.4.2 || 6.0 || - || - || -
|-
| 1.3.1-r2 || 1.3.1-2 || 0.6.3 || 1.3.1 || 5.0.8/9 || 1.3.1 || 5.0 || - || - || -
|-
| 1.3.1 || 1.3.1-1.1 || 0.6.3 || 1.3.1 || 5.0.8/9 || 1.3.1 || 5.0 || - || - || -
|-
| 1.3.0 || 1.3.0 || 0.6.3 || 1.3.0 || 3.0.8/9 || 1.3.0 || 4.0 || - || - || -
|-
| 1.2.7 || - || 0.6.2 || 1.7.0 || 1.0.0 || 1.0.0 || - || 2.1 || 2.12 || 2.12
|-
| 1.2.0 || - || 0.6.2 || 1.2.0 || 1.0.0 || 1.0.0 || - || 1.0 || 1.2 || 1.2
|}

Note: For 1.3.1+, "Tag" should match the output of <code>sysctl -a | grep kext_version</code>

== 1.3.1-r2 ==
File: OpenZFS_on_OS_X_1.3.1-r2.dmg

OpenZFS on OS X 1.3.1 Mountain Lion.pkg (1.3.1):
* zfs108.pkg (5.0)
OpenZFS on OS X 1.3.1 Mavericks or higher.pkg (1.3.1):
* zfs109.pkg (5.0)

== 1.3.1 ==
File: OpenZFS_on_OS_X_1.3.1.dmg

OpenZFS on OS X 1.3.1 Mountain Lion.pkg (1.3.1):
* zfs108.pkg (5.0)
OpenZFS on OS X 1.3.1 Mavericks or higher.pkg (1.3.1):
* zfs109.pkg (5.0)

== 1.3.0 ==
File: OpenZFS_on_OS_X_1.3.0.dmg

OpenZFS on OS X 1.3.0 Mountain Lion.pkg (1.3.0):
* zfs108.pkg (4.0)
OpenZFS on OS X 1.3.0 Mavericks or higher.pkg (1.3.0):
* zfs109.pkg (4.0)

== 1.2.7 ==
File: OpenZFS_on_OS_X_1.2.7.dmg

OpenZFS on OS X 1.2.7.pkg (1.7.0):
* splkernel108.pkg (2.1)
* splkernel109.pkg (2.1)
* zfskernel108.pkg (2.12)
* zfskernel109.pkg (2.12)
* zfsuser108.pkg (2.12)
* zfsuser109.pkg (2.12)

== 1.2.0 ==
File: OpenZFS_on_OS_X_1.2.0.dmg

OpenZFS on OS X 1.2.0.pkg (1.2.0):
* splkernel106.pkg (1.0)
* splkernel106i386.pkg (1.0)
* splkernel109.pkg (1.0)
* zfskernel106.pkg (1.2)
* zfskernel106i386.pkg (1.2)
* zfskernel109.pkg (1.2)
* zfsuser106.pkg (1.2)
* zfsuser106i386.pkg (1.2)
* zfsuser109.pkg (1.2)