OpenZFS on OS X - User contributions [en]

O3XWiki:Donations

2016-05-30T23:38:05Z

210.172.146.228: /* Additional Thanks to Donors */

== Thanks ==

The OpenZFS on OS X project would like to thank the following companies:

'''GMO Internet''' [http://www.gmo.jp] for the hosting and rack space

'''GlobalSign''' [http://globalsign.com] for the Open Source free SSL certificate

'''OpenZFS''' [http://open-zfs.org] The main ZFS software collective

== Donations ==

The best way to show your appreciation for the OpenZFS project is to donate to the upstream project at [http://open-zfs.org http://open-zfs.org]

If you wish to donate specifically to the OS X project, you can do so with PayPal at '''japan@lundman.net'''. But be aware any donations will most likely be used on beer and pizza, and other OpenZFS conferences, possibly not on the feature you wish for. :)

=== Additional Thanks to Donors ===

Tommy Lätti
Daniel Bretoi
Luke Lorenz
Tommy Thorn
Tim Henrion
John Parnaby
John Douglass
Daniel Pearson
John Wood
Raoul Callaghan
Josh Jordan
Ottmar Klaas

and those who wished to remain anonymous!

Changelog

2015-12-16T01:11:06Z

210.172.146.228:

=== OpenZFS_on_OS_X_1.?.?.dmg ===

* SPL: enhanced kmem pressure system ''(rottegift)''
* SPL: Rewrite TSD using AVL tree ''(Jorgen Lundman)''
* Cache names in getattr ''(Jorgen Lundman)''
* InvariantDisks serial fixes ''(cbreak)''
* Hardlink LinkID fixes ''(Jorgen Lundman)''
* ACL fixes (trivials and group) ''(Jorgen Lundman)''
* IOkit deadlock on export fixes ''(Jorgen Lundman)''
* MAF and deadlocks in zvol fixes ''(Jorgen Lundman)''
* 2605 want to resume interrupted zfs send ''(Matthew Ahrens)''

=== OpenZFS_on_OS_X_1.4.5.dmg 2015-10-19 ===

* Remove deadlock with zil_lwb_commit ''(Jorgen Lundman)''
* Remove memory leak in znodes leading to beachball ''(Jorgen Lundman)''
* Do not call ctldir unmount ''(Jorgen Lundman)''
* xcode 7 compile fixes ''(ilovezfs)''
* Adhere to SIP in installer on EC ''(ilovezfs)''

=== OpenZFS_on_OS_X_1.4.2.dmg 2015-09-24 ===

* correct kernel thread priorities ''(Jorgen Lundman)''
* VFS nolocks rework from 10a286 ''(Jorgen Lundman)''
* vnop_pageout_v2 replacement ''(Jorgen Lundman)''
* Permanent Storage work, incomplete ''(Jorgen Lundman)''
* mmapped file data written twice fix ''(Jorgen Lundman)''
* InvariantDisks fixes ''(ilovezfs)'' ''(cbreak)''
* SA corruption fixes ''(ZFSOnLinux)''
* SA recover status alerts when detected ''(Jorgen Lundman)''
* Modify-After-Free bugs and deadlock fixes ''(Jorgen Lundman)''
* Complete Re-port of IllumOS taskq ''(Jorgen Lundman)''
* Revert back to using taskq_dispatch_ent() ''(Jorgen Lundman)''
* Remove async unlinkeddrain ''(Jorgen Lundman)''
* Remove internal unused flag XATTR ''(Brendon Humphrey)''
* Additional ioctls from HFS ''(Brendon Humphrey)''
* Merge with upstream ZOL 20150520
* New pool feature "filesystem_limits"
* New pool feature "large_blocks"

=== OpenZFS_on_OS_X_1.3.2-RC1 2015-05-02 ===
(Forum build)
* Remove serial console debug messages ''(Jorgen Lundman)''
* uiocopy failed to set direction ''(Jorgen Lundman)''
* SMAP work for Broadwell chipset ''(Jorgen Lundman)''
* Device removal panic fixes ''(Jorgen Lundman)''
* LASTUNMOUNT property was incorrect ''(Jorgen Lundman)''
* getxattr FinderInfo fixes ''(Jorgen Lundman)''

=== OpenZFS_on_OS_X_1.3.1-r2.dmg 2015-04-08 ===

* vnop_pagein to zero non-aligned trailing block causing clang to core ''(Jorgen Lundman)''
* ADDEDTIME should updated when moved to a different directory ''(Jorgen Lundman)''
* Remove vnode throttle ''(Jorgen Lundman)''
* zfs create -p fix for non-root ''(ilovezfs)''

=== OpenZFS_on_OS_X_1.3.1.dmg 2015-04-01 ===

* readonly mounts failed to unmount ''(Jorgen Lundman)''
* readonly import failed to create ZVOL devnodes ''(Jorgen Lundman)''
* vnode_getwithvid panic race ''(Jorgen Lundman)''
* sa_modify_attrs SA corruption ''(Tim Chase)''
* zconfigd added for persistent config ''(Brendon Humphrey, ilovezfs)''
* Fix missing FIFO named-pipes VNOPs ''(Jorgen Lundman)''
* Fake HFS related IOCTLs for _kMDQueryScope filter in Spotlight ''(Jorgen Lundman)''
* Add new 10.10 ATTR to vnop_getattr ''(Jorgen Lundman)''
* FNDRINFO and ADDEDTIME support ''(Jorgen Lundman)''
* InvariantDisks fixes ''(ilovezfs)''
* Skip optical media on zpool import and add timeout ''(Jorgen Lundman)''
* Initial secpolicy framework ''(ilovezfs)''
* zpool status -L to resolve symlinks ''(ilovezfs)''
* mmap pageout/pagein partial requests fix ''(Jorgen Lundman)''
* kstat fixes and addition ''(Jorgen Lundman)''
* Unmount/reboot delay fixed, direct reclaim ''(Jorgen Lundman)''
* rollback/suspendfs would delay waiting for reclaim ''(Jorgen Lundman)''
* 'com.apple.mimic_hfs' property added to identify as 'hfs' ''(Brendon Humphrey)''

=== 1.3.1-RC5 ===

spl [https://github.com/openzfsonosx/spl/commit/367a1108b174ee81e4ed128741b23e797afb8f16 367a1108b174ee81e4ed128741b23e797afb8f16]

zfs [https://github.com/openzfsonosx/zfs/commit/63a9a59e7de2353a974da0fe65004f59a8bf5946 63a9a59e7de2353a974da0fe65004f59a8bf5946]

* New daemon called "InvariantDisks" providing persistent paths to use with the zpool command: /var/run/disk/by-id, by-path, by-serial (cf. https://github.com/cbreak-black/InvariantDisks) ''(Gerhard Röthlin)''
* Speed up ZVOL unmap by skipping unmaps that are fully unaligned and by only using zil_commit for unmap if sync=always ''(Evan Susarret and Jorgen Lundman)''
* Fix lacking force-positive mount options ''(Jorgen Lundman)''
* Simplified zed daemonization ''(ilovezfs)''
* Bump Spotlight auto-enable until Sun, 01 Feb 2015 00:00:00 UTC ''(ilovezfs)''
* Fix Finder tags modification bug by truncating xattr before overwriting. ''(Jorgen Lundman)''

=== 1.3.1-RC4 ===

spl [https://github.com/openzfsonosx/spl/commit/367a1108b174ee81e4ed128741b23e797afb8f16 367a1108b174ee81e4ed128741b23e797afb8f16]

zfs [https://github.com/openzfsonosx/zfs/commit/96c4b5c8296e7482abfb6b2f018ef932b68248cf 96c4b5c8296e7482abfb6b2f018ef932b68248cf]

* ZFS: Return correct VA_NAME in vnop_getattr for dataset mountpoints ''(Jorgen Lundman)''

=== 1.3.1-RC3 ===

spl [https://github.com/openzfsonosx/spl/commit/367a1108b174ee81e4ed128741b23e797afb8f16 367a1108b174ee81e4ed128741b23e797afb8f16]

zfs [https://github.com/openzfsonosx/zfs/commit/73ead71a49e2530ecfef8017b3552b37e11c65e4 73ead71a49e2530ecfef8017b3552b37e11c65e4]

* ZFS: ZEVO empty SA panic fix ''(Jorgen Lundman)''
* Set B_NOCACHE to stop possibly double caching block data ''(Jorgen Lundman and Evan Susarret)''
* arcstat.pl included

=== 1.3.1-RC2 ===

spl [https://github.com/openzfsonosx/spl/commit/f4581407d18ea555fe5cd07e9e7912e96575ac5d f4581407d18ea555fe5cd07e9e7912e96575ac5d]

zfs [https://github.com/openzfsonosx/zfs/commit/8bf68a82822d492ec9aae0bc8e93d2917ec79937 8bf68a82822d492ec9aae0bc8e93d2917ec79937]

* ZFS: Release XATTRs in vnop_remove quicker ''(Jorgen Lundman)''
* ZFS: Early clearing of z_vnode cause NULL vp panic ''(Jorgen Lundman)''
* ZFS: Fix deadlock in vnop_reclaim ''(Jorgen Lundman)''

=== 1.3.1-RC1 ===

spl [https://github.com/openzfsonosx/spl/commit/8c89b46ca872572281ed62b506958a66a912f243 8c89b46ca872572281ed62b506958a66a912f243]

zfs [https://github.com/openzfsonosx/zfs/commit/91b0052b9167c5447ee8c29d90126af3b621acf7 91b0052b9167c5447ee8c29d90126af3b621acf7]

* SPL: kstat support, including tunables. ''(Brendon Humphrey)''
* SPL: change from mutex allocations to inline ''(Jorgen Lundman)''
* SPL: port of IllumOS kmem ''(Brendon Humphrey)''
* memory pressure sensor and memory reap support ''(Brendon Humphrey)''
* Improve unmount/export code ''(Jorgen Lundman)''
* Handle vnop_pageout() calls during vnode_create ''(Jorgen Lundman)''
* Fix reply to getattrlist regarding case sensitivity to fix install of Adobe software ''(Jorgen Lundman)''
* Fix vfs_vget() for Spotlight and SMB. Enable spotlight on mounts. ''(Jorgen Lundman)''
* Fix zfs.util for whole disk checks ''(ilovezfs)''
* Add working arcstat.pl ''(Brendon Humphrey)''
* Work around for legacy mount points and unsupported versions. ''(ilovezfs)''
* Fix bug for fragmentation when spacemap_histogram is disabled ''(ilovezfs)''
* Open disks as root to fix scrub hang as user. ''(Jorgen Lundman)'' ''(ilovezfs)''
* Fix zfs diff ''(Jorgen Lundman)''
* SPL: condvar timeout, fix cache devices sometimes not being used ''(Jorgen Lundman)''
* reclaim restructuring. Enable delete fast path, and actual release of xattrs ''(Jorgen Lundman)''
* enable userquota/groupquota accounting ''(Jorgen Lundman)''
* Temporary fix for missing .Trashes folder ''(Jorgen Lundman)'' ''(ilovezfs)''
* Automatically remove old .metadata_never_index before Dec 15th
* Make unlinked_drain async, and optional user disable ''(Jorgen Lundman)''
* Merge with ZOL upstream-20141120 ''(Jorgen Lundman)''
* Attempt to detech and remove invalid entries on unlinked-drain list ''(Jorgen Lundman)''
* Move mount default to /Volumes ''(ilovezfs)''

Add pool features: async_destroy empty_bpobj lz4_compress spacemap_histogram enabled_txg hole_birth extensible_dataset embedded_data bookmarks

Illumos 5138
Illumos 4753
Illumos 5116
Illumos 5135
Illumos 5139
Illumos 5147
Illumos 5161
Illumos 5177
Illumos 5174
Illumos 5140
Illumos 5117
Illumos 5049
IllumoS 4873
Illumos 4970-4974
Illumos 5034
Illumos 4631
Illumos 4976-4984
Illumos 4914
Illumos 4881
Illumos 4897
Illumos 4390
Illumos 4757, 4913
Illumos 3835
Illumos 4754, 4755
Illumos #4374
Illumos 4368, 4369
Illumos 4370, 4371
Illumos 4171, 4172
Illumos #4756
Illumos #4730
Illumos #4101, #4102, #4103, #4105, #4106

=== OpenZFS_on_OS_X_1.3.0.dmg 2014-07-24 ===

spl [https://github.com/openzfsonosx/spl/commit/80e411aecac0716d779703ecc0f032232bdad91c 80e411aecac0716d779703ecc0f032232bdad91c]

zfs [https://github.com/openzfsonosx/zfs/commit/b223a573025bb5ef84e6e08b74c9f24b61eacc0b b223a573025bb5ef84e6e08b74c9f24b61eacc0b]

* Print the spl version found instead of "v0.01" ''(ilovezfs)''
* Only replace a pre-existing custom icon if it's the snowflake ''(ilovezfs)''
* Run osascript as the logged-in user so the notifications actually show up ''(ilovezfs)''
* Check for ZEVO either still installed or uninstalled but pre-reboot, and display error for the user ''(ilovezfs)''
* Fix "Load the module manually by running ..." when kexts are in /Library/Extensions on OS X 10.9+ ''(ilovezfs)''
* Fix mutex leaks, resulting in eventual panic in "mutex_enter()". ''(Jorgen Lundman)''
* Fix spa_strdup freeing wrong size, causing kmem havok. ''(Jorgen Lundman)''
* Enhance bmalloc to include free size, bounds and use after free; checks. ''(Brendon Humphrey)''
* Fix zdb 'hang' waiting for reclaim_thread ''(Jorgen Lundman)''
* Autoimport work and fixes ''(ilovezfs)''
* sysctl normalization code from legacy port, default off ''(Jorgen Lundman)''
* Fix hang at export due to spotlight references ''(ilovezfs)''
* Reboot hang fix (wait for reclaim thread) ''(Jorgen Lundman)''
* Reboot hang fix, take 2. (zed ignoring TERM) ''(Jorgen Lundman)''
* Added spl_wait_interruptible functions ''(Jorgen Lundman)''
* Merged ZOL-0.6.3
* ZVOL unmap support ''(Evan Susarret)''
* Better disk icon support ''(ilovezfs)''
* onexit fixes, clean zfs send holds ''(Jorgen Lundman)''
* Replace MALLOC calls to use bmalloc for performance ''(Brendon Humphrey)''
* OS X Yosemite 10.10 compile fixes ''(ilovezfs)''
* zp reclaim vs zget remodel fix deadlocks ''(Jorgen Lundman)''
* Support legacy mountpoints ''(ilovezfs)''
* Initial non-root support ''(ilovezfs)''
* Rewrite ioctl after upstream ''(Jorgen Lundman)''
* Normalized lookup panic fix ''(Jorgen Lundman)''

and, of course, all the fixes in ZFS on Linux 0.6.3. Thanks guys!

== OpenZFS_on_OS_X_1.2.7.dmg 2014-05-15 ==

* Merged with ZFSOnLinux pre-0.6.3 dated Apr 8 2014 ''(6ac770b1961b9468daf0c69eae6515c608535789)''
* create_thread( 75%*num_cpus ) would create literal 75 threads, instead of the intended 3 threads on quad core machine ''(Jorgen Lundman)''
* VMEM allocate changed to use bmalloc (slice, SLAB, allocator on top of k_m_a) ''(Brendon Humphrey)''
* Add ZED (ZFS Event Daemon) to handle events (send alerts, emails) on pool issues. ''(Chris Dunlap)''
* name cache fixes (existing files claimed as missing, missing files claimed as existing) ''(Jorgen Lundman)''
* Change pool sync to remove 'idle' pool writes every 30s. ''(Jorgen Lundman)''
* Work around ZFS recv deadlock ''(ilovezfs)''
* vnop_pageout fixes for zerod blocks beyond EOF (POSIX) ''(Jorgen Lundman)''
* Add autoimport, zed startup scripts ''(ilovezfs)''
* ctldir (.zfs) fixes and cleanup ''(Jorgen Lundman)''
* Finder hardlinks fixes ''(Jorgen Lundman)''
* Reclaim fixes, throttle and waiting on vp changes ''(Jorgen Lundman)''
* ZVOL upstream incompatibility fixes ''(Evan Susarret)'' '''*1'''
* ZFS rollback and promote fixes ''(ilovezfs)''
* Rework EFI label, and wholedisk detection, Core Storage ''(Jorgen Lundman, ilovezfs)''

Which should result in greater stability, large performance enhancements, and finally capable of using more of the available memory.

'''The Installer no longer contain 32bit versions.'''

'''*1''' Note that 1.2.0's ZFS Volumes are unintentionally incompatible with other platform version of ZFS, except for volblocksize = 512.

== 1.2.0.dmg 2014-03-13 ==

* First release

Panic

2015-05-12T00:53:06Z

210.172.146.228: /* Alternate symbol lookup with lldb */

== Kernel panics ==

One of the most useful settings to assist with debugging is telling Darwin kernel to keep the symbols from kexts. This can
be set using the nvram command, and requires a reboot.

First check to see if you have any special boot-args set and add the new keepsyms instruction.
# nvram boot-args="keepsyms=y debug=0x144"

and reboot the machine for it to take effect.

[[https://developer.apple.com/library/mac/documentation/Darwin/Conceptual/KernelProgramming/build/build.html#//apple_ref/doc/uid/TP30000905-CH221-BABCCIDH Table 20-1]] in Apple's Kernel Programming Guide has a summary of the meaning of the debug options.

=== Panic decoding ===

If you get a panic but you do not have symbols enabled, it can be decoded using the atos command.

In a standard panic log, you will see something like:

# cd /Library/Logs/DiagnosticReports/
# less Kernel_2014-03-13-093629_OSX109.panic
Backtrace (CPU 0), Frame : Return Address
0xffffff8088843b10 : 0xffffff7f85e25759 : '''0xffffff7f80dcf760'''
0xffffff8088843b40 : 0xffffff7f85e25560 : '''0xffffff7f80dcf423'''
0xffffff8088843be0 : 0xffffff7f85e08f27 : '''0xffffff7f80dc491a'''

Kernel Extensions in backtrace:
net.lundman.spl(1.0)[7F69C13B-C730-3475-99E9-53861AC6C54E]@0xffffff7f85d2a000->0xffffff7f85d36fff
net.lundman.zfs(1.0)[5637421D-EE17-33F1-ACB2-8FA38BC5A5A6]@'''0xffffff7f80d54000'''->0xffffff7f85f38fff

We can then run the command

# xcrun '''atos''' -arch '''x86_64''' -l '''0xffffff7f80d54000''' -o ../zfs/module/zfs/zfs.kext/Contents/MacOS/zfs '''0xffffff7f80dcf760 0xffffff7f80dcf423 0xffffff7f80dc491a'''
got symbolicator for ../zfs/module/zfs/zfs.kext/Contents/MacOS/zfs, base address 0
spa_taskqs_init (in zfs) (spa.c:888)
spa_create_zio_taskqs (in zfs) (spa.c:972)
spa_activate (in zfs) (spa.c:1094)

Which can be repeated for spl, and spl load address as well, if needed.

And for kernel addresses, look for "kernel slide:" value, I assumed 0 in this example
xcrun atos -arch x86_64 -d -o /Volumes/KernelDebugKit/mach_kernel -s 0 0xffffff8000222f79 0xffffff80002dc24e 0xffffff80002f3746

If you are not panicing, but would like to print the stack at a certain point in the kext, you can use

OSReportWithBacktrace("I am here: vp %p\n", vp);

in `printf` style notation.

=== Alternate symbol lookup with lldb ===

Panic:
panic(cpu 5 caller 0xffffff80088d1066): trying to interlock destroyed mutex (0xffffff8029196000)
Backtrace (CPU 5), Frame : Return Address
0xffffff81f49fba80 : 0xffffff8008822fa9
0xffffff81f49fbb00 : 0xffffff80088d1066
0xffffff81f49fbb10 : 0xffffff800889c75e
0xffffff81f49fbbe0 : 0xffffff80088ae60c
0xffffff81f49fbc00 : '''0xffffff7f8a4252e0'''
0xffffff81f49fbdf0 : 0xffffff80089ffea9
net.lundman.zfs(1.0)[0EC79B06-3C9F-3529-8450-42222507F310]@'''0xffffff7f8a33c000'''->0xffffff7f8a545fff

Assuming you have the same build as panic report, in this case 1.2.7
# lldb
(lldb) target create --no-dependents --arch x86_64 module/zfs/zfs #Binary before moved into zfs.kext
(lldb) target modules load --file zfs __TEXT '''0xffffff7f8a33c000'''
(lldb) image lookup --verbose --address '''0xffffff7f8a4252e0'''

Address: zfs[0x00000000000e92e0] (zfs.__TEXT.__text + 950160)
Summary: zfs`zfs_vnop_pageout + 1264 at zfs_vnops_osx.c:1236
Module: file = "/Users/lundman/x/zfs/module/zfs/zfs", arch = "x86_64"
CompileUnit: id = {0x00000000}, file = "/Users/lundman/x/zfs/module/zfs/zfs_vnops_osx.c", language = "c89"
'''LineEntry''': [0xffffff7f8a4252da-0xffffff7f8a4252f0): /Users/lundman/x/zfs/module/zfs/'''zfs_vnops_osx.c:1236'''

zfs_vnops_osx.c:1236
tx = dmu_tx_create(zfsvfs->z_os);
dmu_tx_hold_write(tx, zp->z_id, off, len);
'''dmu_tx_hold_bonus(tx, zp->z_id);'''
err = dmu_tx_assign(tx, TXG_NOWAIT);

Or just for the kernel

(lldb) target create --no-dependents --arch x86_64 mach_kernel
(lldb) target modules load --file mach_kernel --slide 0x000000000b600000
(lldb) image lookup -a 0xffffff800b8d6aa7

=== Links ===

https://developer.apple.com/library/mac/qa/qa1264/_index.html

https://developer.apple.com/library/mac/documentation/Darwin/Conceptual/KernelProgramming/build/build.html#//apple_ref/doc/uid/TP30000905-CH221-BABDGEGF

https://developer.apple.com/library/mac/documentation/Darwin/Reference/Manpages/man8/kext_logging.8.html

Development

2014-03-25T05:13:01Z

210.172.146.228: /* vnode_create thread */

[[Category:O3X development]]
== Development ==

=== Flamegraphs ===

Huge thanks to [http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html BrendanGregg] for so much of the dtrace magic.

dtrace the kernel while running a command:

dtrace -x stackframes=100 -n 'profile-997 /arg0/ {
@[stack()] = count(); } tick-60s { exit(0); }' -o out.stacks

it will run for 60 seconds.

Convert it to a flamegraph

./stackcollapse.pl out.stacks > out.folded
./flamegraph.pl out.folded > out.svg

This is '''rsync -ar /usr/ /BOOM/deletea/''' running;

[[File:rsyncflamegraph.svg|thumb|rsync flamegraph]]

Or running '''bonnie++''' in various stages;

<gallery mode="packed-hover">
File:create.svg|Create files in sequential order|alt=[[File:create.svg]]
File:stat.svg|Stat files in sequential order|alt=Stat files in sequential order
File:delete.svg|Delete files in sequential order|alt=Delete files in sequential order
</gallery>

=== ZVOL block size ===

At the moment, we can only handle block size of 512 and 4096 in ZFS. And 512 is handled poorly. To write a single 512 block, iokit layer will read in 8 blocks (to make up a PAGE_SIZE read) modify the buffer, then write 8 blocks. This makes ZFS think we wrote 8 blocks, and all stats are updated as such. This is undesirable since compression ratio etc can not be reported correctly.

This limitation is in specfs, which is applied to any BLK device created in /dev. For usage with Apple and the GUI, there is not much we can do. But we are planning to create a secondary blk/chr nodes (maybe in /var/run/zfs/dsk/$POOL/$name or similar for compatibility) which will have our implementation attached as vnops. This will let us handle any block size required.

=== vnode_create thread ===

Currently, we have to protect the call to vnode_create() due to the possibility that it calls several vnops (fsync, pageout, reclaim) and have a reclaim thread to deal with that. One issue is reclaim can both be called as a separate thread (periodic reclaims) and as the ''calling thread'' of vnode_create. This makes locking tricky.

One idea is we create a vnode_create thread (with each dataset). The in zfs_zget and zfs_znode_alloc, which calls vnode_create, we simply place the newly allocated zp on the vnode_create thread's ''request list'', and resume execution. Once we have passed the "unlock" part of the functions, we can wait for the vnode_create thread to complete the request so we do not resume execution without the vp attached.

In the vnode_create thread, we pop items off the list, call vnode_create (guaranteed as a separate thread now) and once completed, mark the node done, and signal the process which might be waiting.

In theory this should let us handle reclaim, fsync, pageout as normal upstream ZFS. no special cases required. This should alleviate the current situation where the reclaim_list grows to very large numbers (230,000 nodes observed).

It might mean we need to be careful in any function which might end up in zfs_znode_alloc, to make sure we have a vp attached before we resume. For example, zfs_lookup and zfs_create.

----

The branch '''vnode_thread''' is just this idea, it creates a vnode_create_thread per dataset, when we need to call ''vnode_create()'' it simply adds the '''zp''' to the list of requests, then signals the thread. The thread will call ''vnode_create()'' and upon completion, set '''zp->z_vnode''' then signal back. The requester for '''zp''' will sit in ''zfs_znode_wait_vnode()'' waiting for the signal back.

This means the ZFS code base is littered with calls to ''zfs_znode_wait_vnode()'' (46 to be exact) placed at the correct location. Ie, '''after''' all the locks are released, and ''zil_commit()'' has been called. It is possible that this number could be decreased, as the calls to ''zfs_zget()'' appear to not suffer the ''zil_commit()'' issue, and can probably just block at the end of ''zfs_zget()''. However the calls to ''zfs_mknode()'' is what causes the issue.

'''sysctl zfs.vnode_create_list''' tracks the number of '''zp''' nodes in the list waiting for ''vnode_create()'' to complete. Typically, 0, or 1. Rarely higher.

Appears to deadlock from time to time.

-----

The second branch '''vnode_threadX''' takes a slightly different approach. Instead of a permanent vnode_create_thread, it simply spawns a thread when ''zfs_znode_getvnode()'' is called. This new thread calls ''_zfs_znode_getvnode()'' which functions as above. Call ''vnode_create()'' then signal back. The same ''zfs_znode_wait_vnode()'' blockers exist.

'''sysctl zfs.vnode_create_list''' tracks the number of '''vnode_create threads''' we have started. Interestingly, these remain 0, or 1. Rarely higher.

Has not yet deadlocked.

Conclusions;

* It is undesirable that we have ''zfs_znode_wait_vnode()'' placed all over the source, and care needs to be taken for each one. Although it does not hurt to call it in excess, as no wait will happen if '''zp->z_vnode''' is already set.
* It is unknown if it is OK to resume ZFS execution while '''z_vnode''' is still NULL, and only block (to wait for it to be filled in) once we are close to leaving the VNOP.

* However, that '''vnop_reclaim''' are direct and can be cleaned up immediately is very desirable. We no longer need to check for the '''zp without vp''' case in ''zfs_zget()''.
* We no longer need to lock protect '''vnop_fsync''', '''vnop_pageout''' in case they are called from ''vnode_create()''.
* We don't have to throttle the '''reclaim thread''' due to the list being massive (populating the list is much faster than cleaning up a '''zp''' node - up to 250,000 nodes in the list has been observed).

Development

2014-03-25T05:12:40Z

210.172.146.228: /* vnode_create thread */

[[Category:O3X development]]
== Development ==

=== Flamegraphs ===

Huge thanks to [http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html BrendanGregg] for so much of the dtrace magic.

dtrace the kernel while running a command:

dtrace -x stackframes=100 -n 'profile-997 /arg0/ {
@[stack()] = count(); } tick-60s { exit(0); }' -o out.stacks

it will run for 60 seconds.

Convert it to a flamegraph

./stackcollapse.pl out.stacks > out.folded
./flamegraph.pl out.folded > out.svg

This is '''rsync -ar /usr/ /BOOM/deletea/''' running;

[[File:rsyncflamegraph.svg|thumb|rsync flamegraph]]

Or running '''bonnie++''' in various stages;

<gallery mode="packed-hover">
File:create.svg|Create files in sequential order|alt=[[File:create.svg]]
File:stat.svg|Stat files in sequential order|alt=Stat files in sequential order
File:delete.svg|Delete files in sequential order|alt=Delete files in sequential order
</gallery>

=== ZVOL block size ===

At the moment, we can only handle block size of 512 and 4096 in ZFS. And 512 is handled poorly. To write a single 512 block, iokit layer will read in 8 blocks (to make up a PAGE_SIZE read) modify the buffer, then write 8 blocks. This makes ZFS think we wrote 8 blocks, and all stats are updated as such. This is undesirable since compression ratio etc can not be reported correctly.

This limitation is in specfs, which is applied to any BLK device created in /dev. For usage with Apple and the GUI, there is not much we can do. But we are planning to create a secondary blk/chr nodes (maybe in /var/run/zfs/dsk/$POOL/$name or similar for compatibility) which will have our implementation attached as vnops. This will let us handle any block size required.

=== vnode_create thread ===

Currently, we have to protect the call to vnode_create() due to the possibility that it calls several vnops (fsync, pageout, reclaim) and have a reclaim thread to deal with that. One issue is reclaim can both be called as a separate thread (periodic reclaims) and as the ''calling thread'' of vnode_create. This makes locking tricky.

One idea is we create a vnode_create thread (with each dataset). The in zfs_zget and zfs_znode_alloc, which calls vnode_create, we simply place the newly allocated zp on the vnode_create thread's ''request list'', and resume execution. Once we have passed the "unlock" part of the functions, we can wait for the vnode_create thread to complete the request so we do not resume execution without the vp attached.

In the vnode_create thread, we pop items off the list, call vnode_create (guaranteed as a separate thread now) and once completed, mark the node done, and signal the process which might be waiting.

In theory this should let us handle reclaim, fsync, pageout as normal upstream ZFS. no special cases required. This should alleviate the current situation where the reclaim_list grows to very large numbers (230,000 nodes observed).

It might mean we need to be careful in any function which might end up in zfs_znode_alloc, to make sure we have a vp attached before we resume. For example, zfs_lookup and zfs_create.

----

The branch '''vnode_thread''' is just this idea, it creates a vnode_create_thread per dataset, when we need to call ''vnode_create()'' it simply adds the '''zp''' to the list of requests, then signals the thread. The thread will call ''vnode_create()'' and upon completion, set '''zp->z_vnode''' then signal back. The requester for '''zp''' will sit in ''zfs_znode_wait_vnode()'' waiting for the signal back.

This means the ZFS code base is littered with calls to ''zfs_znode_wait_vnode()'' (46 to be exact) placed at the correct location. Ie, '''after''' all the locks are released, and ''zil_commit()'' has been called. It is possible that this number could be decreased, as the calls to ''zfs_zget()'' appear to not suffer the ''zil_commit()'' issue, and can probably just block at the end of ''zfs_zget()''. However the calls to ''zfs_mknode()'' is what causes the issue.

'''sysctl zfs.vnode_create_list''' tracks the number of '''zp''' nodes in the list waiting for ''vnode_create()'' to complete. Typically, 0, or 1. Rarely higher.

Appears to deadlock from time to time.

-----

The second branch '''vnode_threadX''' takes a slightly different approach. Instead of a permanent vnode_create_thread, it simply spawns a thread when ''zfs_znode_getvnode()'' is called. This new thread calls ''_zfs_znode_getvnode()'' which functions as above. Call ''vnode_create()'' then signal back. The same ''zfs_znode_wait_vnode()'' blockers exist.

'''sysctl zfs.vnode_create_list''' tracks the number of '''vnode_create threads''' we have started. Interestingly, these remain 0, or 1. Rarely higher.

Has not yet deadlocked.

Conclusions;

* It is undesirable that we have ''zfs_znode_wait_vnode()'' placed all over the source, and care needs to be taken for each one. Although it does not hurt to call it in excess, as no wait will happen if '''zp->z_vnode''' is already set.
* It is unknown if it is OK to resume ZFS execution while '''z_vnode''' is still NULL, and only block 8to wait for it to be filled in) once we are close to leaving the VNOP.

* However, that '''vnop_reclaim''' are direct and can be cleaned up immediately is very desirable. We no longer need to check for the '''zp without vp''' case in ''zfs_zget()''.
* We no longer need to lock protect '''vnop_fsync''', '''vnop_pageout''' in case they are called from ''vnode_create()''.
* We don't have to throttle the '''reclaim thread''' due to the list being massive (populating the list is much faster than cleaning up a '''zp''' node - up to 250,000 nodes in the list has been observed).

Development

2014-03-25T03:46:49Z

210.172.146.228: /* vnode_create thread */

[[Category:O3X development]]
== Development ==

=== Flamegraphs ===

Huge thanks to [http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html BrendanGregg] for so much of the dtrace magic.

dtrace the kernel while running a command:

dtrace -x stackframes=100 -n 'profile-997 /arg0/ {
@[stack()] = count(); } tick-60s { exit(0); }' -o out.stacks

it will run for 60 seconds.

Convert it to a flamegraph

./stackcollapse.pl out.stacks > out.folded
./flamegraph.pl out.folded > out.svg

This is '''rsync -ar /usr/ /BOOM/deletea/''' running;

[[File:rsyncflamegraph.svg|thumb|rsync flamegraph]]

Or running '''bonnie++''' in various stages;

<gallery mode="packed-hover">
File:create.svg|Create files in sequential order|alt=[[File:create.svg]]
File:stat.svg|Stat files in sequential order|alt=Stat files in sequential order
File:delete.svg|Delete files in sequential order|alt=Delete files in sequential order
</gallery>

=== ZVOL block size ===

At the moment, we can only handle block size of 512 and 4096 in ZFS. And 512 is handled poorly. To write a single 512 block, iokit layer will read in 8 blocks (to make up a PAGE_SIZE read) modify the buffer, then write 8 blocks. This makes ZFS think we wrote 8 blocks, and all stats are updated as such. This is undesirable since compression ratio etc can not be reported correctly.

This limitation is in specfs, which is applied to any BLK device created in /dev. For usage with Apple and the GUI, there is not much we can do. But we are planning to create a secondary blk/chr nodes (maybe in /var/run/zfs/dsk/$POOL/$name or similar for compatibility) which will have our implementation attached as vnops. This will let us handle any block size required.

=== vnode_create thread ===

Currently, we have to protect the call to vnode_create() due to the possibility that it calls several vnops (fsync, pageout, reclaim) and have a reclaim thread to deal with that. One issue is reclaim can both be called as a separate thread (periodic reclaims) and as the ''calling thread'' of vnode_create. This makes locking tricky.

One idea is we create a vnode_create thread (with each dataset). The in zfs_zget and zfs_znode_alloc, which calls vnode_create, we simply place the newly allocated zp on the vnode_create thread's ''request list'', and resume execution. Once we have passed the "unlock" part of the functions, we can wait for the vnode_create thread to complete the request so we do not resume execution without the vp attached.

In the vnode_create thread, we pop items off the list, call vnode_create (guaranteed as a separate thread now) and once completed, mark the node done, and signal the process which might be waiting.

In theory this should let us handle reclaim, fsync, pageout as normal upstream ZFS. no special cases required. This should alleviate the current situation where the reclaim_list grows to very large numbers (230,000 nodes observed).

It might mean we need to be careful in any function which might end up in zfs_znode_alloc, to make sure we have a vp attached before we resume. For example, zfs_lookup and zfs_create.

----

The branch '''vnode_thread''' is just this idea, it creates a vnode_create_thread per dataset, when we need to call ''vnode_create()'' it simply adds the '''zp''' to the list of requests, then signals the thread. The thread will call ''vnode_create()'' and upon completion, set '''zp->z_vnode''' then signal back. The requester for '''zp''' will sit in ''zfs_znode_wait_vnode()'' waiting for the signal back.

This means the ZFS code base is littered with calls to ''zfs_znode_wait_vnode()'' (46 to be exact) placed at the correct location. Ie, '''after''' all the locks are released, and ''zil_commit()'' has been called. It is possible that this number could be decreased, as the calls to ''zfs_zget()'' appear to not suffer the ''zil_commit()'' issue, and can probably just block at the end of ''zfs_zget()''. However the calls to ''zfs_mknode()'' is what causes the issue.

'''sysctl zfs.vnode_create_list''' tracks the number of '''zp''' nodes in the list waiting for ''vnode_create()'' to complete. Typically, 0, or 1. Rarely higher.

Appears to deadlock from time to time.

-----

The second branch '''vnode_threadX''' takes a slightly different approach. Instead of a permanent vnode_create_thread, it simply spawns a thread when ''zfs_znode_getvnode()'' is called. This new thread calls ''_zfs_znode_getvnode()'' which functions as above. Call ''vnode_create()'' then signal back. The same ''zfs_znode_wait_vnode()'' blockers exist.

'''sysctl zfs.vnode_create_list''' tracks the number of '''vnode_create threads''' we have started. Interestingly, these remain 0, or 1. Rarely higher.

Has not yet deadlocked.

Conclusions;

* It is undesirable that we have ''zfs_znode_wait_vnode()'' placed all over the source, and care needs to be taken for each one. Although it does not hurt to call it in excess, as no wait will happen if '''zp->z_vnode''' is already set.

* However, that '''vnop_reclaim''' are direct and can be cleaned up immediately is very desirable. We no longer need to check for the '''zp without vp''' case in ''zfs_zget()''.
* We no longer need to lock protect '''vnop_fsync''', '''vnop_pageout''' in case they are called from ''vnode_create()''.
* We don't have to throttle the '''reclaim thread''' due to the list being massive (populating the list is much faster than cleaning up a '''zp''' node - up to 250,000 nodes in the list has been observed).

Development

2014-03-18T02:43:36Z

210.172.146.228: /* Development */

== Development ==

Huge thanks to [http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html BrendanGregg] for so much of the dtrace magic.

dtrace the kernel while running a command:

dtrace -x stackframes=100 -n 'profile-997 /arg0/ {
@[stack()] = count(); } tick-60s { exit(0); }' -o out.stacks

it will run for 60 seconds.

Convert it to a flamegraph

./stackcollapse.pl out.stacks > out.folded
./flamegraph.pl out.folded > out.svg

This is '''rsync -ar /usr/ /BOOM/deletea/''' running;

[[File:rsyncflamegraph.svg|thumb|rsync flamegraph]]

Or running '''bonnie++''' in various stages;

<gallery mode="packed-hover">
File:create.svg|Create files in sequential order|alt=[[File:create.svg]]
File:stat.svg|Stat files in sequential order|alt=Stat files in sequential order
File:delete.svg|Delete files in sequential order|alt=Delete files in sequential order
</gallery>

Development

2014-03-18T02:41:55Z

210.172.146.228: /* Development */

== Development ==

Huge thanks to [http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html BrendanGregg] for so much of the dtrace magic.

dtrace the kernel while running a command:

dtrace -x stackframes=100 -n 'profile-997 /arg0/ {
@[stack()] = count(); } tick-60s { exit(0); }' -o out.stacks

it will run for 60 seconds.

Convert it to a flamegraph

./stackcollapse.pl out.stacks > out.folded
./flamegraph.pl out.folded > out.svg

This is '''rsync -ar /usr/ /BOOM/deletea/''' running;

[[File:rsyncflamegraph.svg|thumb|rsync flamegraph]]

Or running '''bonnie++''' in various stages;

<gallery>
File:create.svg|Create files in sequential order [[File:create.svg]]|alt=[[File:create.svg]]
File:stat.svg|Stat files in sequential order|alt=Stat files in sequential order
File:delete.svg|Delete files in sequential order|alt=Delete files in sequential order
</gallery>