This document is the May, 2017 progress report for TPF Standardization, Test Coverage, and Documentation of Perl 6 I/O Routines grant. I believe I reasonably satisfied the goals of the grant and consider it completed. This is the final report and may reference some of the work/commits previously mentioned in monthly reports.

Thank You!

I'd like to thank all the donors that support The Perl Foundation who made this grant possible. It was a wonderful learning experience for me, and it brings me joy to look back and see Perl 6 improved due to this grant.

Thank You!

Completeness Criteria

Here are the original completeness criteria (in bold) that are listed on the original grant proposal and my comments on their status:

rakudo repository will contain the IO Action Plan document and it will be fully implemented. The promised document exists. It's fully implemented except for three items that I listed on the IO Action Plan, but which are currently a bit beyond my skill level to implement. I hope to do them eventually, but outside the scope of this grant. They are: IO::Handle's Closed status. My original proposal would cause some perfomance issues, so it was decided to improve MoarVM errors instead. Optimize multiple stat calls. This involves creating a new nqp op, with code for it implemented in MoarVM and JVM backends. Use typed exceptions instead of X::AdHoc. I made typed exceptions be thrown whereever I could. The rest require VM-level exceptions and is on the same level as the handle closed status issue (first item above).

The promised document exists. It's fully implemented except for three items that I listed on the IO Action Plan, but which are currently a bit beyond my skill level to implement. I hope to do them eventually, but outside the scope of this grant. They are: All of the I/O routines will have tests in roast and documented on docs.perl6.org. If any of the currently implemented but unspecced routines are decided against being included in Perl 6 Language, their implementation will no longer be available in Rakudo. To the best of my knowledge, this is completed in full.

To the best of my knowledge, this is completed in full. The test coverage tool will report all I/O routines as covered and the information will be visible on perl6.wtf (Perl 6's Wonderful Test Files) website. Note: due to current experimental status of the coverage tool, its report may still show some lines or conditionals untested despite them actually being tested; however, it will show the lines where routines' names are specified as covered. To the best of my knowledge, all IO routines currently have tests covering them. Due to its experimental status, the coverage tool shows some attributes as uncovered. I did manually verify all the attributes/routines whose names the tool shows as uncovered contain tests for them. One exception is IO::Notification type (and IO::Path.watch method). While it has full coverage for OSX operating system, it lacks it for other OSes. I tried writing some tests for it, but it looks like the behaviour of the nqp op handling these is broken on Linux and the class needs more work.

Extra Deliverables

I produced these extra deliverables while working on the grant:

The Definitive I/O Guide. Providing tutorial-like documentation for Perl 6's I/O, including documenting some of the bad practices I noticed in the ecosystem (and even a Perl 6 book!) and the correct way to perform those tasks. (N.B. as I write this report, the guide could still use a few extra sections to be considered "The Definitive"; I'll write them in upcoming weeks)

Providing tutorial-like documentation for Perl 6's I/O, including documenting some of the bad practices I noticed in the ecosystem (and even a Perl 6 book!) and the correct way to perform those tasks. (N.B. as I write this report, the guide could still use a few extra sections to be considered "The Definitive"; I'll write them in upcoming weeks) Performance improvements. I made 23 performance-enhancing commits, with many commits making things more than 200% faster, with highest improvement making a routine 6300% faster.

I made 23 performance-enhancing commits, with many commits making things more than 200% faster, with highest improvement making a routine 6300% faster. Trait::IO module. Provides does auto-close pseudo-trait to simplify closing of IO handles.

Provides pseudo-trait to simplify closing of IO handles. IO::Path::ChildSecure module. Due to large ecosystem usage, IO::Path.child was left as is until 6.d language, at which point it will be made secure (as outlined in the IO Plan). This module provides the secure version in the mean time.

Due to large ecosystem usage, was left as is until 6.d language, at which point it will be made secure (as outlined in the IO Plan). This module provides the secure version in the mean time. IO::Dir module. Provides IO::Path.dir -like functionality, with ability to close open directory without needing to fully exhaust the returned Seq .

Provides -like functionality, with ability to close open directory without needing to fully exhaust the returned . Die module. Implements Perl-5-like behaviour for &die routine.

Implements Perl-5-like behaviour for routine. The "Map of Perl 6 Routines" (or rather the "table") is available on map.perl6.party with its code in perl6/routine-map repo. In near future, I plan to use it to identify incorrect or incomplete entries in our documentation

In addition, I plan to complete these modules some time in the future; the ideas for them were birthed while working on the grant: - NL module. Targeted for use in one liners, the module will provide $*NL dynvar that behaves like Perl 5's $. variable (providing current $*ARGFILES 's file's line number). Its implementation became possible thanks to newly-implemented IO::CatHandle type - FastIO module. A re-imagination of core IO, the biggest part of which will be the removal of (user-exposed) use of IO::Spec::* types and $*SPEC variable, which—it is believed—will provide improved performance over core IO. The module is a prototype for some of the proposals that were made during the IO grant and if it offers significant improvements over core IO, its ideas will be used by core IO in future language versions.

Work Performed in May

For the work done in May, many of my commits went into going through the IO routine list, and adding missing tests and documentation, along with fixing bugs (and reporting new ones I found).

The major work was implementation of the IO::CatHandle class that fixed all of the bugs and NYIs with the $*ARGFILES . This work saw the addition of 372 lines of code, 800 lines of tests and 793 lines of documentation.

Work by Other Core Members

jnthn++ completed the handle encoding refactor that will eventually let us get rid of using libuv for syncronous IO and, more importantly, allow us to support user-defined encoders/decoders.

Along with fixing a bunch of bugs, this work altered the performance landscape for IO operations (i.e. some operations may now be a bit faster, others a bit slower), though overall the performance appeared to stay the same.

Tickets Fixed

Grant Commits

During this grant, I've made 417 commits, that are: 134 Rakudo commits + 23 performance-enchancing Rakudo commits + 114 Perl 6 Specification commits + 146 documentation commits,

Performance Rakudo Commits

I've made 23 performance enchancing commits to Rakudo's repository:

4032953 Make IO::Handle.open 75% faster

dcf1bb2 Make IO::Spec::Unix.rel2abs 35% faster

c13480c IO::Path.slurp: make 12%-35% faster; propagate Failures

0e36bb2 Make IO::Spec::Win32!canon-cat 2.3x faster

c6fd736 Make IO::Spec::Win32.is-absolute about 63x faster

894ba82 Make IO::Spec::Win32.split about 82% faster

277b6e5 Make IO::Spec::Unix.rel2abs 2.9x faster

74680d4 Make IO::Path.is-absolute about 80% faster

ff23416 Make IO::Path.is-relative about 2.1x faster

d272667 Make IO::Spec::Unix.join about 40% faster

50429b1 Make IO::Handle.put($x) about 5%-35% faster

204ea59 Make &say(**@args) 70%− faster

6d7fc8e Make &put(**@args) up to 70% faster

76af536 Make 1-arg IO::Handle.say up to 2x faster

aa72bde Remove dir's :absolute and :Str; make up to 23% faster

48cf0e6 Make IO::Spec::Cygwin.is-absolute 21x faster

c96727a Fix combiners on SPEC::Win32.rel2abs; make 6% faster

0547979 Make IO::Spec::Unix.path consistent and 4.6x faster

8992af1 Fix IO::Spec::Win32.path and make 26x faster

7d6fa73 Make IO::Spec::Win32.catpath 47x faster

494659a Make IO::Spec::Win32.join 26x faster

6ca702f Make IO::Spec::Unix.splitdir 7.7x faster

2816ef7 Make IO::Spec::Win32.splitdir 25x faster

Non-Performance Rakudo Commits

Other than perf commits, I've also made 134 commits to the Rakudo's repository:

dd4dfb1 Fix crash in IO::Special .WHICH/.Str

76f7187 Do not cache IO::Path.e results

212cc8a Remove IO::Path.Bridge

a01d679 Remove IO::Path.pipe

55abc6d Improve IO::Path.child perf on *nix

4fdebc9 Make IO::Spec::Unix.split 36x Faster

0111f10 Make IO::Spec::Unix.catdir 3.9x Faster

fa9aa47 Make R::I::SET_LINE_ENDING_ON_HANDLE 4.1x Faster

c360ac2 Fix smartmatch of Cool ~~ IO::Path

0c7e4a0 Do not capture args in .IO method

9d8d7b2 Log all changes to plan made during review period

87987c2 Remove role IO and its .umask method

role IO 36ad92a Remove 15 methods from IO::Handle

a5800a1 Implement IO::Handle.spurt

aa62cd5 Remove &tmpdir and &homedir

a0ef2ed Improve &chdir, &indir, and IO::Path.chdir

ca1acb7 Fix race in &indir(IO::Path …)

2483d68 Fix regression in &chdir's failure mode

5464b82 Improve &*chdir

4c31903 Add S32-io/chdir-process.t to list of test files to run

cb27bce Clean up &open and IO::Path.open

099512b Clean up and improve all spurt routines

b62d1a7 Give $*TMPDIR a container

b1e7a01 Implement IO::Path.extension 2.0

15a25da Fix ambiguity in empty extension vs no extension

50aea2b Restore IO::Handle.IO

966a7e3 Implement IO::Path.concat-with

94a6909 Clean up IO::Spec::Unix.abs2rel a bit

a432b3d Remove IO::Path.abspath (part 2)

954e69e Fix return value of IO::Special methods

67f06b2 Run S32-io/io-special.t test file

a0b82ed Make IO::Path::* actually instantiate a subclass

0c8bef5 Implement :parent in IO::Spec::Cygwin.canonpath

0a442ce Remove type constraint in IO::Spec::Cygwin.canonpath

b4358af Delete code for IO::Spec::Win32.catfile

e681498 Make IO::Path throw when path contains NUL byte

6a8d63d Implement :completely param in IO::Path.resolve

b6838ee Remove .f check in .z

184d499 Make IO::Handle.Supply respect handle's mode

f1b4af7 Implement IO::Handle.slurp

90da80f Rework read methods in IO::Path/IO::Handle

8c09c84 Fix symlink and link routines

da1dea2 Fix &symlink and &link

7f73f92 Make IO::Path.new-from-absolute-path private

ff97083 Straighten up rename, move, and copy

0d9ecae Remove multi-dir &mkdir

6ee71c2 Coerce mode in IO::Path.mkdir to Int

d46e8df Add IO::Pipe .path and .IO methods

c01ebea Make IO::Path.mkdir return invocant on success

1f689a9 Fix up IO::Handle.Str

490ffd1 Do not use self.Str in IO::Path errors

40217ed Swap .child to .concat-with in all the guts

fd503f8 Revert "Remove role IO and its .umask method"

role IO c95c4a7 Make IO::Path/IO::Special do IO role

214198b Implement proper args for IO::Handle.lock

9a2446c Move Bool return value to signature

51e4629 Amend rules for last part in IO::Path.resolve

b8458d3 Reword method child for cleaner code

method child 1887114 Implement IO::Path.child-secure

9d8e391 Fix IO::Path.resolve with combiners; timotimo++

0b5a41b Rename IO::Path.concat-with to .add

a98b285 Remove IO::Path.child-secure

8bacad8 Implement IO::Path.sibling

7112a08 Add :D on invocant for file tests

b2a64a1 Fix $*CWD inside IO::Path.dir's :test Callable

6fa4bbc Straighten out &slurp/&spurt/&get/&getc/&close

34b58d1 Straighten out &lines/&words

d0cd137 Make dir take any IO(), not just Cool

7412184 Make $*HOME default to Nil, not Any

475d9bc Fix display of backslashes in IO::Path.gist

6ef2abd Revert "Fix display of backslashes in IO::Path.gist"

134efd8 Fix .perl for IO::Path and subclasses

69320e7 Fix .IO on :U of IO::Path subclasses

eb8d006 Make IO::Handle.iterator a private lines iterator

08a8075 Fix IO::Path.copy/move when source/target are same

973338a Fix IO::Handle.comb/.split; make them .slurp

b43ed18 Make IO::Handle.flush fail with typed exceptions

276d4a7 Remove .tell info in IO::Handle.gist

f4309de Fix IO::Spec::Unix.is-absolute for combiners on /

06d8800 Fix crash when setting .nl-in ...

7e9496d Make IO::Handle.encoding settable via .new

95e49dc Make IO::Handle.open respect attribute values

6ed14ef Remove :directory from IO::Spec::*.split

:directory 9021a48 Make IO::Path.parts a Map instead of Hash

a282b8c Fix IO::Handle.perl.EVAL roundtrippage

a412788 Make IO::Path.resolve set CWD to $!SPEC.dir-sep

84502dc Implement $limit arg for IO::Handle.words

613bdcf Make IO::Handle.print/.put sig consistent

0646d3f Allow no-arg &prompt

4a8aa27 Implement IO::CatHandle.close

4ad8b17 Implement IO::CatHandle.get

3b668b6 Implement IO::CatHandle.getc

25b664a Implement IO::CatHandle.words

7ebc386 Implement IO::CatHandle.slurp

52b34b7 Implement IO::CatHandle.comb/.split

beaa925 Implement IO::CatHandle.read

ccc90fd Implement IO::CatHandle.readchars

40f4dc9 Implement IO::CatHandle.Supply

0c9aea7 Implement IO::CatHandle.encoding

ee1e185 Implement IO::CatHandle.eof

80686a7 Implement IO::CatHandle.t/.path/.IO/.native-descriptor

993de50 Implement IO::CatHandle.gist/.Str/.opened/.open

677c4ea Implement IO::CatHandle.lock/.unlock/.seek/.tell

e657ed1 Implement IO::CatHandle.chomp/.nl-in

a452e42 Implement IO::CatHandle.on-switch

f539a62 Swap IO::ArgFiles to IO::CatHandle impl

fa7aa1c Implement IO::CatHandle.perl method

21fd2c4 Remove IO::Path.watch

65941b2 Revert "Remove IO::Path.watch"

a47a78f Remove useless :SPEC/:CWD on some IO subs

d13d9c2 Throw out IO::Path.int

Perl 6 Specification Commits

I've made 114 commits to the Perl 6 Specification (roast) repository:

63370fe Test IO::Special .WHICH/.Str do not crash

465795c Test IO::Path.lines(*) does not crash

091931a Expand &open tests

8d6ca7a Cover IO::Path.ACCEPTS

14b6844 Use Numeric instead of IO role in dispatch test

5a7a365 Expand IO::Spec::*.tmpdir tests

f48198f Test &indir

bd46836 Amend &indir race tests

04333b3 Test &indir fails with non-existent paths by default

73a5448 Remove two fudged &chdir tests

86f79ce Expand &chdir tests

430ab89 Test &*chdir

86c5f9c Delete qp{} tests

3c4e81b Test IO::Path.Str works as advertised

ba3e7be Merge S32-io/path.t and S32-io/io-path.t

79ff022 Expand &spurt and IO::Path.spurt tests

1d4e881 Test $*TMPDIR can be temp ed

temp b23e53e Test IO::Path.extension

2f09f18 Fix incorrect test

305f206 Test empty-string extensions in IO::Path.extension

0e47f25 Test IO::Path.concat-with

e5dc376 Expand IO::Path.accessed tests

43ec543 Cover methods of IO::Special

bd8d167 Test IO::Path::* instantiate a subclass

d8707e7 Cover IO::Spec::Unix.basename

c3c51ed Cover IO::Spec::Win32.basename

896033a Cover IO::Spec::QNX.canonpath

7c7fbb4 Cover :parent arg in IO::Spec::Cygwin.canonpath

8f73ad8 Change \0 roundtrip test to \t roundtrip test

b16fbd3 Add tests to check nul byte is rejected

ee7f05b Move is-path sub to top so it can be reused

a809f0f Expand IO::Path.resolve tests

feecaf0 Expand file tests

a4c53b0 Use bin IO::Handle to test its .Supply

7e4a2ae Swap .slurp-rest to .slurp

d4353b6 Rewrite .l on broken symlinks test

416b746 Test symlink routines

8fa49e1 Test link routines

link 637500d Spec IO::Pipe.path/.IO returns IO::Path type object

64ff572 Cover IO::Path/IO::Pipe's .Str/.path/.IO

4194755 Test IO::Handle.lock/.unlock

a716962 Amend rules for last part in IO::Path.resolve

f3c5dae Test IO::Path.child-secure

92217f7 Test IO::Path.child-secure with combiners

39677c4 IO::Path.concat-with got renamed to .add

7a063b5 Fudge .child-secure tests

3b36d4d Test IO::Path.sibling

41b7f9f Test $*CWD in IO::Path.dir(:test) Callable

18d9c04 Cover IO::Handle.spurt

8f78ca6 Test &words with IO::ArgFiles

ea137f6 Cover IO::Handle.tell

71a6423 Add $*HOME tests

95d68a2 Test IO::Path.gist does escapes of backslashes

de89d25 Revert "Test IO::Path.gist does escapes of backslashes"

9e8b154 Test IO::Handle.close can be...

853f76f Test IO::Pipe.close returns pipe's Proc

d543e75 Test IO::Handle.DESTROY closes the handle

1ed18b4 Add test for .perl.EVAL roundtrip with combiners

704210c Test we can roundtrip IO::Path.perl

2689eb1 Test .IO on :U of IO::Path subclasses

40353f1 Test for IO::Handle:D { ... } loops over handle

4fdb850 Test IO::Path.copy/move when source/target are same

98917dc Test IO::Path.dir's absoluteness behaviour

71eebc7 Test IO::Spec::Unix.extension

4495615 Test IO::Handle.flush

60f5a6d Test IO::Handle.t when handle is a TTY

31e3993 Test IO::Path*.gist

c481433 Test .is-absolute method for / with combiners

8ee0a0a Test IO::Spec::Win32.rel2abs with combiners

a41027f Test IO::Handle.nl-in can be set

e82b798 Test IO::Handle.open respects attributes

2c29150 Test IO::Handle.nl-in attribute

03ce93b Test IO::Handle.encoding can be set

8ae81c0 Test no-arg candidate of ¬e

fb61306 Test IO::Path.parts attribute

7266522 Test return type of IO::Spec::Unix.path

6ac3b4a Test IO::Spec::Win32.path

dbbea15 Test IO::Handle.perl.EVAL roundtrips

5eb513c Test IO::Path.resolve sets CWD to $!SPEC.dir-sep

b0c4a7a Test &words, IO::Handle.words, and IO::Path.words

f3d1f67 Test $limit arg with &lines/IO::*.lines

4f5589b Add test for handle leak in IO::Path.lines

4d0f97a Add &put/IO::Handle.put tests

125fe18 Add &prompt tests

939ca8d Test IO::CatHandle.close

9833012 Test IO::CatHandle.get

2f65a72 Test IO::CatHandle.getc

a4a7eaa Test IO::CatHandle.words

1131c09 Add &put/IO::Handle.put tests

80de9b6 Add &prompt tests

bacfd9f Test IO::CatHandle.slurp

e78e3c0 Test IO::CatHandle.comb/.split

f1c1125 Test IO::CatHandle.read

e9e78e1 Test IO::CatHandle.readchars

0479087 Test IO::CatHandle.Supply

71953e3 Test IO::CatHandle.encoding

db4847e Test IO::CatHandle.eof

175ba45 Test IO::CatHandle.t/.path/.IO/.native-descriptor

c6cc66a Test IO::CatHandle.gist/.Str/.opened/.open

dcdac1a Test IO::CatHandle.lock/.unlock/.seek/.tell

f48c26e Test IO::CatHandle.chomp/.nl-in

8afd758 Test IO::CatHandle.DESTROY

c7eff2b Test IO::CatHandle.on-switch

e87e20d Test IO::CatHandle.next-handle

28717f0 Test IO::CatHandle.perl method

432bf94 Test IO::Path.watch

ce1b637 Test IO::Handle.say

0bb6298 Test IO::Handle.print-nl

47c88ab Test IO::Pipe.proc attribute

945621d Test IO::Path.SPEC attribute

5fb4b63 Test IO::Path.CWD/.path attributes

d0e5701 Test IO::Path.Numeric and other .numeric methods

94d7133 Test 0-arg &say/&put/&print

38c61cd Test &slurp() and &slurp(IO::Handle)

Perl 6 Documentation Commits

I've made 146 commits to the Perl 6 Documentation repository: