Grant Proposal: Curating Perl 6 Documentation

The Grants Committee has received the following grant proposal for the September/October 2019 round. Before the Committee members vote, we would like to solicit feedback from the Perl community on the proposal. Due to my delay in posting this, I'm extending the comment period. Please review the proposal below and please comment here by September 27th, 2019. The Committee members will start the voting process following that. - Name: Richard Hainsworth (on IRC #perl6 as finanalyst) - Amount requested: $5000 # Synopsis There is a need for full time work on Perl 6 Documentation. A new system is being implemented with the inevitable problems. There are some 300 open issues on the github repo for perl6/doc. When considering Perl 6 documentation, it is important to distinguish between several issues, some of which are easy to resolve, others involve matters of language design and arose emotion. This curation proposal is to facilitate progress on each of these issues, but will directly seek to close certain types of issues. The [currently on-going documentation redesign](https://github.com/perl6/doc/issues/2983) will split the existing `perl6/doc` repo into at least four new repos. One repo will be dedicated solely to the content of the documents, and the others to the software for processing and testing the documents. This means that the 300 or so issues already registered will need to be reassigned to appropriate repos. Some of the 300 issues will disappear as a result of the split. This proposal is in essence a continuation of the work done by JJ Merelo under a previous Perl Foundation Grant, see his [final report](https://jj.github.io/TPF-Grant/Final), together with his continuing development and the GSoC projects. Several contributors to Perl 6 regularly update the documentation. This is a process that should be facilitated by curation. In the categories of issues below, some of the issues naturally generate emotion. The task of a curator is not to make decisions and make changes arbitrarily, but to identify what sort of changes are being proposed, and hence facilitate progress. # Usefulness to Perl 6 community Documentation is essential to any language. Perl 6 is fairly well documented, but given the size of the language, there remain parts where documentation does not exist or is not systematic. This is an on-going process and works quite well. However, documenting something does not mean the documentation is easily available. There remain issues with the delivery to the user of the documentation. In essence, the documentation site is in need of some TLC. JJ Merelo's [proposal](http://news.perlfoundation.org/2018/02/grant-proposal-curating-and-im.html) already sets out much of the case for this proposal. This proposal was made at the suggestion of JJ. # Categories of Issues - Documenting Perl 6 as used by a programmer, which can be divided into - questions that pertain to the Perl 6 language The documentation is constantly being updated and improved. The curation proposal is to facilitate answers being included into the documentation. Sometimes, there is an issue where documented behaviour, eg., sample code, does not work. - the release of 6.d required changes to the documentation, some of these issues still need resolving. - questions that pertain to the Rakudo implementation There was a debate about whether or how to document classes and routines that are implementation specific. The issue - I understand from JJ - has been resolved and everything is being documented, with Rakudo specific classes etc, being mentioned. - The software that renders documentation content into ** html ** for the online version The software for doing this was created in an organic manner and has become fragile. Considerable effort has already been spent on refactoring the software. There is a milestone issue that needs to be closed so that the refactoring is completed. Most of the work of the curation proposal will be to close this milestone issue. - Improving the software for using and locating information within existing content. There are problems finding certain types of entities, eg "`.`" Another question relates to the relationship graphs, sometimes the graphs are too large, sometimes too small. It is an aim of the curation project to move forward on these questions. The first step will be to work on caching the existing graphs. - The software that can be used on a PC with an installed distribution and additional installed modules The primary tool is p6doc, which has been improved and will work for the standard Perl 6 documentation (that is the locally installed clone of perl6/doc). However, it needs to be extended to handle documentation in locally installed modules. It is an aim of the curation project to make progress on this. - Issues related to the POD6 specification and the integration of the compiler and documentation. It is an aim of the curation project to collect these issues and facilitate progress. - Issues relating to the "*look and feel*" of the Documentation site These tend to be enhancements that appeal to individuals or are reactions to documentation sites for other languages/software. The curation proposal is to collect these and facilitate a decision about them. # Deliverables - Review and re-allocate issues to the appropriate repos after the redesign. This requires a review of all 300 existing issues. - Some issues have been closed and re-opened by reference. These zombies need to be removed. - Write a blog report every two weeks regarding the issues closed and identifying questions which have wider implications. - Close issues related to generation of online **html** - Close issues related to `p6doc` - Improve efficiency of generation of **html** site. # Bio Born in 1957, I have lived 32 years in Moscow, then 5 years in Hong Kong, and now I am in Wales. I have been using Perl since the mid 1990s, and followed the development of Perl 6 from its inception. I have written several Perl 6 modules, including software that gathers data and presents the Citation index. The Citation index can be found via a link on the modules.perl6.org page (currently the data upto March 2019, which is when I relocated from Hong Kong to the UK and updating ceased). I have been involved in the documentation project already, working with JJ Merelo. I wrote the Pod::To::Cached module, which underlies Perl6::Documentable, and which has been moved into the Perl 6 organisation repo. I have also contributed to Perl 6 documentation when I have come across problems with it, for example there is a short tutorial in the NativeCall documentation showing how to write the Perl 6 interface to a useful C function in a common library. I was a mentor for the recent GSoC Perl 6 work on p6doc and Documentable. I have worked with the GRAV CMS, and contributed a number of published modules to that project. Currently I am able to allocate at least one full working day a week full time to this project. The project request appears to be reasonable for part time work for two months.

Category: (none)

Comments (6)

We need all the help in documentation and its tooling. Richard has been very helpful already, and in fact started the foundational module, Pod::To::Cached, for the new documentation tools, so it will be great if this work can be continued.



Hello.



Firstly, it is not clear from the proposal how much hours will be spent. Considering "a day a week full time" and "work for two months" results in 8 * 4 * 2, which is 64 hours. Calculating the rate it appears to be times bigger compared to a "normal" core contributor and is bigger than a rate of core developers who work on the most tricky parts of Rakudo and MoarVM. Am I correct?



Secondly, some parts of the current documentation system are not optimal in a long term perspective, for example, versioning. There is a 6.c version spec, a 6.d version spec, and preparation for 6.e version spec, possibly more to come. Currently, there are no separation of documentation for them, and changes are noted in different styles, for example "In 6.foo version, this works in another way, foo bar baz". This adds up confusion, hard to maintain and is not very sustainable. A better option would be to have a clear distinction on the site, allowing the user to check the version.



Examples be https://docs.python.org/3/ (a select at the top).

Or http://www.cplusplus.com/reference/cmath/acos/ (tabs with differences).

Or https://docs.oracle.com/javase/8/docs/api/ (a spec of edition 8, but it is there for 7, 9, 10 etc).



This proposal says nothing about the issue and considering the rate, it would be interesting to explore ways to address it.



Thirdly, the same goes for languages (natural ones). There are already some attempts for translating the documentation existing and if we plan to make the language more popular, appearing of translations is inevitable (and we should strive for it). Having means to make it easy is both related to versions and important.



Fourthly, the site search. With all respect, it is known to be incomplete, sometimes inconsistent and have usability issues. For example, even search categories are not standardized.

As coming from an experienced Perl 6 user, I feel like this proposal should address both search consistency and its numerous usability issues.



I do not think I can strongly disagree with this proposal nor I can strongly agree with it, but I had to point out the things above.



10 days of work for $5000? $500/day to triage documentation tickets? Looking at the work other people have done for similar amounts makes me wonder what I’m missing here...



Thanks for the feedback. And the questions are all understandable.

a) Costing. I looked at recent grant proposals and at the time spent on them. There have been grants from USD 5000 - 10,000. I have taken $5000 from JJ Merelo's proposal. I have requested that JJ supervises this grant proposal and if he thinks I am not putting in enough work/time, I will work more hours.

b) Triaging the documentation tickets is NOT the only part of the proposal, but it is the one where a concrete mile (inch) stone can be set. The proposal also contains goals for work on p6doc, html & other render formats (via the --doc module switch of the perl6 compiler).

c) Some of the documentation tickets are in themselves quite onerous, and some have already taken weeks of work. For example, to bring the content up to date with 6.d, new pages need to be written for CompUnit et al.



Regarding some of the other requests for the documentation site, namely (a) other languages, (b) site versioning for language version, (c) search functionality.



Multi lingual support is a major can of worms because of the need for synchronising content translations. Since the English content is constantly being changed, currently about a dozen changes a week, in over a hundred source files, and given that translations do not map directly, simply tracking new changes and locating in the other language source files which content needs changing, is non-trivial.



The current approach, which understandably is sub-optimal, is to point to translations of source documents in other languages. These should reference the version of the document collection from which they are made.



The document structure of the documentation pod files would need to be specified in such a way as to make translation synchronisation easier. This is of course possible. But the real question is whether the resources needed to accomplish the goal would not be better spent getting better underlying documentation.



Versioning the documentation collection. The documentation is in the form of pod files in a github repo, which is versioned. In principle, the documentation collection prior to the release date of the perl6 upgrade refers to the current version of Perl 6.



Changes made after the release may refer to both changes needed to reflect new and altered behaviours, and to clarifications of behaviours common to all releases. Correctly identifying what a change in the documentation refers to is not simple.



Both of these requests, whilst apparently reasonable, are bedevilled by details.



Search functionality is included in the proposal under 'look and feel'. Personally I absolutely agree with this request. But making changes to the look and feel are always difficult. Improving the existing functionality is a part of the current proposal. Some of the concrete changes needed have been mentioned in existing issues.



However, I found it difficult to find a concrete goal to cover this sort of intangible progress.



For the difficult requests, eg., multilingual coverage, I see the proposal includes them in so far as I would want to see the problem identified, the resources needed to accomplish some part of the request defined, and some debate amongst developers about the need for these requests.



I do not suggest it would be a good idea to debate concrete changes to the documentation system. The place for requesting desireable changes is in the issues section of the docs repo. Then the pros and cons can be discussed and a way forward agreed on. However, issues then have to be implemented. Getting that done is a part of the proposal as submitted.



Right, $500/day using the community money sounds unrealistic



Versioning of documentation would be something that could be done, and there's in fact a (very old) issue that requests just that. It's not theoretically impossible, and as a matter of fact Perl 6 might be better prepared for doing this thing that other languages, since you can add metadata at the block level.

But.

There are two problems.

First one, there are 400 document files right now, and we would need to add metadata to every block to every one of them. Even if we add metadata only to obviously versioned paragraphs, like the ones mentioned, it would be an impossible amount of (grunt) work. Which would be OK if it pays off.

But.

We'll need to write the tooling to support that. Although right now, with the introduction of Documentable, support for block versioning can be added, well, it needs to be done. Right now we're in the middle of actually implementing Documentable at all levels and redoing all the documentation, and there are dozens of issues related to that across 4 repositories that need to be done, with only Antonio, Richard, Tom and me helping whenever we can. So I don't think that's on the table for the time being, or the foreseeable future.

