AutoDoc

Tools Overview How To

How to include a tool in the tools overview is describe here: https://www-acc.gsi.de/wiki/Frontend/ToolsOverviewHowTo. Before diving into details here, make sure to read that.

What is autodoc?

autodoc is

• a python module that can be imported in tools written in python to support generating output needed for the tools overview.
• a tool that can collect information from other tools and generate a html document containing an overview.

The aim is to create an overview similar to https://www-acc.gsi.de/wiki/Frontend/BelTools. The preliminary overview created by autodoc is ToolsOverview.

The autodoc tool

• description: Generate tools overview from output of autodoc enabled tools
• usage: autodoc [-h] [--tools TOOLS [TOOLS ...]] [--tagged_files TAGGED_FILES [TAGGED_FILES ...]] [--black_list BLACK_LIST] [--format FORMAT [FORMAT ...]] [--outfile OUTFILE] [--verbose] {generate}
• author: T.H.
• tags: doc,html,overview
• version: 2.0
• documentation: https://www-acc.gsi.de/wiki/Frontend/AutoDoc
• help:

  usage: autodoc [-h] [--tools TOOLS [TOOLS ...]]
                 [--tagged_files TAGGED_FILES [TAGGED_FILES ...]]
                 [--black_list BLACK_LIST] [--format FORMAT [FORMAT ...]]
                 [--outfile OUTFILE] [--verbose]
                 {generate}
  
  Generate tools overview from output of autodoc enabled tools
  
  positional arguments:
    {generate}            
                          generate - generates tools overview from tools (--tools & --tagged_files). 
                              The output is stored in files named --outfile with extensions depending  
                              on the chosen output format (--format). 
                              No information will be collected from tools listed in the --black_list file.
                              (more options will be added later, currently this argument is redundant)
  
  optional arguments:
    -h, --help            show this help message and exit
    --tools TOOLS [TOOLS ...]
                          list of directories or paths to tools
    --tagged_files TAGGED_FILES [TAGGED_FILES ...]
                          list of files containing tools information in tagged format
    --black_list BLACK_LIST
                          black list file. Contains one tool per line that should be skipped
    --format FORMAT [FORMAT ...]
                          List of output formats to be generated. 
                          Each output will be stored in a seperate file with the respective extension. 
                          Possible formats are pretty, wiki, html, tagged.
                          Only .tagged files can be read later by autodoc.
                          
                          "Expert":
                          The output format for the individual tools and the overview can be selected separately.
                          pretty, wiki, html and tagged are shortcuts for pp, ww, hh, tt.
                          For example pw can be useful when generating wiki text for a single tool.
                          The first character selects the overview format:
                            p/t - no formatting. only concatenate formatted tools
                            w/h - wiki/html overview including a table of contents
                          The second character selects format of individual tools:
                            p - pretty (plain string for debugging, not actually pretty)
                            t - tagged
                            w - wiki
                            h - html
    --outfile OUTFILE
    --verbose             flag to enable verbose mode of generate and dump
  
  

This last paragraph was generated via autodoc generate --tools /common/usr/cscofe/bin/autodoc --format pw --outfile auto --verbose

How to use autodoc in tools?

The autodoc tool collects information from tools by calling them with a --generate_doc_tagged command line argument. autodoc comes with a python package (pylib/utility/autodoc) that supports implementing this argument. The easiest way is to use a parser from either optparse or argparse, but it can also be done without. In principle the autodoc tool can also collect information from non-python tools, provided that they implement a --generate_doc_tagged command line argument. The argument that has to be present is:

Two more arguments are added to tools:

Note that the tagged output contains all items in arbitrary order. The html output leaves out optional items that are empty and the plain string output also contains "hidden" items that are for internal use only and are not included in the html output (see below for an example).

How to use autodoc in python tools with argparse / optparse?

When the tool already uses a parser from the argparse or optpase package, enabling autodoc is straightforward. For example, the code the autodoc tool itself uses is:

    import argparse
    from pylib.utility import autodoc
    # .....   
    parser = argparse.ArgumentParser( .... )
    autodoc.add_toolsinfo(name = autodoc.FROM_PARSER,
                          topic = autodoc.TOPIC_DEV_REL_ROLL,
                          desc = autodoc.FROM_PARSER,
                          usage = autodoc.FROM_PARSER,
                          author = "T.H.",
                          tags = "doc,html,overview",
                          documentation = "https://www-acc.gsi.de/wiki/Frontend/AutoDoc",
                          parser = parser)

The full signature of the method is:

def add_toolsinfo(name,topic,desc,usage,author,tags="",version="",documentation="",env="",req="",parser = None,more = None)

• The first 5 arguments are mandatory. Those are the items that are always listed in the generated overview (even when they are blank). For more details on the meaning of the items see below.
• Some fields can be retrieved from the parser object. This can be done by passing the autodoc.FROM_PARSER flag. The flag can be used for the parameters name, desc, usage and version. Note that the result depends on how you set up the parser (eg the name known to the parser may contain an unwanted .py)
• Predefined topics are (see here)
  
  • autodoc.TOPIC_EQUIP_ACCESS
  • autodoc.TOPIC_DEVICE_INFO
  • autodoc.TOPIC_MAINTENANCE
  • autodoc.TOPIC_DEV_REL_ROLL
  • autodoc.TOPIC_HELPERS
  • autodoc.TOPIC_OTHERS
• It is possible to choose a custom topic (a string). Tools with a custom topic will be listed in the section "Others" in the overview.
• More items can be added via the more argument. It expects a dictionary that maps strings to strings. For example more = {"note" : "this is a note"}
• Calling the function with a parser will add the command line arguments to the parser. Note that those arguments will not be listed in the help generated by the parser.

How to use autodoc in python tools that do not use a argparse / optparse parser?

Altough not recommended, autodoc can also be used in tools that do not use an argparse / optparse parser. An example is:

import sys
from pylib.utility import autodoc

if __name__ == '__main__':
    autodoc.add_toolsinfo(name = "no_parser_example",
                          topic = "example",
                          desc = "beschreibung",
                          usage = "no_parser_example",
                          author = "T.H.",
                          more = {"note" : "this tool does not use argparse/optparse"}
                        )
    autodoc.print_and_exit_if_autodoc_arg(sys.argv)

This will accept all three arguments: --generate_doc_tagged, --generate_doc_html and --generate_doc_pretty as the first argument to the call. If print_and_exit_if_autodoc_arg finds one of the autodoc arguments in sys.argv[1] it produces the output and calls sys.exit().

How to use autodoc in non-python tools?

A tool must support a --generate_doc_tagged command line argument and produce the necessary output. See here for what output is expected and below for another example.

Example Output

Below is an example output of the three command line arguments --generate_doc_tagged, --generate_doc_html and --generate_doc_pretty. Only --generate_doc_tagged needs to be supported by a tool. The others are for debugging and generating output to be used elsewhere and a custom implementation is only recommended to support them also. The examples below are from the autodoc tool itself.

[thaberm@asl744 sandbox]$ autodoc --generate_doc_tagged
<toolinfo>
<name>autodoc</name>
<topic>Development, Release, Rollout</topic>
<description>Generate tools overview from output of autodoc enabled tools</description>
<usage>autodoc [-h] [--tools TOOLS [TOOLS ...]]
               [--tagged_files TAGGED_FILES [TAGGED_FILES ...]]
               [--black_list BLACK_LIST] [--format FORMAT [FORMAT ...]]
               [--outfile OUTFILE] [--verbose]
               {generate}
</usage>
<author>T.H.</author>
<tags>doc,html,overview</tags>
<version>2.0</version>
<documentation>https://www-acc.gsi.de/wiki/Frontend/AutoDoc</documentation>
<environment></environment>
<requires></requires>
<help></help>
<autodocversion></autodocversion>
</toolinfo>

• The tagged output always contains all items (also optional ones when they are empty). It is not sorted.
• Each item is enclosed in tags with the name of the item and the whole block is enclosed in <toolinfo></toolinfo>

[thaberm@asl744 sandbox]$ autodoc --generate_doc_html
<h3>autodoc</h3>
<ul>
<li><b>description: </b>Generate tools overview from output of autodoc enabled tools</li>
<li><b>usage: </b>autodoc [-h] [--tools TOOLS [TOOLS ...]]
               [--tagged_files TAGGED_FILES [TAGGED_FILES ...]]
               [--black_list BLACK_LIST] [--format FORMAT [FORMAT ...]]
               [--outfile OUTFILE] [--verbose]
               {generate}
</li>
<li><b>author: </b>T.H.</li>
<li><b>tags: </b>doc,html,overview</li>
<li><b>version: </b>2.0</li>
<li><b>documentation: </b>https://www-acc.gsi.de/wiki/Frontend/AutoDoc</li>
</ul>

• The html output is the same as what the autodoc tool will generate from the tagged output.
• Optional items are only included when they are not empty.

[thaberm@asl744 sandbox]$ autodoc --generate_doc_pretty
autodoc:
   topic: Development, Release, Rollout
   description: Generate tools overview from output of autodoc enabled tools
   usage: autodoc [-h] [--tools TOOLS [TOOLS ...]]
               [--tagged_files TAGGED_FILES [TAGGED_FILES ...]]
               [--black_list BLACK_LIST] [--format FORMAT [FORMAT ...]]
               [--outfile OUTFILE] [--verbose]
               {generate}

   author: T.H.
   tags: doc,html,overview
   version: 2.0
   documentation: https://www-acc.gsi.de/wiki/Frontend/AutoDoc
   environment: 
   requires: 
   help: 
   autodocversion: 

• The string output also includes the autodocversion, which is only for internal use and is not included in the others or in the tools overview.

TODO / Whishlist / Issues

Feel free to add to this list. Eventually points will move down to the next section.

• the overview @ wiki should be updated automatically once per week.
• how to include third-party tools?
• overview should in the public/private wiki?
• generated overview should contain some meta-information. For example what sources were used to collect the overview. Currently the header says "This content was generated by autodoc on 2020-09-02 12:26:45.496479 from all tools in [!nothing here!]".
• the list option should be reactivated. The preliminary implementation is removed for now. Should be more similar to lstools.
• the --generate_doc_html and --generate_doc_pretty arguments turned out to be not that useful. Only tools that use the autodoc python package have them, but whether a tool uses python is an implementation detail. Since now the autodoc tool can be used to generate the exact same output, those two arguments don't have to be supported anymore.
• tools printing their own name is redundant and error prone (example: archive_rader vs archive_reader). Instead of using the name from the tools output the autodoc tool could use the name used to invoke the tool. On the other hand dex makes use of this "feature" and does output a different name (call: dex, name in output: (p|d|i)dex. Perhaps the best is to issue a warning when the names do not match.
• bugs:
  • If an item in the output of generate_doc_tagged contains <foo>, the parser gets confused and cannot interpret the output correctly. (example: fesa-instance, < and > are replaced when writing but not correctly replaced when reading)
  • the autodoc-version item is no longer correctly populated. (example: autodoc example output. Not a problem because currently there is no need to use that information, there is also no need to remove the item)

What's new

• 02.09.2020:
  • Output format: Output for the overview is no longer limited to html. It is possible to choose between wiki-formatting, html and plain text. Moreoever formatting of individual tools and how this is combined to an overview can be selected seperately. Actually the output format is more flexible than needed, thats why there are shortcuts for typical use-cases (wiki, html, pretty). Other output formats could be realized with little effort if needed.
  • New wiki output: Includes table of contents, links are now links and nicer layout (because no html)
  • Usage of the autodoc tool is simpler and more flexible. Information from single tools, multiple paths and files with already tagged tool information can be collected in one call. No more hidden default arguments with surprising effects.
  • The overview now also contains the --help= of tools.
  • Support for optparse is still in, but about to be deprecated. There are currently no FE-Tools using the python module optparse.
  • The list option is temporarily removed.
  • The autodoc bashscripts (to be included in other bash-scripts to support generating the output) are deprecated
  • autodoc @ pylib is now a package not a single module (nothing changes for imports)

-- TobiasHabermann - 20 May 2020