You are here: Foswiki>Frontend Web>AutoDoc (03 Jul 2020, TobiasHabermann)Edit Attach


What is autodoc?

autodoc is
  • a python module that can be imported in tools written in python to support generating a short summary of the tools information.
  • a tool that can collect information from other tools and generate a html document containing an overview.
The aim is to create an overview similar to The preliminary overview created by autodoc is ToolsOverview.

The autodoc tool

  • description: Generate tools overview from output of autodoc enabled tools
  • usage: autodoc [-h] [--tools_dir TOOLS_DIR] [--html HTML] [--dump DUMP] [--search SEARCH] [--black_list BLACK_LIST] [--verbose] {generate,dump,list}
  • author: T.H.
  • tags: doc,html,overview
  • documentation:
positional arguments:
                        generate - generates a html document from tool infos collected from 
                            tools in --tools_dir and stores it in --html.
                            autodoc will call each tool (actually each file) located in --tools_dir
                            with the --generate_doc_tagged argument to collect the tools info. 
                            Use --black_list for tools in --tools_dir that should be 
                            excluded (because they don't support autodoc, open a gui,
                            or should not be called for other reasons)
                        dump - dumps tool infos to the --dump file.
                            similar to generate, but instead of html, a file containing the tagged 
                            tools info will be written.
                        list - lists tool infos that have been dumped to the --dump file

optional arguments:
  -h, --help            show this help message and exit
  --tools_dir TOOLS_DIR
                        directory where tools are located
  --html HTML           file where the html document will be stored
  --dump DUMP           file where the tool info dump will be stored
  --search SEARCH       search string for the list option. Format is "item=str[,item=str,...]"
  --black_list BLACK_LIST
                        file with tools to be ignores, format is one filename per line
  --verbose             flag to enable verbose mode of generate and dump

How to use autodoc in tools?

The autodoc tool collects information from tools by calling them with a --generate_doc_tagged command line argument. autodoc comes with a python package (pylib/utility/autodoc) that supports implementing this argument. The easiest way is to use a parser from either optparse or argparse, but it can also be done without. In principle the autodoc tool can also collect information from non-python tools, provided that they implement a --generate_doc_tagged command line argument. The argument that has to be present is:
--generate_doc_tagged: Prints the tool info in tagged format (see below) on the screen. This is the option that is used by the autodoc tool to collect info from other tools

Two more arguments are added to tools:
--generate_doc_html: Prints the tool info as html. This is the same as the autodoc tool generates from the output of --generate_doc_tagged. You can use this to preview how your tools info will appear in the overview generated by autodoc. Also this option can come in handy when writing additional documentation for a tool. For example the listing for the autodoc tool above was generated in this way.
--generate_doc_pretty: The output of the other two options is not made to be readable. This generates human readable output and can be used for debugging.

Note that the tagged output contains all items in arbitrary order. The html output leaves out optional items that are empty and the plain string output also contains "hidden" items that are for internal use only and are not included in the html output (see below for an example).

How to use autodoc in python tools with argparse / optparse?

When the tool already uses a parser from the argparse or optpase package, enabling autodoc is straightforward. For example, the code the autodoc tool itself uses is:

    import argparse
    from pylib.utility import autodoc
    # .....   
    parser = argparse.ArgumentParser( .... )
    autodoc.add_toolsinfo(name = autodoc.FROM_PARSER,
                          topic = autodoc.TOPIC_DEV_REL_ROLL,
                          desc = autodoc.FROM_PARSER,
                          usage = autodoc.FROM_PARSER,
                          author = "T.H.",
                          tags = "doc,html,overview",
                          documentation = "",
                          parser = parser)

The full signature of the method is:

def add_toolsinfo(name,topic,desc,usage,author,tags="",version="",documentation="",env="",req="",parser = None,more = None)
  • The first 5 arguments are mandatory. Those are the items that are always listed in the generated overview (even when they are blank). For more details on the meaning of the items see below.
  • Some fields can be retrieved from the parser object. This can be done by passing the autodoc.FROM_PARSER flag. The flag can be used for the parameters name, desc, usage and version. Note that the result depends on how you set up the parser (eg the name known to the parser may contain an unwanted .py)
  • Predefined topics are
    • autodoc.TOPIC_EQUIP_ACCESS
    • autodoc.TOPIC_DEVICE_INFO
    • autodoc.TOPIC_DEV_REL_ROLL
    • autodoc.TOPIC_HELPERS
    • autodoc.TOPIC_OTHERS
  • It is possible to choose a custom topic (a string). Tools with a custom topic will be listed in the section "Others" in the overview.
  • More items can be added via the more argument. It expects a dictionary that maps strings to strings. For example more = {"note" : "this is a note"}
  • Calling the function with a parser will add the command line arguments to the parser. Note that those arguments will not be listed in the help generated by the parser.

How to use autodoc in python tools that do not use a argparse / optparse parser?

Altough not recommended, autodoc can also be used in tools that do not use an argparse / optparse parser. An example is:

import sys
from pylib.utility import autodoc

if __name__ == '__main__':
    autodoc.add_toolsinfo(name = "no_parser_example",
                          topic = "example",
                          desc = "beschreibung",
                          usage = "no_parser_example",
                          author = "T.H.",
                          more = {"note" : "this tool does not use argparse/optparse"}
    if len(sys.argv)>1:

This will accept all three arguments: --generate_doc_tagged, --generate_doc_html and --generate_doc_pretty.

How to use autodoc in non-python tools?

For non-python tools there are bash scripts that support generating the output needed by the autodoc tool. Example:

. ../scripts/autodoc/topics

TOOLINFO_MORE=("more:bla bla"  "more2:bla bla bla")

. ../scripts/autodoc/generate

#autodoc_print tagged
#autodoc_print html
#autodoc_print pretty

#... your actual script ...
  • /autodoc/topics defines variables for the available topics. The names of the variables are listed in the table below. Plain strings can be used and sourcing the topics script is not absolutely necessary.
  • /autodoc/generate parses command line arguments. If one of the autodoc command line arguments is present (--generate_doc_tagged, --generate_doc_html or --generate_doc_pretty), then the corresponding output will be generatated and exit 0 will be called. If none of the arguments is present, the script does nothing and your actual script is executed.
  • If for some reason you want to disable parsing of command line arguments in generate, then you can define the AUTODOC_NOPARSE variable with a non-empty value. In that case generate will only define the functions that generate the output, but does not automatically parse arguments and call the functions. You can then call the functions yourself (autodoc_print tagged/html/pretty).
  • TOOLINFO_MORE can be used for custom items that will appear in the tools overview. Its value is an array of strings with format "custom_item_name : value". Custom item names may not contain whitespace or ":".

Overview: Topics and Items

The generated overview is inspired by and uses the same topics. The following table lists the topics and the corresponding constants from the autodoc module.

Equipment Access autodoc.TOPIC_EQUIP_ACCESS
Device Information autodoc.TOPIC_DEVICE_INFO
Maintenance autodoc.TOPIC_MAINTENANCE
Development, Release, Rollout autodoc.TOPIC_DEV_REL_ROLL
Helpers autodoc.TOPIC_HELPERS
Others autodoc.TOPIC_OTHERS
Tools should fit in one of the first 5 categories if possible. autodoc also allows custom topics for tools. Tools with custom topics will be listed under Others in the overview.

The predefined items are listed in the following table.
item from Parser description
mandatory name x Name of the tool
topic Choose a predefined topic (see above). Custom topics are allowed.
description x Short description. This should fit on a single line.
usage x Similar to what you get from --help (only list of arguments, no detailed description)
author Author of the tool
optional tags Tags help for "findability". For example a tool that uses a GUI to display timing information could be tagged as "graphics,interactive,timing"
version x The version of the tool
documentation The overview itself should be terse and short. More detailed documentation can be linked here
environment Some tools can/should only be using in a certain environment. The item can hold arbitrary string, but one should use dev/int/pro.
requires Some tools have preconditions to be used. For example snoopy-rec requires the WRSnoop FESA Class to be running. Listing requirements here can help users in case of failure/errors
more The predefined items should be sufficient for the overview. If a tool author still feels the need to add a custom item, they can do so. Item names may not contain whitespaces.
hidden autodocversion Version of autodoc that was used to generate the output.
  • Mandatory items are always included in the overview even when they are empty (they should not be empty).
  • Optional items are only included in the overview when they are not empty.
  • Hidden items are for autodoc internal use only. Currently there is only one item: autodocversion.
  • The "from Parser" column indicates whether the item supports the autodoc.FROM_PARSER tag.

Example Output

Below is an example output of the three command line arguments --generate_doc_tagged, --generate_doc_html and --generate_doc_pretty. The autodoc tool itself also implements the three command line arguments.

[thaberm@asl744]$ autodoc --generate_doc_tagged
<toolinfo><description>Generate tools overview from output of autodoc enabled tools</description><tags>doc,html,overview</tags><topic>Development, Release, Rollout</topic><autodocversion>1.0</autodocversion><name>autodoc</name><author>T.H.</author><documentation></documentation><environment></environment><version></version><usage>autodoc [-h] [--tools_dir TOOLS_DIR] [--html HTML] [--dump DUMP]
               [--search SEARCH] [--black_list BLACK_LIST] [--verbose]
  • The tagged output always contains all items (also optional ones when they are empty). It is not sorted.
  • Each item is enclosed in tags with the name of the item and the whole block is enclosed in <toolinfo></toolinfo>
[thaberm@asl744]$ autodoc --generate_doc_html
<h3>autodoc</h3><ul><li><b>description: </b>Generate tools overview from output of autodoc enabled tools</li><li><b>usage: </b>autodoc [-h] [--tools_dir TOOLS_DIR] [--html HTML] [--dump DUMP]
               [--search SEARCH] [--black_list BLACK_LIST] [--verbose]
               {generate,dump,list}</li><li><b>author: </b>T.H.</li><li><b>tags: </b>doc,html,overview</li><li><b>documentation: </b></li></ul>
  • The html output is the same as what the autodoc tool will generate from the tagged output.
  • Optional items are only included when they are not empty.
[thaberm@asl744]$ autodoc --generate_doc_pretty
   topic: Development, Release, Rollout
   description: Generate tools overview from output of autodoc enabled tools
   usage: autodoc [-h] [--tools_dir TOOLS_DIR] [--html HTML] [--dump DUMP]
                     [--search SEARCH] [--black_list BLACK_LIST] [--verbose]
   author: T.H.
   tags: doc,html,overview
   autodocversion: 1.0
  • The string output also includes the autodocversion, which is only for internal use and is not included in the others or in the tools overview.

TODO / Whishlist

  • Output from --generate_doc_html copied directly to a does not look nice.
  • Currently the html and the dump are independent. This is good because one can create dump and html from a local workspace, but it is bad, because what you get from list and what is in the wiki should be the same. Maybe change it to create the html from the dump.
  • html links in the overview are not working when copied into the wiki
  • The list action has only a preliminary implmentation. Currently --search allow to search in all items, but only a limited fixed set of items is included in the printed table. Items to be displayed should be selectable by the user (this requires to truncate content when it is too long)
  • Currently the Overview is just copied here to the wiki. On the long run, the html overview should be create automatically (eg jenkins once per week) and only linked here
  • Tools Overview should not be in the public wiki
  • in the wiki a outlook can be generated via TOC{title="Contents"}
  • Also more detailed information could be included with autodoc. Eg example of how to call a tool. However, this would probably be too much for the overview. Maybe it could be included in an additional section of the overview ("Examples") and the short summary only links to extended examples. Ideas & suggestions are welcome.
  • Defaults are not obvious. autodoc generate should report what dirs/files are used/written. Related bug: when no --tools_dir argument is specified, then the generated html say "... generated from ./"
-- TobiasHabermann - 20 May 2020
Topic revision: r18 - 03 Jul 2020, TobiasHabermann
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback