Sphinx documentation and links to Markdown

Question

I'm trying to use Sphinx to build some documentation from Markdown source. My conf.py is as follows...

conf.py

from recommonmark.parser import CommonMarkParser
project = 'DS'
copyright = '2018, DS'
author = 'DS, Work'

version = ''
release = ''

extensions = []

templates_path = ['_templates']

source_suffix = ['.rst', '.md']

master_doc = 'index'

language = None
exclude_patterns = []
pygments_style = 'sphinx'
html_theme = 'classic'
html_static_path = ['_static']
source_parsers = {
   '.md': CommonMarkParser,
}
htmlhelp_basename = 'DSDocumentationdoc'
latex_elements = {
}
latex_documents = [
    (master_doc, 'DSDocumentation.tex', 'DS Documentation',
     'DS, Work', 'manual'),
]
man_pages = [
    (master_doc, 'dsdocumentation', 'DS Documentation',
     [author], 1)
]
texinfo_documents = [
    (master_doc, 'DSDocumentation', 'DS Documentation',
     author, 'DSDocumentation', 'One line description ofproject.',
 'Miscellaneous'),
]

index.rst

Welcome to DS Documentation!
======================================

The following documentation is produced and maintained by the Data Science team.

Contents:

.. toctree::
   :maxdepth: 2
   :glob:

   README.md
   documentation.md
   getting_started/*
   how-tos/*
   statistics_data_visualisation.md

The documents build and html output is generated, however README.md has links to other markdown documents in the two sub-directories such as the following...

... [this document](./getting_started/setting_your_machine_up.md)...

...which in the translated README.html document the target has not been converted to the translated HTML target as its been recognised as reference external...

...<a class="reference external" href="./getting_started/setting_your_machine_up.md">this document</a>...

...I was half-expecting/hoping it would output as reference internal and convert the file extension approrpiately...

...<a class="reference internal" href="./getting_started/setting_your_machine_up.html">this document</a>...

...so that links worked in the same vein as the Table of Contents does in the sidebar.

Any suggestions as to whether this can be achieved would be appreciated.

Cheers.

EDIT

Trying out the solution suggested by @waylan I have added the following to by conf.py to enable_auto_doc_ref...

def setup(app):
    app.add_config_value('recommonmark_config', {
        'enable_auto_doc_ref': True,
    }, True)
    app.add_transform(AutoStructify)

...and on running make html I get the following error.....

❱ cat /tmp/sphinx-err-57rejer3.log 
# Sphinx version: 1.8.0
# Python version: 3.6.6 (CPython)
# Docutils version: 0.14 
# Jinja2 version: 2.10
# Last messages:
#   building [mo]: targets for 0 po files that are out of date
#   
#   building [html]: targets for 16 source files that are out of date
#   
#   updating environment:
#   
#   16 added, 0 changed, 0 removed
#   
#   reading sources... [  6%] README
#   
# Loaded extensions:
#   sphinx.ext.mathjax (1.8.0) from /home/[email protected]/.local/lib/python3.6/site-packages/sphinx/ext/math
jax.py                                                                                                                                
#   alabaster (0.7.11) from /home/[email protected]/.local/lib/python3.6/site-packages/alabaster/__init__.py
Traceback (most recent call last):
  File "/home/[email protected]/.local/lib/python3.6/site-packages/sphinx/cmd/build.py", line 304, in build_ma
in                                                                                                                                    
    app.build(args.force_all, filenames)
  File "/home/[email protected]/.local/lib/python3.6/site-packages/sphinx/application.py", line 341, in build
    self.builder.build_update()
  File "/home/[email protected]/.local/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 347, in 
build_update                                                                                                                          
    len(to_build))
  File "/home/[email protected]/.local/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 360, in 
build                                                                                                                                 
    updated_docnames = set(self.read())
  File "/home/[email protected]/.local/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 468, in 
read                                                                                                                                  
    self._read_serial(docnames)
  File "/home/[email protected]/.local/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 490, in 
_read_serial                                                                                                                          
    self.read_doc(docname)
  File "/home/[email protected]/.local/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 534, in 
read_doc                                                                                                                              
    doctree = read_doc(self.app, self.env, self.env.doc2path(docname))
  File "/home/[email protected]/.local/lib/python3.6/site-packages/sphinx/io.py", line 318, in read_doc
    pub.publish()
  File "/home/[email protected]/.local/lib/python3.6/site-packages/docutils/core.py", line 218, in publish
    self.apply_transforms()
  File "/home/[email protected]/.local/lib/python3.6/site-packages/docutils/core.py", line 199, in apply_trans
forms                                                                                                                                 
    self.document.transformer.apply_transforms()
  File "/home/[email protected]/.local/lib/python3.6/site-packages/sphinx/transforms/__init__.py", line 90, in
 apply_transforms                                                                                                                     
    Transformer.apply_transforms(self)
  File "/home/[email protected]/.local/lib/python3.6/site-packages/docutils/transforms/__init__.py", line 171,
 in apply_transforms                                                                                                                  
    transform.apply(**kwargs)
  File "/home/[email protected]/.local/lib/python3.6/site-packages/recommonmark/transform.py", line 325, in ap
ply                                                                                                                                   
    self.traverse(self.document)
  File "/home/[email protected]/.local/lib/python3.6/site-packages/recommonmark/transform.py", line 297, in tr
averse                                                                                                                                
    self.traverse(child)
  File "/home/[email protected]/.local/lib/python3.6/site-packages/recommonmark/transform.py", line 297, in tr
averse                                                                                                                                
    self.traverse(child)
  File "/home/[email protected]/.local/lib/python3.6/site-packages/recommonmark/transform.py", line 297, in tr
averse                                                                                                                                
    self.traverse(child)
  File "/home/[email protected]/.local/lib/python3.6/site-packages/recommonmark/transform.py", line 287, in tr
averse                                                                                                                                
    newnode = self.find_replace(c)
  File "/home/[email protected]/.local/lib/python3.6/site-packages/recommonmark/transform.py", line 267, in fi
nd_replace                                                                                                                            
    newnode = self.auto_doc_ref(node)
  File "/home/[email protected]/.local/lib/python3.6/site-packages/recommonmark/transform.py", line 175, in au
to_doc_ref                                                                                                                            
    return self.state_machine.run_role('doc', content=content)
  File "/home/[email protected]/.local/lib/python3.6/site-packages/recommonmark/states.py", line 134, in run_r
ole                                                                                                                                   
    content=content)
TypeError: 'NoneType' object is not callable

I've looked through the last two calls and I think this might be down to content not being set, which may be something to do with my index.rst but I'm really out of my depth here.

Waylan · Answer 1 · 2018-09-21T17:42:03.443

The recommonmark documentation suggests enabling AutoStructify by adding the following to your config.py file:

from recommonmark.transform import AutoStructify

github_doc_root = 'https://github.com/rtfd/recommonmark/tree/master/doc/'
def setup(app):
    app.add_config_value('recommonmark_config', {
            'url_resolver': lambda url: github_doc_root + url,
            'auto_toc_tree_section': 'Contents',
            }, True)
    app.add_transform(AutoStructify)

This will give you the following features:

enable_auto_toc_tree: whether enable Auto Toc Tree feature.

auto_toc_tree_section: when enabled, Auto Toc Tree will only be enabled on section that matches the title.

enable_auto_doc_ref: whether enable Auto Doc Ref feature.

enable_math: whether enable Math Formula

enable_inline_math: whether enable Inline Math

enable_eval_rst: whether Embed reStructuredText is enabled.

url_resolver: a function that maps a existing relative position in the document to a http link

Of note is the Auto Doc Ref feature:

It is common to refer to another document page in one document. We usually use reference to do that. AutoStructify will translate these reference block into a structured document reference. For example
[API Reference](api_ref.md)
will be translated to the AST of following reStructuredText code
:doc:`API Reference </api_ref>`
And it will be rendered as API Reference

Why is this necessary? Because, unlike Rst, Markdown does not have any knowledge of anything outside of the given document and has no support for Rst style directives. Therefore, there is no mechanism to transform a URL.

Instead, AutoStructify waits until after the recommonmark bridge converts the Markdown to Sphinx's underlying document structure (docutils document object), then it runs a series of transformers on it to provide limited Rst like functionality. Even with AutoStructify, you will never get full feature support when using Markdown. That would require Markdown to have native support for directives, which is not likely to ever happen.

@slackline I added an explanation of why AutoStructify is necessary with an overview of how it works. — Waylan, Sep 21 '18 at 13:11
Thats for taking the time to explain all that @Waylan very much appreciated (I've only been using Python for 4.5 months after starting a new job). On trying your suggestion out I'm hitting an error which I've detailed in my above original post. — slackline, Sep 21 '18 at 15:36
I just edited my answer to include `from recommonmark.transform import AutoStructify`. I expect that will resolve your problem. — Waylan, Sep 21 '18 at 17:44
I'd read that on the [Autostructify section](https://recommonmark.readthedocs.io/en/latest/index.html) of recommonmark that `AutoStructify` needed explicit importing so had already included that in the header of `conf.py`, apologies for not including it in the above edit. — slackline, Sep 22 '18 at 07:29
I've been looking at this again and found on recommonmark that `enable_auto_doc_ref` is now deprecated (see [issue #85](https://github.com/rtfd/recommonmark/issues/85), also marked as such on the GitHub repos [README](https://github.com/rtfd/recommonmark)). It suggests that "*links in markdown are added as :any: role by default, and resolved by sphinx.*", so I'm going to look into that. Noting here in case anyone else encounters this problem. — slackline, Sep 25 '18 at 10:31
Interesting. That deprecation appears to have happened in [#63](https://github.com/rtfd/recommonmark/pull/63), where it is suggested that the feature was broken anyway. Apparently they are now letting Sphinx resolve the links natively which removes the need for the add-on. — Waylan, Sep 25 '18 at 14:37

Sphinx documentation and links to Markdown

1 Answers1

Linked