improve docstring parsing

changed milestone to %Someday

changed title from imporve docstring parsing to improve docstring parsing

That looks really promising!

It would be indeed nice to have a clearer docstring composition and parsing implementation, that wraps all the re-spells included conveniantly.

I also remember having wondered about that there seems no such thing out there, together with @schaefed once.

The package suggested looks indeed promising, i tried it out, sadly it doesnt seem to be as plug and play as it looks first glance:

It seems to be a residual module from a certain project, since it only supports a limited subset of the numpy style section types (For example, it is not able to parse the common See Also section. It merges unknown sections into the returns section and messes up the docstrings when recomposing.)
One can conveniantly parse parameter descriptions and return value descriptions - but sadly, the class resulting from the parsing is imutable - so it is not straight forwardly (or at all) possible to remove or replace certain parameter/value entries. This means, re magic comes into play again, when constructing the new docstring.
there is only one combine-decorator packaged, that seems to aim at a very specific module/documentation construction, wich doesnt match ours. We would have to make some changes. Wich still wouldnt be enough, since the parser doesnt recognize all section types we use.

So i think we postpone / close this issue. I will fix the listing of the data/flags parameters as reported in #339 (closed) within the old framework. Maybe the issue can stay active, for collecting other suggestions for parser frameworks.

It seems to be a residual module from a certain project, since it only supports a limited subset of the numpy style section types (For example, it is not able to parse the common See Also section. It merges unknown sections into the returns section and messes up the docstrings when recomposing

as i digged into the package for the SaqcConfigurator, it is able to parse any kind of section, including but not limited to:

DEFAULT_SECTIONS = [
    ParamSection("Parameters", "param"),
    ParamSection("Params", "param"),
    ParamSection("Arguments", "param"),
    ParamSection("Args", "param"),
    ParamSection("Other Parameters", "other_param"),
    ParamSection("Other Params", "other_param"),
    ParamSection("Other Arguments", "other_param"),
    ParamSection("Other Args", "other_param"),
    ParamSection("Receives", "receives"),
    ParamSection("Receive", "receives"),
    RaisesSection("Raises", "raises"),
    RaisesSection("Raise", "raises"),
    RaisesSection("Warns", "warns"),
    RaisesSection("Warn", "warns"),
    ParamSection("Attributes", "attribute"),
    ParamSection("Attribute", "attribute"),
    ReturnsSection("Returns", "returns"),
    ReturnsSection("Return", "returns"),
    YieldsSection("Yields", "yields"),
    YieldsSection("Yield", "yields"),
    Section("Examples", "examples"),
    Section("Example", "examples"),
    Section("Warnings", "warnings"),
    Section("Warning", "warnings"),
    Section("See Also", "see_also"),
    Section("Related", "see_also"),
    Section("Notes", "notes"),
    Section("Note", "notes"),
    Section("References", "references"),
    Section("Reference", "references"),
    DeprecationSection("deprecated", "deprecation"),
]

the last commit was on 27th of April 2022, so its not abandoned also..

One can conveniantly parse parameter descriptions and return value descriptions - but sadly, the class resulting from the parsing is imutable - so it is not straight forwardly (or at all) possible to remove or replace certain parameter/value entries. This means, re magic comes into play again, when constructing the new docstring.

this is not true, one can remove or append params and/or change the description etc

eg.:

parsed = docstring_parser.parse(func.__doc__)

# remove 2nd param
parsed.params.remove(parsed.params[1])

# change description
parsed.params[0].description = "new text"

there is only one combine-decorator packaged, that seems to aim at a very specific module/documentation construction, wich doesnt match ours. We would have to make some changes. Wich still wouldnt be enough, since the parser doesnt recognize all section types we use.

From their description: support ReST, Google, Numpydoc-style and Epydoc docstrings. and because we use Numpydoc style it should work like a charm..i can tell, because i already use it in some minimal way in the SaQCConfigurator to parse saqc methods..

@luenensc maybe you refer to another package ? or it was still in development, once you tried it ?

anyways, to switch to the package, we will for sure need some more adjustment, so we wont do it in %2.1.0 😺

mentioned in merge request !471 (merged)

it should work like a charm

hm.. it actually doesnt. Maybe i did something wrong. But try to parse and recompose np.sum docstring like so:

>>> from docstring_parser import parse, compose
>>> import numpy as np
>>> print(compose(parse(np.sum.__doc__)))

It messes up headings containing whitesspaces and all the doctest Examples. Maybe you only tried it with Parameters section and returns section? Maybe the parser works - but at least not the reconstruction ...

this is not true, one can remove or append params and/or change the description etc

That would be cool. But somehow neither .remove nor .append worked when i tried it out (i also tried as suggested in your example.) Also, when i try to alter the list entries directly, i get cant change attribute. But description alternation works... Maybe the project is just not completed yet. It went public only in april.

it should work like a charm

yea it was a bit over enthusiastic ;)..

somehow neither .remove nor .append worked when i tried it out..

ups my bad.. one need to work on parsed.meta not parsed.params, i had this issue before myself 😅

But in general i admit its hard to work with the module, because nearly no documentation exist and sometimes a bit searching is needed.. in my experience the parsing worked quite well, i diddnt worked with the composing.. in the end i like this project and like to use with saqc, some day. maybe we could bring our efforts of docstring parsing into the project, instead of home-brew everything..

hehehe... i just found https://de.wikipedia.org/wiki/Not-invented-here-Syndrom

ah ok but thats good!

When the alternation of parameter entries via .meta works, a huge part of our parsing code can be replaced by using the package already. yes but its right now more a "someday" thingy. Maye when the next problem with current parsing comes up.

With respect to the Not-invented-Syndrome -> it says in the article:

"Als Gegenmaßnahme wurden bei der NASA spezielle Stellen geschaffen und mit Personen besetzt, die keiner Abteilung fest zugeordnet waren. Von Inhabern dieser Stellen war und ist heute noch die Aufgabe zu erfüllen, zwischen den Abteilungen zu pendeln und Informationen hin- und herzutragen, womit sich die jeweils ausführenden Personen zu keiner Zeit besonders beliebt machten." hehe: "hin und her pendel" aka: meetings

added PR label

added needs discussion label

closed

removed needs discussion label

improve docstring parsing

Designs

Child items ...

Activity