Abandon or Modify ElementTree?

The short version:
As part of version 3.0 (see #391) should Python-Markdown perhaps abandon ElementTree for a different document object like Docutils' node tree or use a modified ElementTree for internally representing the Parsed HTML document?
Any and all feedback is welcome.
The long version:
Starting in Python-Markdown version 2.0, internally  parsed documents have been represented as ElementTree objects. While this mostly works, there are a few irritations. ElementTree (hereinafter ET) was designed for XML, not HTML and therefore a few of its design choices are less than ideal when working with HTML.
For example, by design, XML does not generally have text and child nodes interspersed like HTML does. While ET provides text and tail attributes on each element, it is not as easy to work with as it would be if the text was contained in child "TextNodes" (much like JavaScript's DOM). Additionally, ET nodes have no knowledge of their parent(s), which can be a problem in certain HTML specific situations (some elements cannot contain other elements as children or grandchildren or great-grandchildren...).
I see two possible workarounds to this: Modify ET or use a different type of object.
Modifying ElementTree
We already have a modified serializer which gives us better HTML output (it is actually a modified HTML serializer from ET) and we already import ET and document that all extensions should import ET from Markdown. Therefore, if we were to change anything (via subclasses, etc) those changes would propagate throughout all extensions without too much change.
In fact, some time ago, I played around with the idea of making ET nodes aware of their parents. While it worked, I quickly abandoned it as I realized that it would not work for cElementTree. However, on further consideration, we don't really need cElementTree (most of the benefits are in a faster XML parser which we don't use).
Interestingly, in Python 3.3 cElementTree is deprecated. What actually happens is that ET defines the Python implementation and then at the bottom of the module, it tries to import the C implementation, which upon success, overrides the Python objects of the same name. What is interesting about this is that the Python implementation of the Element class (ET's node object) is preserved as _Element_Py for external code which needs access to it (as explained in the comments).
I envision a modified ET lib to basically subclass the Python Element object to enforce knowledge of parents for all nodes. Then a TextNode would be created which works essentially like Comments work now:
def TextElement(text=None):
    element = Element(TextElement)
    element.text = text
    return element

The serializer would then be updated to properly output TextElements. In fact, at some point, the serializer might even be able to loose knowledge of the text and tail attributes on regular nodes. However, that last bit could wait for all extensions to adopt the new stuff.
In addition to TextElement we could also have RawTextElement and AtomicTextElement. Both would be ignored by the parser (no additional parsing would take place). However, a RawTextElement would be given special treatment by the serializer in that no escaping would take place (raw HTML could be stored inline in the document rather than in a separate store with placeholders in the document), whereas an AtomicTextElement would be serialized like a regular TextElement.
The advantage of an AtomicTextElement (over the existing AtomicString) is that a single node could have multiple child text nodes. Today, each node only gets one text attribute. Therefore, when a AtomicString is concatenated with an existing text string, we lose the 'atomic' quality of the sub-string. However, with this change each sub-string can reside in its own separate text node and maintain the 'atomic' quality when necessary.
Using Docutils
Rather that creating our own one-off hacked version of ET, we could instead use an already existing library which gives us all of the same features (and more). Today, the only widely supported and stable library I'm aware of is Docutils' Document Tree. While the Document Tree is described as an XML representation of a document, Docutils provides a Python API to work with the Document Tree which is very similar to the modified ET API I described above (known parents, TextElement, FixedTextElement...). Unfortunately that API is not documented. Although, the the source code is easy enough to follow.
Until recently, I was of the assumption that to implement something that used Docutils, one would need to define a bunch of directives (etc) which more-or-less modify the ReST parser. However, take a look at the Overview of the Docutils Architecture. A parser simply needs to create a node tree. In fact, the base Parser class is only a few simple methods. The entire directives thing is in a separate directory under the ReST Parser only. Theoretically, one could subclass the base Parser class, and build a node tree using whatever parsing method desired and Docutils wouldn't care.
For that matter, Python-Markdown would not have to replicate Docutils "Parser" API. We could just use the node tree internally. As a plus, this would give us access to all of the built-in and third party Docutils writers (serializers). In other words, we would get all of Docutils output formats for free.
Additionally, Docutils' node tree also provides for various meta-data to be stored within the node tree. For example, each node can contain the line and column at which its contents were found in the original source document. This provides an easy way for a spellchecker to run against the parser and report misspelled words in the document without first converting it to HTML, among other uses which do not require serialized output.
No, this would not make Python-Markdown suddenly able to be supported by Sphinx. Sphinx is mostly a collection of custom directives built on top of the ReST parser. ReST directives do not make sense in Markdown. However, we could convert Markdown to ReST as many other third party parsers convert various formats to ReST via a ReST writer. There is also at least one third party writer which outputs Markdown from a node tree. By adopting Docutils node tree, Python-Markdown could become part of an ecosystem  for converting between all sorts of various document formats (an expandable competitor to Pandoc?).
The downsides to using Docutils are that we are then relying on a third party library (up till now, Python-Markdown has not) and all extensions would absolutely be forced to change to support the new version. It is also possible that we wouldn't be able to use the available HTML writer as the default because of some inherent differences with Markdown and ReST (ReST is much more verbose and we might need to hack the node tree or the writer to get the writer to output correct HTML from a Markdown perspective -- I have not investigated this).
As it stands now, there are various small changes required of extensions between version 2 and 3, but I expect that most extensions would be able to support both without much effort. If we went with Docutils, that would no longer be the case.
Or, maybe this whole thing is a bad idea and we should just continue to use ET as-is.
Any and all feedback is welcome.

No, this would not make Python-Markdown suddenly able to be supported by Sphinx. Sphinx is mostly a collection of custom directives built on top of the ReST parser.

Why not? The main feature of Sphinx is not directives, but beautiful themeable output (HTML, LaTeX, etc). I think that if we start using Docutils document tree, it will be possible to add Markdown support to Sphinx.

By the way, here is an attempt by someone else to write a Markdown parser for Docutils.

By the way, here is an attempt by someone else to write a Markdown parser for Docutils.

Yes, I was aware of that. However, it does not offer the kind of extension API we do and it is based on a PEG Parser [^1].  With our extension API, suppose for Docutils could really be something.
Regarding Sphinx, if it turns out to work, great, but that is not at all a goal of mine at this time.
[^1]:  While PEG parsers are great, there are a few nuances of the Markdown Syntax that they don't handle well and I don't consider them a good fit. Note that the most well know PEG parser for Markdown was written my John McFarlane (which all others seem to be based on) and he has since written other Markdown parsers, including ComonMark (which he also wrote the spec for) and he has chosen to not use a PEG parser for any of them.

I guess that is the real question: does the extra Docutil possibilities sway you enough to use it?  It sounds like you will have to do some customizing whether you use Docutils or ET, but you would have more control over say ET.  But I suspect most users would say go Docutils.  If you tell them you have an option that gives more potential features and one that does not, I suspect most will pick the former :smile:.

I did some work mapping the Markdown Elements (as they are output in HTML) to Docutils Node Elements. I have made no effort to map anything used by extensions. This is solely the base syntax. This is what I have come up with before doing any testing:

MD Element
Docutils Element

paragraph

nested section with child title

block_quote

enumerated_list (interesting attrs: enumtype, start)

bullet_list

list_item

literal_block

transition

line_block with children line (
 is implied between line elements)

            |

           | reference
<emphasis>    | emphasis
      | strong
        | literal
         | image
|
References:     | target
Raw HTML:       | raw
A few things to note:
Headers (
) are weird. In Docutils, you have sections and titles. The first child of a section is the title. Then all non-header elements following the header are children of that section. If a lower numbered header occurs, then is also is a child of the first section and any elements after is are children of that section. You only break out of a section when you have a higher level header. This does not translate well to standard Markdown. Interestingly, a few popular third-party extensions have replicated this nested section behavior. Apparently some people find it more useful when styling their docs.
Hard lines breaks (
) are odd as well. The entire block must be contained in a line_block element (which would presumably be a child of a paragraph). The line_block element then contains children line elements. The line break is implied between each line element.
A target element holds data defined anywhere within the document (like link or image references in Markdown). A later transform would then copy the URL stored in the target to the corresponding reference or image element.
Raw HTML should go in a raw element. However, I need to confirm that the HTML writer will not wrap the raw text with any HTML tags.
I also need to confirm that literal_block and literal elements get wrapped with the proper HTML tags (
 and  respectively) by the HTML writer.
Images are interesting as Docutils allows an image to either be an inline element or a block level element, which is inline with the HTML spec. However, in Markdown, images only ever appear as inline elements. Shouldn't be a problem. Just need to be careful.
Everything else should be pretty straightforward.

Given the situation with headers  (
) and Docutils sections, I'm thinking that docutils is a non-starter.  This is not valid Markdown output. I could create my own element subclasses which emulate Markdown's headers, but then Docutils writers won't work any more, which is the main benefit of using Docutils.
I should note that because each header generates a nested section, Docutils does not allow header levels to be skipped. In fact, it will crash hard on inconsistently nested levels. Markdown should never crash hard on bad input.
Another issue is that there is no hard rule for which characters in the ReST syntax represent which level. It is simply assumed that they appear in the order they are found (the inconsistency comes when you step back up, then down again -- it is assumed that you use the same pattern going back down). Therefore, the first header is always a level 1 header (
). However, in Markdown the levels are explicit in the syntax. If a user starts with ### Header, then that first header in the document must be level 3 (
). Docutils has no mechanism for retaining that info, or at least, the writers have no mechanism for accounting for it.
Given the above, I don't think I'll be pursuing the use of Docutils at this time. However, I think reviewing it had been beneficial. It is made it more clear how I want to modify/subclass ElementTree.
A benefit of using ElementTree is that the changes can be made more incrementally, running tests as I go, which feels much less intimidating.

Yeah, I was going to comment that when I saw the wacky header tag stuff that I was thinking it was looking less desirable.  But seeing that it can actually break things makes it quite a bit worse considering Python Markdown's goals.

Over the past week I've been slowly putting together an altered ElementTree lib which has ended up re-implementing almost everything (only ElementTree's XPath is used which we don't really use anyway). It is hardly close to done, but it occurred to me that I was more-or-less re-implementing Beautiful Soup's document object.
The one weird thing about Beautiful Soup is that you can't create a document unless you parse something. So to create an empty document which elements can be added to, you need to parse an empty string. I assume parsing as empty string is not too much of a performance hit (I should probably confirm that), so its not too big of a deal, just weird.
Once you get past that, the API is very extensive and easy to work with. It is specifically designed for working with HTML and even gives more control that anything I would have custom built myself. Text is represented as child nodes alongside child element nodes. Every node in the document tree knows about its parents, siblings, children, etc. Methods are provided on each node to insert, insert_before, insert_after, append, and the list go on...
I should note that I would be using version 4 of Beautiful Soup (which breaks compatibility with the more popular version 3,  but is necessary to get Python 3 support). Unlike earlier versions, Beautiful Soup 4 is not an HTML Parser. Is simply wraps existing third party parsers (lxml, html5lib, and Python's HTMLParser) and provides an easy-to-use API for accessing and manipulating the parsed document. Python-Markdown's use case is manipulating/building an HTML document, so the goals align fairly well.
As the home page states: "Beautiful Soup is licensed under the MIT license, so you can also download the tarball, drop the bs4/ directory into almost any Python application (or into your library path) and start using it immediately."  That was not case with earlier versions. Although, there is the issue that the 2to3 tool needs to be run for Python 3, so just copying it into the Markdown lib doesn't make much sense. But it can be listed as a dependency and get installed automatically by the setup script as long as an internet connection is available.
An added plus is that Beautiful Soup comes with its own serializer which is built specifically for HTML (with pretty-printing build in). Although we would loose the ability to distinguish between HTML and XHTML,  the only difference in Markdown's syntax is
 verses
. As both are valid in HTML5, I don't think that really matters anymore. The reference implementation (markdown.pl) outputs XHTML and Beautiful Soup outputs
 for HTML output, so that seems like a reasonable compromise. We had added the HTML output format before HTML5 was even a thing so users could use Markdown's output in HTML4 documents. If it is a problem, we might be able to use a custom output formatter to address the issue (the formatter keyword will accept a callable in addition to the string names of the built-in formatters).
Any thoughts?

Can we change the BeautifulSoup API (i.e. submit a pull request) to make it easier to create an empty document? So that we can simplify our code, at least in the future?

Can we change the BeautifulSoup API (i.e. submit a pull request) to make it easier to create an empty document?

I looked into that (although I have not submitted or requested anything upstream). As I understand it, the assumption is that any BeautifulSoup document is always assumed to have been created by a  parser and that assumption is interwoven throughout the code. For example, the serializer checks which parser was used for various branches in its behavior (eg: output
 or
 depending on which parser was used).
In fact, if you create a fragment and try to serialize it, it will crash hard. It needs to be contained in a "document" object, which is a special Node which holds reference to the parser, among other things. When serializing a child, it looks up the tree to the document root for various data to determine how it should behave. I tried creating a subclass of that document root class which statically sets all the moving parts on the document root, but I am still getting weird errors I can't seem to figure out.

Apparently, I didn't make a copy of my attempt to override the default BeautifulSoup document root class (with one that skips the parsing step and sets defaults). However, I did do this. As I explain in the comments:

Just maybe, the way to parse HTML within a Markdown document is to run the document
through an HTML Parser first. Some parsers, like the HTMLParser included in the
Python Standard lib will properly parse the plain text not wrapped in HTML tags
as plain text and simply return it unaltered. The problem is with Markdown's
autolinks<EMAIL_ADDRESS>and <http://example.com>).

However, as of Python 2.7.3 and 3.2.2, the HTMLParser can now handle invalid HTML
without crashing hard. Below is a subclass of Beautiful Soup's HTML Parser which
accounts for those autolinks and passes them through as text. The crazy idea is that
those text nodes could then be parsed by Markdown and the Markdown Parser would not
need to reimplement a lousy HTML parser with regex.

Perhaps it is just a crazy idea, but it might actually work. If so, the first step in parsing a document would be to pass it to the HTML parser. All of the non-HTML parts would simply be text nodes. Then, loop through those text nodes and convert them to the appropriate block level nodes, then inline nodes, etc.
If we went that way, the no-need-for-a-HTML-parser would no longer exist. What do you think? A bad idea? or brilliant?

If anyone is interest, my (failed) attempt to create a BeautifulSoup document root subclass is here.

Perhaps it is just a crazy idea, but it might actually work. If so, the first step in parsing a document would be to pass it to the HTML parser. All of the non-HTML parts would simply be text nodes. Then, loop through those text nodes and convert them to the appropriate block level nodes, then inline nodes, etc.

This looks like a giant hack, doesn't it?

This looks like a giant hack, doesn't it?

Yeah, your right. Moving on...

Well, it's up to you to make the decision. I didn't even look at the code, just read your summary, so my opinion shouldn't really matter…

I am not sure I understood the rationale for dismissing Docutils so quickly.  It seems to me skipping header levels is pretty easy, provided that you put in the right level of sections.  I have published a Gist to that effect.
The advantages you get are immense — you get a lot of interesting writers (which you also don't have to maintain, so you have that going for you) from both the Docutils and Sphinx ecosystems, the very intriguing Transform API allowing you to modify doctrees after the fact (adding figure numbering, translations, whatever), and a clean contract that this tree format was indeed made for documents, not XML.

@lehmannro the issue is that the HTML output is not valid according to the Markdown rules. According to the Markdown rules, ## Header must result in
Header
. If we use Docutils, we can't guarantee that that is what we will get. Additionally, because Docutils assumes each header starts a new section, it wraps each section in a div. Again, according to the Markdown rules, those divs break the rules. Yes, I understand that without any css rules, the divs have no effect on how the browser displays the HTML, but that is beside the point.
In fact, if you look at the test frameworks for the various Markdown implementations (including the reference implementation), they all contain a bunch of Markdown files with matching HTML files. The tests are run by passing the Markdown file through the parser and comparing the output to the HTML file. Even one character of difference results in a test failure. The Markdown syntax is expected to produce very specific HTML. Any significant variation from that specific HTML is an error. Markdown is very much tied to HTML. See here for why this matters.
That is very different from the approach taken by Docutils. While parts of Docutils closely mirror HTML, that is coincidental. AFAICT, from the get-go, Docutils was designed for representing a document structure regardless of the output format. Therefore, even the ReST to HTML conversion is not always the most obvious. It simply does not give us the option to output the HTML that Markdown users expect/require.
While I agree that a Markdown to Docutils tool would be very useful, it does not serve the Markdown community at large very well. As we are the leading (most downloaded from PyPI at least) implementation of Markdown in Python, unfortunately I don't think we can adopt the use of Docutils for the reasons explained above. That said, a less-mainstream Markdown implementation which supports Docutils and is upfront about the fact that it does not output the expected HTML certainly has a place in the world. As mentioned previously, such an implementation already exists and nothing it stopping anyone from creating others.

I'm not sure the document you linked and your statement are consistent.  Look at any of the examples under What are some examples of interesting divergences between implementations? (eg. ATX headers with escapes) — they ALL have different output.  From the description of the document, its purpose is to “promote discussion of how and whether certain vague aspects of the markdown spec should be clarified.”  I don't think it's trying to publicly shame parsers which do not adhere to the standard (which, and please correct me if I'm wrong, is very loose as illustrated by the document.)
While I cannot stop you from implementing your favorite Markdown parser in any way you want, a statement such as “If we use Docutils, we can't guarantee that that is what we will get.” is simply FUD.  There are strict (and, I would claim, stricter than in the Markdown spec) guarantees as to what Docutils does and doesn't produce.  The first section's title is always a
, the second gets
 and so on.
I don't have any experience with the “Markdown test frameworks” but if they test something that's not in the spec they are plain and simple wrong.  (That being said, if stripping extraneous
s is a requirement for you, you could either look at the new HTML5 writer or at Transforms.  I think dismissing Docutils after a —as it seems— very shallow investigation is unfair and misinforming others who would like to learn from the “leading implementation of Markdown in Python.”)

The first section's title is always a
, the second gets
 and so on.

There is the problem. If I start a Markdown document with ### Header, then the first header must  be an
. This is not possible with Docutils. Docutils forces the first title to be an
. And there is no way to retain that info and alter the behavior of the HTML serializer/writer (if I'm wrong, please enlighten me). This in itself is a sufficient reason to reject Docutils for Markdown.
Regarding my linking to the Babelmark2 FAQ, the point is that implementations should not differ in their output. Yes, unfortunately many do. However, we should not be creating more differences which would only make matters worse. Adopting Docutils would do just that.

if stripping extraneous
s is a requirement for you, you could either look at the new HTML5 writer or at Transforms.

Sorry, I missed this point the first time, so I'll address it now. AFAICT, Transforms run on the Docutils document object, which does not yet contain any
s (the divs are added by the writer). Therefore, Transforms would have no effect. The other option would be to create our own writer. While this is possible, I'm not that motivated to do so. Besides, given my last comment, how would the custom writer know which level to assign to each section's title as that data is not contained in the document object and is not dependent on the order or nesting of the sectionss in the document?
To make this work with Markdown, the way I would want to do it would be to not use Docutils' sections, but then how would I represent headers? According to the Docutils document spec, a title must be the first child of a section and the titles level must be determined by the nested level of the section. Therefore, the spec for the Docutils document object is in direct opposition to the Markdown rules.
As I can't use titles without sections and sections don't work for Markdown, I considered creating my own node subclass which would represent a heading (
). Then, the Markdown parser would insert these objects into the document tree at parse time and the proper level (according to the Markdown rules) could be preserved. However, none of the existing writers would be able to use such a document and the benefits of using Docutils would be lost.
I'm open to alternate solutions here. But personally, I'm not seeing them.

Did you have a look at https://gist.github.com/lehmannro/2d2127b7c839282a673d which I linked earlier?  It produces a
 without any
 just fine.  Sure, you would have to do some bookkeeping for how many more sections you need to nest to get the user's requested header level, but that's far from impossible.

Did you have a look at https://gist.github.com/lehmannro/2d2127b7c839282a673d which I linked earlier?  It produces a
 without any
 just fine.  Sure, you would have to do some bookkeeping for how many more sections you need to nest to get the user's requested header level, but that's far from impossible.

In my opinion, the work (and headaches) to create and maintain code which implements that hack far outweigh the benefits. Give me an example that does not use sections (Markdown has no concept of sections), and perhaps I'll reconsider. In order words, Docutils is not an HTML document object, which is what I need.
My personal opinion is that given the very close mapping between Markdown and HTML, the best way to get from Markdown to Docutils is to do Markdown => HTML => Docutils. An HTML to Docutils tool can exist separately from Markdown and serve a much wider audience but also provide a decent way to get from Markdown to Docutils.
In fact, whatever HTML document object library Markdown uses could also have a [document object] to Docutils tool which would eliminate the need to first serialize the Markdown document and then parse the HTML into another document object. Think ElementTree2Docutils or BeautifulSoup2Docutils. Personally, I'm surprised those tools don't exist already. They could work great for converting Markdown to all of Docutils supported output formats and would serve a much broader audience as well. In fact, BeautifulSoup2Docutils would be immensely useful. You could use it to parse HTML using your choice of any of the decent Python HTML Parsers (as per BeautifulSoup's API) and could output to any of Docutils supported output formats. At that point, any markup language's lack of explicit support for Docutils would only be an optimization issue (skipping the HTML serialization and subsequent parsing would obviously be an optimization -- but offers no additional advantages that I can see).

To be clear, this is a valid Markdown document:
##### Level 5

# Level 1

### Level 3

## Level 2

#### Level 4

# Level 1

###### Level 6

I could keep going, but the point should be obvious. Keeping track of nested section levels would be a real headache when building a Docutils document. Unless someone can point me to a way to not use Docutils sections, consider the subject closed for discussion.

For completeness, I just stumbled on this project: AdvancedHTMLParser. It looks interesting, but its history is limited and I know nothing of its stability. The interesting part is the AdvancedTag object, which is both a node and self printing (using innerHTML and outerHTML). The lib more-or-less mirrors the JavaScript DOM, which may or may not be a good idea.

Just a quick update. My work on an [HTML Node toolkit][1] has stabilized. I'd do a release, except that I haven't actually used it for any real work yet. In any event, its ready to use.
However, I'm not sure I want to use it in Markdown. Now I'm thinking a simpler node structure would be preferred. Perhaps only the nodes represented in the Markdown text. For example, Markdown only has list items, but no parent ul or ol it actually represented in the document (they are only implied). So perhaps the node tree should reflect that. Each list item node could retain which type it is (alphanumeric (and value), dash, asterisk, plus...), but have no parent list node. Then, when rendering (or perhaps in some intermediary transform) the specifics could be worked out (parent node, list type, item value, etc). That should give much more control regarding the various ways that people prefer to have lists rendered and doesn't actually require any modification of the parser, only the rendering (or transform) step would need to be modified.

What about using a json-like structure? Would it be faster to process? also probably it would make it easier to create writers for new output formats.
I think pandoc does something like this, at least looking at this page of  their docs

@andya9 I'm thinking of something very similar to that. It would be more performant to use native Python objects, but yes, something very JSON-like. Perhaps a string representation of the document tree would even be in JSON.

I’m glad to hear that! :smiley:

Also thanks for the link to Pandoc's documentation. I have looked but never found the definition of their internal document structure before. Apparently it was more recently broken out into a separate package and they have a complete definition of the structure. Could be helpful.

I’m really glad I could be useful!

There’s also remarkjs allowing json output

remarkjs is very cool. See mdast for a definition of the Markdown Abstract Syntax Tree which it uses.

Cool indeed! Thank’you for the link, I’ll take a good look

Here is a simple JSON based AST in Python I threw togeather.

Beautiful!
It should make the code much simpler, am I right? We don’t need an ordereddict anymore, if we encode line+column’s start and end inside each element object...
If I understand correctly, now the parser will just need to get Content + Position (start, end; this is the trickiest probably) + Type, your Node class will use them to build a dict object, and we can immediately use it to produce our output (in any format we want!)

I’m not skilled enough to deal with the preprocessing part, but I’d be glad to help with transforming the ast into output (dict object > html element + extensions)

Just wanted to say that I support the decision to leave out Docutils.
Personally I like simple, straight-forward tools that stick to a specific problem ahead, even if this means less features.
Just my personal point of view.

We have decided to defer implementing this until a future release. Therefore I am removing this from the 3.0 milestone.
The reason is that out most important asset is the rich collection of extensions (both first and third party). Making a change of this magnitude would require every third party extension to undergo a complete refactor. At this time the costs outweigh the gains. However, we will re-evaluate again in the future.

When considering a different AST, it would be useful to revisit #215. I think it's very useful for an application to be able to get access to the AST, but the fact that currently essential post-processing happens after serialization prevents that.
Is it necessary for the post-processing to happen on the serialized output or could it, in theory, be done on the AST instead? I guess that with ElementTree as the AST, post-processing on the AST would mean you'd have to parse inline HTML and insert that into the tree as well. This would mean that HTML errors would have to be handled (escalated or repaired) instead of forwarded to the output. In my opinion that's a positive, because catching errors early is generally a good thing, but it might not be the Markdown way.
But if you'd go for a different AST implementation, you could have inline HTML as raw HTML nodes inside the AST and still post-process those textually, while post-processing all generated HTML in the AST instead.


Don't add ICECandidate before SetRemoteDescription

Resolves #1029

I ran all the tests 7 times and no failures! Merging.


computer reboot when run test.py

i dont know why this happen.can anyone help me? tks.

@Hx96 I am sorry I haven't met your problem before.


connect mongodb with srv protocal, and no port

like: mongodb+srv://anker:@xxxx-yyyy.mongodb.net/test?retryWrites=true
so i recommend in DatabaseHelper.js
line: 53
del:            case 'port':
del:                return '27017';
add:            case 'protocal':
add:                return 'mongodb://';
line: 75
del:            let connectionString = 'mongodb://';
add:           let connectionString = this.getConfig('protocal');

This commit should fix it.
Now you can either specify it like this in the /config/database.cfg:
{
    "protocol": "mongodb+srv"
}

Or like this as an environment variable:
MONGODB_PROTOCOL="mongodb+srv"

As for "no port", I don't see how that's possible. If you specify no port, the client will default it to 27017 anyway?

This commit should fix it.
Now you can either specify it like this in the /config/database.cfg:
{
    "protocol": "mongodb+srv"
}

Or like this as an environment variable:
MONGODB_PROTOCOL="mongodb+srv"

As for "no port", I don't see how that's possible. If you specify no port, the client will default it to 27017 anyway?

thanks for answer,
about "no port", it because of i used the mongodb.com atlas service,
the sandbox url is just "mongodb+srv://anker:@xxxx-yyyy.mongodb.net/test?retryWrites=true",
with no port, -_- , sorry about that
and, hashbrown-cms is best api based cms  i ever meet, open source and node server and react client, perfect

Ah, I see. No worries.
Thanks for the compliment, I appreciate it :smile:


unable to clone

remote: Repository not found.
fatal: repository 'https://github.com/upes-open/PeerPool.git01~/' not found
when i am doing git clone https://github.com/upes-open/PeerPool.git01, i am getting this issue

that is prolly the wrong url
try
git clone https://github.com/upes-open/PeerPool.git

yes, that works.. thanks for your help


Alaboudi1 patch 1

This pull request seems to be done by mistake? Please feel free to reopen if that was not the case.


:arrow_up: Update aiohttp

Changes
Lets 3.7.4post0 until the last version of 3.x.x
https://docs.aiohttp.org/en/stable/faq.html#what-is-the-api-stability-and-deprecation-policy
Check off the following

[x] I have tested my changes with the current requirements
[x] My Code follows the pep8 code style.

I don't think we use deprecated stuff, so this should be ready for merge. Waiting for someone to test it tho.

it probably works

This cannot be merged right now as it causes an error. (as caught by Scrutinizer)
error: Could not find suitable distribution for Requirement.parse('aiohttp<3.8.0,>=3.7.4post0')

This cannot be merged right now as it causes an error. (as caught by Scrutinizer)
error: Could not find suitable distribution for Requirement.parse('aiohttp<3.8.0,>=3.7.4post0')

This is kinda messed up because it still has websockets. Im closing for that reason.


chore: release main

:robot: I have created a release beep boop
bitswap: 1.1.1
1.1.1 (2024-05-20)
Dependencies

The following workspace dependencies were updated

dependencies

@helia/utils bumped from ^0.3.0 to ^0.3.1

block-brokers: 3.0.1
3.0.1 (2024-05-20)
Dependencies

The following workspace dependencies were updated

dependencies

@helia/bitswap bumped from ^1.1.0 to ^1.1.1
@helia/utils bumped from ^0.3.0 to ^0.3.1

http: 1.0.8
1.0.8 (2024-05-20)
Dependencies

The following workspace dependencies were updated

dependencies

@helia/block-brokers bumped from ^3.0.0 to ^3.0.1
@helia/utils bumped from ^0.3.0 to ^0.3.1

interop: 6.1.4
6.1.4 (2024-05-20)
Dependencies

The following workspace dependencies were updated

dependencies

@helia/block-brokers bumped from ^3.0.0 to ^3.0.1
@helia/http bumped from ^1.0.7 to ^1.0.8
helia bumped from ^4.2.1 to ^4.2.2

utils: 0.3.1
0.3.1 (2024-05-20)
Bug Fixes

check eviction filter for new providers (#542) (f46700f), closes #501
type error (#537) (e6b976a)

helia: 4.2.2
4.2.2 (2024-05-20)
Dependencies

The following workspace dependencies were updated

dependencies

@helia/block-brokers bumped from ^3.0.0 to ^3.0.1
@helia/utils bumped from ^0.3.0 to ^0.3.1

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/ipfs/helia/releases/tag/bitswap-v1.1.1 :sunflower:

:robot: Release is at https://github.com/ipfs/helia/releases/tag/helia-v4.2.2 :sunflower:

:robot: Release is at https://github.com/ipfs/helia/releases/tag/http-v1.0.8 :sunflower:

:robot: Release is at https://github.com/ipfs/helia/releases/tag/interop-v6.1.4 :sunflower:

:robot: Release is at https://github.com/ipfs/helia/releases/tag/utils-v0.3.1 :sunflower:


Fix issue https://github.com/davisking/dlib/issues/925

This fix suggested by davisking make unit tests more reliable. Fix issue https://github.com/davisking/dlib/issues/925

Sweet, thanks for the PR :)
Can you change the tab to a space?


Develop an API to list the top questions

Develop an API to list the top questions

The api was modified to listing the most recent questions which is the default option to sort questions


chore: change background color

Hey this is a new PR

Your preview environment has been successfully deployed !
Click on the link below to open your service:
:point_right: TODO app


proposal: Go 2: named parameter in function invocation

Context:
Go does not allow specifying the name of function parameters in function invocations like in other languages. I don't understand the reasoning behind this.
Proposal:
allow the function parameter name to be specified explicitly in the function invocation.
func Foo (p1 int, p2 int, p3 int) {}
Foo(p1=1, p2=2, p3=3)
The reason for this is not necessarily because I would like to list them in another order in the invocation.
I don't even mid if the compiler forces the parameters to be specified in the same order as the declaration.
The reason is to avoid errors by making it clear what the intended use for the parameter is.
Remember, the function declaration and function invocation are in different places. They are often written by different people in a team or in different teams.
So when I read the invocation Foo(1,2,3), then I need to go look up the function declaration either in the code or in the documentation to understand what these parameters do. When I read Foo(p1=1, p2=2, p3=3), i don't need to look that up, it gives me all I need.
Also, code evolves. When the definition of function changes, and the function parameters are not named in the function invocation, then there is a potential for unintended bugs. You cannot expect the programmer who needs to modify the function declaration to go fix every invocation in every project (some of which he might not even have privilege to). By changing the name of a parameter, the function programmer can indicate a change in the intended use of the parameter.
When everyone consistently names their parameters during invocations, then a change in the function declaration is immediately caught by the compiler and the API users can fix their code accordingly.
I have seen comments on similar proposals like "I don't see a need for this" do not seem to have much experience with the power of this suggested feature.
IN MY OPINION (and I understand that does not necessarily have to be other people's opinion): it makes code readable, unambiguous and prevents unintended errors. A lot of programmers coming from other programming languages probably agree with me.  If you don't see a need for it, then don't use it but at least allow people who do feel a need for it to use it. It's the lack of features like this that makes the transition from Python to Go less palatable for me.

Closing as dup of #12296.

Hi, is there an update on this? Have the problems addressed in #12296 been discussed any further? I think this would be an awesome feature!

Sorry, these proposals have been declined.


zombie will pick up the glow squid

Cultural Delights 1.0.5+1.20.1 [Refabricated]

If someone needs this mod now
https://github.com/north82/Cultural-Delights-Fabric/releases/tag/1.20.1-1.0.5-fix

#34


hotfix: allow free user to update pre-existing card info

permit Free users to update their pre-existing billing info (on file from past Professional plan subscriptions)
blocks generate repo PR 1579

@tachyonbrush we must make sure to revert the final commit 0ac87869ee00331d5ea95248860cd5fe5883fdaa message revert this egg file commit once djstripe PR merged hotfix/billing-info-permission into master once you merge the djstripe PR so that this repo points to the correct master branch of djstripe


interpreter on_functiondef uses  node...pop()

This causes a bug (for instance if a function def is in a function, it will not have "popped" arguments the second time through).
Similar to error recently fixed in on_boolop().

fixed.


Feat/4.1.2

Description
Cloud functions to obtain restricted algolia search keys for the client to use
Motivation
Testing Guidelines
Release Checklist

[x] This code has unit tests
[x] I have a plan to verify that these changes are working as expected after deploy
[ ] I have a plan for how to revert, rollback, or disable these changes if things are not working as expected

Security Checklist
This PR creates, modifies, or deletes:

[x] API routes: API routes, parameters, or user authorization
[x] Authentication: Authentication mechanism
[x] Credentials: Server side credentials, or secrets in configuration / source code
[ ] Cryptography: Encryption, hashing, certificates, signatures, random numbers, etc.
[ ] User data: personal information handling, logs, error messages, etc.

Additional Notes

Scratch what I suggested about two endpoints. Can we just add an isAuthenticated boolean field to the request, check auth if requested, and return {isAuthenticated, indexName, searchKey} ?
I am finding it easiest on the frontend to have separate paths just in the public redux actions and passing the isAuthenticated along.

sure, doing it now

@alexjball I have made all the changes you requested except for the additional test cases. will be doing that after my next PR


Bringing the flows to Front

Is there any way to bring the canvas flows to the front ? I have 2 sets of objects, geojsonLayers and the canvas lines. I tried to use bringToFront() but it seems its not implemented for Canvas elements.
I read in a forum that there is this solution:

mapInstance.getPanes()['tilePane'].style.zIndex =<PHONE_NUMBER>;

here is the link of the example:
http://playground-leaflet.rhcloud.com/xopi/1/edit?js,output

Hey @Zahma sorry I didn't get to this before you closed it.  Did you figure it out?  You can certainly change the z-index for the 2 <canvas> DOM elements that this layer manually creates and inserts into Leaflet's layer DOM structure.  The default CSS is:
.leaflet-map-pane canvas {
    z-index: 100;
}

The layer will, however, respect your wishes if you tell it during construction to use a different Leaflet JS map pane.  If you haven't used these before, look at these references:

http://leafletjs.com/reference-1.2.0.html#map-pane
http://leafletjs.com/examples/map-panes/

Thank you, I found the solution, I used the setPane method

Hi, thanks for the suggestion. In my development I had included jQuery so I was able to detach the container and append it to the pane where I want the flows to be. It works well, but you suggestion is more 'self-contained' and would certainly work better for several flow layers.


fix: Mod directory being used before it's setup

Description
This fixes ModsFolderPath in UE4SS-settings.ini being broken.
Type of change
Please delete options that are not relevant.

[x] Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?
Move Binaries/Win64/Mods to Binaries/Mods, and set ModsFolderPath to ../Mods.
Checklist
Please delete options that are not relevant. Update the list as the PR progresses.

[x] I have added the necessary description of this PR to the changelog, and I have followed the same format as other entries.

This does not work for me. And your test case seems weird as Mods are in Binaries/Win64/ue4ss/Mods.
Default directory is <UE4SS.dll directory>/Mods. I tested moving Mods into Content/Paks folder and set override path config to ../../Content/Paks/Mods, it doesn't find mods and there is no error. If I set it to ../../../Content/Paks/Mods I get Error: directory_iterator::directory_iterator: unknown error: "../../../Content/Paks/Mods". The latter is the correct path.

I do not have any override.txt file.

This does not work for me. And your test case seems weird as Mods are in Binaries/Win64/ue4ss/Mods. If I set mine to ../../Mods and move Mods into Binaries/Mods it does not find mods and there is no error.
Default directory is <UE4SS.dll directory>/Mods. I tested moving Mods into Content/Paks folder and set override path config to ../../Content/Paks/Mods, it doesn't find mods and there is no error. If I set it to ../../../Content/Paks/Mods I get Error: directory_iterator::directory_iterator: unknown error: "../../../Content/Paks/Mods". The latter is the correct path.

I do not have any override.txt file.

Okay obviously I typo'd it, I meant to say that I moved Mods into Content/Paks.
As far as that error, the path is from the Win64 dir, not the ue4ss dir.

Note that there's another problem with the mod directory override that I'm currently working on.
I'll update this PR.

This is now ready for testing again.

I've updated my test-case, sorry again for messing that up.


roate x axis labels in the reverse direction

http://www.d3noob.org/2013/01/how-to-rotate-text-labels-for-x-axis-of.html
Can we rotate x axis labels in this manner? I have done this: http://c3js.org/samples/axes_x_tick_rotate.html, but rotating the other way seems much better. Any inputs would be appreciated.

This is still a bug with timeseries as @agdev says.

Any news? November 2017 and the bug is still there.


Validate schema fields have valid presto characters

Datastillery/smartcitiesdata#909

Shouldn't be a problem until we need to repost them, in which case we would want valid names


EventEmitter

➤ ./bin/dew
> obj={};obj.on("foo",foo);obj.emit("foo","bar")
native foo test run 1 args 1 arg1 bar

Adds a .on() and .emit() method to objects that act as expected for the EventEmitter pattern, is super minimalistic but functions.
Also has methods to emit/receive events directly from the system:
dew_t dew_lib_emit(dew_t stack, dew_t obj, char *event, uint8_t len, dew_t val, dew_t res);
dew_t dew_lib_on(dew_t stack, dew_t obj, char *event, uint8_t len, dew_t callback, dew_t res);

Bespoke. @codycollier


Question: Supported RDS Databases (and versions) List

Hello SST Community! I have a question about support for more database servers and versions:
Do we know if mysql8.x or microsoft sql server or more are supported (besides the mentioned three below code block)?
If so, is there a list of databases and its versions somewhere to refer to? Similar to how we look up docker images & its versions available - see docker php tags example.
Below documentation snippet only mentions mysql5.6, mysql5.7 and postgresql10.14:
engine

Type : string

Supported engine are mysql5.6, mysql5.7, and postgresql10.14.

And config example of what I would like to use:
import { RDS } from "@serverless-stack/resources";

new RDS(this, "Database", {
  engine: "mysql8.x",
  defaultDatabaseName: "awesomecompany_mysql8",
});

The main reason why I ask this question, it's because AWS CDK and AWS-related online documentation have not been very helpful in querying and looking for answers.
Any kind of help that can point me into the right direction is much appreciated always 😃

It's limited by https://github.com/aws/aws-cdk/issues/20197.

Thanks @karmux . I checked out the thread link you've shared, and participated in the +1 vote, as suggested by skinny85 on that thread.

Cheers, I'll close this issue at this time. Once aws-cdk above has traction, we'll watch for your upcoming SST change logs 👍


Depth image compression by colorization contains ambiguities

This issue is about the Intel Whitepaper about Depth image compression by colorization for Intel® RealSense™ Depth Cameras #6289:
First of all we noticed that the formula for the uniform depth colorization lacks information about the datatype.

Looking at the formula, it seems the calculation is done in float, but it lacks a specification if the value is rounded or floored. We noticed artefacts at very specific points in the encoded depth image if we stored the float calculation directly into an int (flooring). With rounding to an int, the artefacts disappeared.
Second, the inverse hue encoding does not seem to work as described with the formula:

Let's calculate a real-world example:
# input
d = 2200 # depth in mm
d_min = 300
d_max = 6000

# calculate disparity values
# if any value = 0 => set to 0
disp = 1 / d # 0.0004545454545
disp_max = 1 / d_min # 0.003333333333
disp_min = 1 / d_max # 0.0001666666667

# norm
d_norm = (disp - disp_min) / (disp_max - disp_min) # 0.09090909089

And now? We guess this is just the value to multiply with the independent values (1529) as in the original normalisation formula. So the inverse normal calculation would be (again rounded):

Are our assumptions right?

Hi @cansik  In general, it is not recommendable to use the depth compression by colorization paper.  Because the depth recovery part of the process has to be performed outside of the RealSense SDK and because there are only a few programming references for the method outside of the paper, it can be a difficult technique to implement.  It is also best suited to C++ users because of the difficulty of coverting the code in the paper for other languages such as Python.
One also has to make the assumption that the result of the division in the dnormal equation is multiplied by 1529 because a maths sign is not confirmed in the paper's version of the formula.

If you do wish to continue working with the technique in this paper: dmin and dmax (minimum depth and maximum depth) are indeed defined as floats in the paper's C++ code.

In regard to the meaning of 1529, my interpretation of the paper is that the color values are in the range 0-255, and the hue value is arrived at with the following equation:
(hue value x 6) - 1
The hue value will be between 0 and 255.
So the maximum hue value of '1529' would be achieved by multiplying 255 by 6 to get 1530 and then subtracting 1 from 1530 to arrive at the final value of 1529.
(255 x 6) - 1 = 1529
My further understanding is that '6' is derived from RGB hue being able to have 6 gradations of intensity in both directions.

I do not have enough information about the compression technique in this paper to confirm whether your final assumptions about dnormal are correct, unfortunately.

@MartyG-RealSense Thank you very much for this detailed answer. I have written an implementation of this paper in Python (Encode / Decode), as well as decoders for HLSL and GLSL. And I had to make my assumptions for the code to run properly in the first place. Most important was the rounding of the float values after normalisation on the encoder side, so that the if-conditions cover all cases.
I just wanted to ask if these are known problems. Perhaps it would also be possible to expand the white paper.
Do you accept suggestions for improvement via readme.com?

I have not previously seen your particular problem with artifacts in the image when not rounding up floats to ints.  The most recent query before your one - which can be found at https://github.com/IntelRealSense/librealsense/issues/10145 - was regarding why the RGB values were negative.
There are no plans to expand the depth compression white-paper or encourage its use.  It should be considered as experimental, though RealSense users are welcome to try it and adapt it if they so wish.

Hi @cansik  Do you require further assistance with this case, please?  Thanks!

@MartyG-RealSense Thank you for your help, will close this issue.

Thanks very much @cansik for the update!

Although this comes late in the game, for future reference, here is a vectorized Python implementation with the equations corrected (to the best of my knowledge)
https://github.com/cheind/hue-depth-encoding


4 Confluent Kafka Connectors Not Vulnerable (Redshift Sink, S3 Sink, MongoDB Connector and JDBC Connector)

Vendor : Confluent Kafka
Product Name: Confluent Kafka Connectors

Redshift Sink (https://docs.confluent.io/kafka-connect-aws-redshift/current/overview.html)
S3 Sink (https://docs.confluent.io/kafka-connect-s3-sink/current/overview.html)
MongoDb Kafka Connector (https://docs.mongodb.com/kafka-connector/master/)
JDBC Connector (Source and Sink) (https://docs.confluent.io/kafka-connect-jdbc/current/index.html)

Version(s) affected: None
Status: Please choose from one of the following - Not Affected
Update Available: None
Notes: Each of the mentioned connector was download and its source analysed to identify if the vulnerable log4j version or a reference to it exists across property, manifests or other packaged JAR binaries within them. These connectors were not found to be vulnerable.
References:

https://kloudle.com/blog/confluent-kafka-connector-analysis-for-log4j-cve-2021-44228-vulnerability
https://support.confluent.io/hc/en-us/articles/4412615410580-CVE-2021-44228-log4j2-vulnerability#impact-to-connectors

Last Updated: 17th Dec 2021

This product has been added to the repository. Thank you.


CI update

Update the action version and delay the auto-merging until test checks are passed

Codecov Report

Merging #88 (e3bb5dd) into master (b8829b6) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #88   +/-   ##
=======================================
  Coverage   83.80%   83.80%
=======================================
  Files           7        7
  Lines         142      142
=======================================
  Hits          119      119
  Misses         23       23

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

Thanks @zdrawia ! :)


npm run ios fails with "Print: Entry, ":CFBundleIdentifier", Does Not Exist"

npm run ios fails
When I run npm run ios I am encountering the bellow err, I am using<EMAIL_ADDRESS>and<EMAIL_ADDRESS>
Command /bin/sh failed with exit code 2
** BUILD FAILED **
The following build commands failed:
PhaseScriptExecution Start\ Packager /Users/foo/ws/foocode/serverless/aws-mobile-react-native-starter/client/ios/build/Build/Intermediates.noindex/React.build/Debug-iphonesimulator/React.build/Script-006B79A01A781F38006873D1.sh
(1 failure)
Installing build/Build/Products/Debug-iphonesimulator/awsmobilereactnativesample.app
An error was encountered processing the command (domain=NSPOSIXErrorDomain, code=2):
Failed to install the requested application
An application bundle was not found at the provided path.
Provide a valid path to the desired application bundle.
Print: Entry, ":CFBundleIdentifier", Does Not Exist
Command failed: /usr/libexec/PlistBuddy -c Print:CFBundleIdentifier build/Build/Products/Debug-iphonesimulator/awsmobilereactnativesample.app/Info.plist
Print: Entry, ":CFBundleIdentifier", Does Not Exist
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR<EMAIL_ADDRESS>ios: react-native run-ios
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>ios script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR!      /Users/ragmurug/.npm/_logs/2017-10-30T15_00_16_168Z-debug.log

Can you see if the recommendations in this issue help?

I had this error at first and got the :CFBundleIdentifier", Does Not Exist" #20 error to disappear once I downgraded to npm version 4.6.1.
I'm still getting the last few errors though (see my post named " Failed at the<EMAIL_ADDRESS>ios script."

Just solved the outstanding errors I had but Updating Xcode.


http_server package does not always treat the range header correctly.

 Issue by sgjesse
Originally opened as dart-lang/sdk#21587

If there is a Range header in the request the Content-Length header of the response will always be set to the size of this range and the Content-Range to this range. However the resource might be shorter causing less bytes to be served.
For an example see: https://github.com/dart-lang/appengine/issues/15

 Comment by sgjesse

Proposed change: https://codereview.chromium.org/721213002.

 Comment by kevmoo

Available in the latest release of http_server - https://pub.dartlang.org/packages/http_server

 Comment by kevmoo

r41775

Added Fixed label.


Tweaks on top of taralx PR...

Notable Changes

preserve existing signature (no second param) to matching any existing introspection schemes - maybe cost a few characters, but it's a bit more future-proof/robust to potential future options we add.
Router options are now:

base?: string - optional base path
routes?: any[] - optional array of routes (for introspection or preloading with advanced regex)
unrelated: adds a bit of added interface for the internal Request definition to address #47
unrelated: export TS interfaces to address #38
thanks to the @taralx refactor, we can now support preloading routes, which means those rare edge cases of crazy regex (previously not possible) requirements, can now be force-fed into the router (at config stage, or even after... also thanks to the more exposed router.routes array.  Pretty sick!

Example of manual loading routes:
// EXAMPLE 1 mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm
it('allows preloading advanced routes', async () => {
  const basicHandler = jest.fn(req => req.params)
  const customHandler = jest.fn(req => req.params)

  const router = Router({
                  routes: [
                    [ 'GET', /^\/test\.(?<x>[^/]+)\/*$/, [basicHandler] ],
                    [ 'GET', /^\/custom-(?<custom>\d{2,4})$/, [customHandler] ], // match 2 to 4 digits only
                  ]
                })

  await router.handle(buildRequest({ path: '/test.a.b' }))
  expect(basicHandler).toHaveReturnedWith({ x: 'a.b' })

  await router.handle(buildRequest({ path: '/custom-12345' }))
  expect(customHandler).not.toHaveBeenCalled() // custom route mismatch

  await router.handle(buildRequest({ path: '/custom-123' }))
  expect(customHandler).toHaveReturnedWith({ custom: '123' }) // custom route hit
})

// EXAMPLE 2 mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm
it('allows loading advanced routes after config', async () => {
  const handler = jest.fn(req => req.params)

  const router = Router()

  // allows manual loading (after config)
  router.routes.push([ 'GET', /^\/custom2-(?<custom>\w\d{3})$/, [handler] ])

  await router.handle(buildRequest({ path: '/custom2-a456' }))
  expect(handler).toHaveReturnedWith({ custom: 'a456' }) // custom route hit
})

DISCUSSION: If we change r to routes, and prevent the prebuild.js from mangling it, the filesize does grow (405b --> 409b), but so does the legibility around this weird introspection.  Thoughts?  I'm all for absolute minimum filesize, but... legibility.  Torn.  Don't think we would have to count the decision on this one as a breaking change (if we opt for "routes"), because the existing pattern isn't documented, and is hopefully only used in debugging patterns at most.  Regardless of the path chosen here, documentation will be added to disclose this functionality.
UPDATE: unless there's strong objection, I'm opting for readability over ultimate filesize... router.routes it is
Filesize Comparison (pretty sure this includes the type declaration, so appears artificially inflated a bit)

current<EMAIL_ADDRESS>@next ("routes"<EMAIL_ADDRESS>@next ("r"<EMAIL_ADDRESS>

Noteable Changes

preserve existing signature (no second param) to matching any existing introspection schemes
options are now:

base?: string - optional base path
r?: any[] - optional array of routes (for introspection or preloading with advanced regex)

If we change r to routes, and prevent the prebuild.js from mangling it, the filesize does grow (405b --> 409b), but so does the legibility around this weird introspection.  Thoughts?  I'm all for absolute minimum filesize, but... legibility.  Torn.

@taralx

:+1:

:+1: still

Gonna document a bit and cut it loose (hopefully tonight).  BTW, any idea how to cut the size bundlephobia sees?  We're getting 479 bytes gzipped (not really relevant) and minified (definitely relevant)... but I get ~408 bytes testing locally.

What is bundlephobia?

Analyzes bundle size... basically the real cost of adding a module to your project.
<EMAIL_ADDRESS>

I poked at bundlephobia, and they're using webpack to "build" the package. I suspect it's measuring some webpack overhead.


Permutation tutorial

Tutorial with math exmaples

done

Please highlight the formulas and close this issue. Great job.


Fix search by implementing algolia v3

It seems Algolia has made a few changes which

Disabled our scraper
Made some new things mandatory

Fixing this in the V2 implementation did not seem to work. I started implementing V3 and started tweaking the styling.
Done

The search box is pretty much done
The CSS variables are the quickest way to get the overlay styled

TODO

[ ] Test if it works
[ ] Test if it's pulling recent data
[x] Design the overlay through their CSS variables (you might want to export our tailwind colors as CSS variables btw)

I'm in contact with them to see how our scraper should work (they made it much more complicated than plug-and-play). It is giving me results locally, but I'm still unsure which data "index" it pulls data from and if that's up to date.
@noccor it would be cool if you could test some recent pages that were added to confirm it's actually returning information.

@wesoudshoorn @kouliavtsev  - It's not showing any results for Logging.
It is pulling in the Node 3.x docs. We also have node 2.x docs, which means you see duplicate results for each version. Can we ignore the 2.x docs for node?

@kouliavtsev @noccor we should probably use this configuration: https://docsearch.algolia.com/docs/legacy/required-configuration#introduce-global-information-as-meta-tags
We can probably also use it to show relevance for specific programming languages, too.
I would like to ship a fix for the most pressing issues asap and then move forward to improving our configuration, and maybe getting rid of the Algolia logo. There are a few more things we might be able to improve our search with.

@wesoudshoorn
Reverted changes in this commit https://github.com/appsignal/appsignal-docs/pull/893/commits/8a59b1d2e5c9eeeb0fadd6401a010e2b2379f780 due to formatting CI not passing the check.

@jieunization @wesoudshoorn
Implemented the styling changes. I have tried to make the functionality look as much as AppSignal.
Can review the changes before we ship this?

The search input is a bit opinionated, and the order of the logo, placeholder, and key hints is set.
I have tried to keep the way the input looks as close as possible to https://www.appsignal.design/components/form
The shadow has been removed, and the x padding has been set to 0.75rem.
The height of the input has been updated to 36px, which is the height that we used before.

The search modal looks like this:

I have made the modal more flat by removing all the shadows.
The search result items have box shadows with less depth to make them more subtle.
The search input has been made more round.

@kouliavtsev Thanks for building it :)

We have the overlay component : https://www.appsignal.design/components/overlay

the background color should be more darker
I am not sure we can chance it, but probably we can have a title of the overlay box and the close button too?

This should be center aligned, and icon size should be smaller.
We also have the pattern for the empty state : https://www.appsignal.design/patterns/empty-state

Mobile ver: we can see there is a 'cancel' button

'Close' can be the better labeling (we have the cancel feature when they want to clear the keywords)
Can you make the 'close' as the button we have? I am not sure it's possible to change from algolia

@jieunization
It's a bit hard to style due to limitations by Algolia.

Done
I have centered the logo, yet making the same empty-state pattern is a bit challenging.

Because the items below try searching for is a list (we will have to update it) I have not centered it.

Otherwise, it will look like this:

Same as with point 2, it's a bit hard to style due to limitations by Algolia.

Let's ship this one as is, and we will try to implement the feedback at a later stage.

@kouliavtsev I understand it! Let's ship it the style :)
Then can you just change the labeling to 'close'?

@jieunization Thanks for jumping in and providing some design feedback. Unfortunately we have some limitations due to the way this library works. To make it forward-compatible (able to upgrade to a next version) we should also be careful to customize it too much.
@kouliavtsev great job on the styling!


spelling of compiler flag WIFIMODE not consistant

Compiler flag to enable WiFi have different spelling:
RACK32       : WIFIMODE
ROOM8266 : WIFI_MODE
The platformio.ini doesn't take care for it.
Should be the same spelling for consistancy and required if used FW wide in common modules.

Looks like I've to apologize!
I checked again today and found out that WIFI_MODE is spelled the same in RACK32 and ROOM8266.
It's the platformio.ini where the typo crept in.  WIFIMODE has to be replaced with WIFI_MODE.
Please double check. Sorry for the confusion.


spelling fixes

Thanks for the contribution!  This is Square's repo that keeps our tags and any patches from the main pants repo and we don't accept PRs to it directly.
If you would like to contribute any changes to the Pants project, see the instructions at  http://pantsbuild.github.io/howto_contribute.html


look at SIO for textual entities and subclasses

From<EMAIL_ADDRESS>on February 02, 2011 13:27:55
http://semanticscience.org/resource/SIO_000651 SIO describes textual entities and subclasses such as abstract.
From discussion on the SIO list - http://groups.google.com/group/sio-ontology http://semanticscience.org/resource/SIO_000188 An abstract section is a document section that provides brief summary
of a document that explains the main argument(s), topic(s) or
findings. http://purl.obolibrary.org/obo/IAO_0000315 definition: A summary of the entire document that is substantially
smaller than the document it summarizes. It is about the document it
summarizes.
It may be worth looking at reconciling the 2 resources.
Original issue: http://code.google.com/p/information-artifact-ontology/issues/detail?id=101

From<EMAIL_ADDRESS>on February 02, 2011 10:28:28
Cc: michel.dumontier


AppBar is not css'ed

I'm pretty sure that I've missed something since there's no anything about that over Google.
Anyway, when im trying to use AppBar, im getting non css'ed AppBar, created using create-react-app :
import React, { Component } from 'react';
import AppBar from 'react-toolbox/lib/app_bar/AppBar';

App.js
class App extends Component {
  render() {
    return (
      <div className="App">
        <AppBar title="React Toolbox" />
      </div>
    );
  }
}

export default App;

index.js
import React from 'react';
import ReactDOM from 'react-dom';
import ThemeProvider from 'react-toolbox/lib/ThemeProvider';

import theme from './assets/react-toolbox/theme'
import App from './App';
import registerServiceWorker from './registerServiceWorker';
import './assets/react-toolbox/theme.css';
import './index.css';

ReactDOM.render(
  <ThemeProvider theme={theme}>
    <App />
  </ThemeProvider>,
  document.getElementById('root')
);
registerServiceWorker();

package.json:
{
  "name": "jsfs",
  "version": "0.1.0",
  "private": true,
  "dependencies": {
    "react": "^15.5.4",
    "react-dom": "^15.5.4",
    "react-toolbox": "^2.0.0-beta.11",
    "react-toolbox-themr": "^1.0.2"
  },
  "devDependencies": {
    "react-scripts": "1.0.6"
  },
  "reactToolbox": {
    "include": [
      "BUTTON",
      "DATE_PICKER"
    ],
    "customProperties": {
      "animation-duration": "0.3s",
      "color-accent": "var(--palette-pink-a200)",
      "color-accent-dark": "var(--palette-pink-700)",
      "color-primary-contrast": "var(--color-dark-contrast)",
      "color-accent-contrast": "var(--color-dark-contrast)"
    },
    "output": "src/assets/react-toolbox"
  },
  "scripts": {
    "start": "react-scripts start",
    "build": "react-scripts build",
    "test": "react-scripts test --env=jsdom",
    "eject": "react-scripts eject",
    "toolbox": "react-toolbox-themr"
  }
}

@idangozlan You need to include "APP_BAR" in your package.json like this:
  "reactToolbox": {
    "include": [
      "APP_BAR",
      "BUTTON",
      "DATE_PICKER"
    ],

That isn't documented in the README, but I found it here:
https://github.com/react-toolbox/react-toolbox-themr/blob/master/src/constants.js#L20

Thanks :) I've created pull request with a new Readme file that includes also the appbar, so others will be able see that.


quote COLLECTD_HOST in template

This blows up if your hostname has a hyphen in the name

Oh, I'm sorry, @mattrobenolt. I just saw your PR during inbox cleanup. This was probably fixed in #13.
Thank you anyway!

Yup, it's the same patch. :)


Remove version: taers232c.GAMADV-XTD3 version 6.72.14

This version is older than what has been set in max-versions-to-keep by the publisher.
Microsoft Reviewers: Open in CodeFlow

URL: https://github.com/taers232c/GAMADV-XTD3/releases/download/v6.72.14/gamadv-xtd3-6.72.14-windows-x86_64.msi
OK
(Automated message - build 910)


Undefined is not an object(evaluation ''_AppNavigator/.AppNavigator.router")

I am following the redux-example.
https://github.com/react-community/react-navigation/tree/master/examples/ReduxExample
Full detail and code is here:- http://stackoverflow.com/questions/44023043/undefined-is-not-an-objectevaluation-appnavigator-appnavigator-router

I got the issue, Issue was related to something redux store recreation.
Thank you :)

what was the issue? I'm having the exact problem

@anuj-dhanju what was the issue? Im having this same problem


feat: generate auto documentation for internal functions, adds typedoc

Description
Added typedoc generate auto-documentation for internal functions.
Checklist:

[✔ ] This pull request follows our contribution guidelines
[✔ ] I have performed a self-review of my own code
[ ✔] I have commented on my code, particularly in hard-to-understand areas
[✔ ] I have made corresponding changes to the documentation (postman and swagger)
[❌ ] I have written tests for the code
[ ✔] I have updated .env.example file for new .env variables

What is current behavior?

Closes #59
What is new behavior?
Now from this typedoc will auto-generate doc for internal functions
Additional information
https://typedoc.org/guides/installation/

@shivamvishwakarm Closing this due to conflicts


Can I have a demo app with S3 upload code?

Hello, I have read this guide: https://github.com/VeliovGroup/Meteor-Files/wiki/AWS-S3-Integration

I have created S3 bucket, key, secret, cloudfront.
I created a lib folder in meteor and inserted 3 files:

meteor-files.js:

this.Images = new Meteor.Files({
  debug: true,
  collectionName: 'Images',
  allowClientCode: false, // Disallow remove files from Client
  onBeforeUpload: function (file) {
    // Allow upload files under 10MB, and only in png/jpg/jpeg formats
    if (file.size <= 10485760 && /png|jpg|jpeg/i.test(file.extension)) {
      return true;
    } else {
      return 'Please upload image, with size equal or less than 10MB';
    }
  }
});

if (Meteor.isServer) {
  Images.denyClient();
  Meteor.publish('files.images.all', function () {
    return Images.find().cursor;
  });

} else {

  Meteor.subscribe('files.images.all');
}

stream-files.js

this.Videos = new FilesCollection({collectionName: 'Videos'});

// Upload sample files on server's startup:
if (Meteor.isServer) {
  Meteor.startup(function () {
    Images.load('https://raw.githubusercontent.com/VeliovGroup/Meteor-Files/master/logo.png', {
      fileName: 'logo.png'
    });
    Videos.load('http://www.sample-videos.com/video/mp4/240/big_buck_bunny_240p_5mb.mp4', {
      fileName: 'Big-Buck-Bunny.mp4'
    });
  });

  Meteor.publish('files.images.all', function () {
    return Images.find().cursor;
  });
  Meteor.publish('files.videos.all', function () {
    return Videos.find().cursor;
  });

} else {
  Meteor.subscribe('files.images.all');
  Meteor.subscribe('files.videos.all');
}

AWS-S3-Integration.js

var knox, bound, client, Request, cfdomain, Collections = {};

if (Meteor.isServer) {
  // Fix CloudFront certificate issue
  // Read: https://github.com/chilts/awssum/issues/164
  process.env.NODE_TLS_REJECT_UNAUTHORIZED = 0;

  knox    = Npm.require('knox');
  Request = Npm.require('request');
  bound = Meteor.bindEnvironment(function(callback) {
    return callback();
  });
  cfdomain = 'https://xxx.cloudfront.net'; // <-- Change to your Cloud Front Domain
  client = knox.createClient({
    key: 'xxx',
    secret: 'yyy',
    bucket: 'zzz',
    region: 'jjj'
  });
}

Collections.files = new FilesCollection({
  debug: false, // Change to `true` for debugging
  throttle: false,
  storagePath: 'assets/app/uploads/uploadedFiles',
  collectionName: 'uploadedFiles',
  allowClientCode: false,
  onAfterUpload: function(fileRef) {
    // In onAfterUpload callback we will move file to AWS:S3
    var self = this;
    _.each(fileRef.versions, function(vRef, version) {
      // We use Random.id() instead of real file's _id
      // to secure files from reverse engineering
      // As after viewing this code it will be easy
      // to get access to unlisted and protected files
      var filePath = "files/" + (Random.id()) + "-" + version + "." + fileRef.extension;
      client.putFile(vRef.path, filePath, function(error, res) {
        bound(function() {
          var upd;
          if (error) {
            console.error(error);
          } else {
            upd = {
              $set: {}
            };
            upd['$set']["versions." + version + ".meta.pipeFrom"] = cfdomain + '/' + filePath;
            upd['$set']["versions." + version + ".meta.pipePath"] = filePath;
            self.collection.update({
              _id: fileRef._id
            }, upd, function(error) {
              if (error) {
                console.error(error);
              } else {
                // Unlink original files from FS
                // after successful upload to AWS:S3
                self.unlink(self.collection.findOne(fileRef._id), version);
              }
            });
          }
        });
      });
    });
  },
  interceptDownload: function(http, fileRef, version) {
    var path, ref, ref1, ref2;
    path = (ref = fileRef.versions) != null ? (ref1 = ref[version]) != null ? (ref2 = ref1.meta) != null ? ref2.pipeFrom : void 0 : void 0 : void 0;
    if (path) {
      // If file is moved to S3
      // We will pipe request to S3
      // So, original link will stay always secure
      Request({
        url: path,
        headers: _.pick(http.request.headers, 'range', 'accept-language', 'accept', 'cache-control', 'pragma', 'connection', 'upgrade-insecure-requests', 'user-agent')
      }).pipe(http.response);
      return true;
    } else {
      // While file is not yet uploaded to S3
      // We will serve file from FS
      return false;
    }
  }
});

if (Meteor.isServer) {
  // Intercept File's collection remove method
  // to remove file from S3
  var _origRemove = Collections.files.remove;

  Collections.files.remove = function(search) {
    var cursor = this.collection.find(search);
    cursor.forEach(function(fileRef) {
      _.each(fileRef.versions, function(vRef) {
        var ref;
        if (vRef != null ? (ref = vRef.meta) != null ? ref.pipePath : void 0 : void 0) {
          client.deleteFile(vRef.meta.pipePath, function(error) {
            bound(function() {
              if (error) {
                console.error(error);
              }
            });
          });
        }
      });
    });
    // Call original method
    _origRemove.call(this, search);
  };
}

Of course I have inserted my S3 cfdomain, key, secret, bucket, region into the code. When I run the app and submit images it just upload to my server folder only: app\ .meteor\local\build\programs\server\assets\app\uploads\Images.
How can I upload my images to the S3? Can anyone show me a demo app with this S3 upload code please... Thank you very much.

Here is production code: https://github.com/VeliovGroup/Meteor-Files/blob/master/demo/lib/files.collection.coffee

@dr-dimitru  Thanks, but when I insert that code on the server it said: Exception in setInterval callback: ReferenceError: _app is not defined.
I am new to meteor so can you give me a simplest code to upload to the S3 bucket, please...
 Correct me if I am wrong:

User select a file.
It check the file-size, file-extension and upload to server folder 'server\assets\app\uploads\Images'. This process use the RAM and CPU of the client, right?
The server auto upload to the S3 cloudfront bucket? This process use the RAM and CPU of the server, right?

What is the code to auto upload file from server to the S3 cloudfront? I use react to render the client. I want the file is not uploaded until user click the upload button. So I use this code
formSubmit(e) {
  var upload = Images.insert({
    file: e.target.files[0],
    streams: 'dynamic',
    chunkSize: 'dynamic'
  }, false);

  upload.on('start', function () {
  });

  upload.on('end', function (error, fileObj) {
    if (error) {
      console.error('Error uploading', error.message);
      alert('Error during upload: ' + error);
    } else {
      alert('File "' + fileObj.name + '" successfully uploaded');
    }
  });

  upload.start();
}

This code uploaded the image to the server folder. But what is the code for auto uploading the image to the S3 cloudfront after that? Can you give me that code in javascript rather than coffescript please...
Thank you and sorry for my bad English.

You've asked for example of demo app, I gave it to you (sorry it's CoffeeScript)
S3 integration example is well tested, do you have any errors?
"What is the code to auto upload file from server to the S3 cloudfront" - See onAfterUpload in S3 integration example. This hook is responsible for upload to S3

I'll repeat it here:
  onAfterUpload: function(fileRef) {
    // In onAfterUpload callback we will move file to AWS:S3
    var self = this;
    _.each(fileRef.versions, function(vRef, version) {
      // We use Random.id() instead of real file's _id
      // to secure files from reverse engineering
      // As after viewing this code it will be easy
      // to get access to unlisted and protected files
      var filePath = "files/" + (Random.id()) + "-" + version + "." + fileRef.extension;
      client.putFile(vRef.path, filePath, function(error, res) {
        bound(function() {
          var upd;
          if (error) {
            console.error(error);
          } else {
            upd = {
              $set: {}
            };
            upd['$set']["versions." + version + ".meta.pipeFrom"] = cfdomain + '/' + filePath;
            upd['$set']["versions." + version + ".meta.pipePath"] = filePath;
            self.collection.update({
              _id: fileRef._id
            }, upd, function(error) {
              if (error) {
                console.error(error);
              } else {
                // Unlink original files from FS
                // after successful upload to AWS:S3
                self.unlink(self.collection.findOne(fileRef._id), version);
              }
            });
          }
        });
      });
    });
  }

@dr-dimitru Thank you, I have uploaded successfully :)
One more thing, after I uploaded image to server, how can I check the type, size, width and height of the image before upload it to the S3. I want to check the file on the server, not on the client because user can edit the code on the client.
And can you show me how to resize and create thumbnail after image was uploaded to the server? Can I upload that thumbnail to the S3 with the default image?.
Thank you very much.

The onBeforeUpload is isomorphic and runs on both Server and Client. Moreover it checks passed data for every received chunk, so you stay safe from upload interceptions.

For image resize example check out here and here, sorry for CoffeeScript

Hi @MinhTri66 any news on your end?

Hi @dr-dimitru  Sorry for late. I am trying to find a best way to work with images on the server of Meteor.
I have read the resize example CooffeeScript that you showed to me above. It uses the
CollectionFS/Meteor-cfs-graphicsmagick and read the image by the path  image = gm(fileRef.path) , but when I run it return error. . .

Exception in callback of async function: Error: cfs:graphicsmagick could not find "graphicsMagick" or "imageMagick"
I am trying to find another way to work with images in Meteor. Do you know any other packages for this? Tell me know please. . . Thank you.

Is there any reason why can not install imageMagick or graphicsMagick on your server?

How can I setup GraphicsMagick or imageMagick on my meteor server? I there a meteor package that I can install or I have to setup it follow the install guide on the web? http://www.graphicsmagick.org/INSTALL-windows.html#installing-using-installer-package
I am developing my app on my localhost, how can I install GraphicsMagick on my localhost...

Oh that is my mistake! I didn't think imageMagick or graphicsMagick is a sofware that I can setup it on the server. I thought it was a package of meteor or javascript that I must install with the cmd, haha
Thank you very much for your support and guidance for me.  Thank you again.
We can close the issue now.

@MinhTri66 glad you've solved it. Thank you for using this package 👍 . And do not hesitate to ask any other question.
Please, support this project by:

Star on GitHub
Stat on Atmosphere
Share on Facebook and Twitter
Donate via PayPal


doc: update doc for workkload identity setup

Signed-off-by: Binbin Li<EMAIL_ADDRESS>Description

Update outdated Azure Workload Identity section in oras-auth-provider.md
Fix charts template.
Update ratify-verify-azure-cmd.md. Latest Notation(v0.11.0-alpha.4) is incompatible with Raity of version older than v1.0.0-alpha.3

Fixes # (issue)
Type of change
Please delete options that are not relevant.

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[x] This change requires a documentation update

How Has This Been Tested?
Run the workload identity setup with AKS successfully.
Checklist:

[ ] Does the affected code have corresponding tests?
[ ] Are the changes documented, not just with inline documentation, but also with conceptual documentation such as an overview of a new feature, or task-based documentation like a tutorial? Consider if this change should be announced on your project blog.
[ ] Does this introduce breaking changes that would require an announcement or bumping the major version?
[ ] Do all new files have appropriate license header?

Looks good to me. Just a general note: I think azure workload identity now support using user assigned managed identity instead of just AAD application. We might want to consider switching to MI.

Good call! I'll create an issue to track it. For now, I think it's enough to unblock new users to try it.


Uploading custom steps not working

Running test with custom step does not work - just uploading gives the same result.
Steps to reproduce
following example
yaks test extension.feature -u steps/
2020/05/07 17:30:40 log.go:70: snap/installer "level"=0 "msg"="Snap is already installed: skipping"
2020/05/07 17:30:40 log.go:55: snap/java-deployer "level"=0 "msg"="Executing maven release phase on project steps/"
Error: error while creating deployment for source code: exit status 1

Unfortunately I don't know where to check for more logs (the operator pod doesn't show anything)

I think the path to the steps directory must be relative to the directory where yaks CLI tool is executed. AFAIK it can not be a relative path to the feature file given. Does that make a difference in your case?

In my case both are the same. Here the details (just upload, no test) with version 0.0.43
C:\app\b.vorberg\product\citrus\yaks\examples\extension> yaks upload steps/
2020/05/08 08:03:13 log.go:70: snap/installer "level"=0 "msg"="Snap is already installed: skipping"
2020/05/08 08:03:13 log.go:55: snap/java-deployer "level"=0 "msg"="Executing maven release phase on project steps/"
Error: error while creating deployment for source code: exit status 1

I also tried with absolute path. Could it be operating system related (My client runs on Windows)?

yes could be the OS. @nicolaferraro can you say something regarding OS support for Snap/Minio?

Tested with linux subsystem. Still same result:
boris@IRIX-8Y4X3H2:/mnt/c/app/b.vorberg/product/citrus/yaks/examples/extension$ /mnt/c/app/b.vorberg/product/yaks/linux/yaks upload steps/
2020/05/08 14:13:53 log.go:70: snap/installer "level"=0 "msg"="Snap is already installed: skipping"
2020/05/08 14:13:53 log.go:55: snap/java-deployer "level"=0 "msg"="Executing maven release phase on project steps/"
Error: error while creating deployment for source code: exit status 1

Does it work for you?

I think it's an OS problem on snap. I've opened an issue to fix it.


mypy thinks a config variable can never be None when nested in inherited BaseSettings class

Hello,
I just switched a project to pydantic v2 and encountered a bug with mypy thinking that some variable can never be None when the typing and the default value clearly indicate otherwise.
Here's a minimal code that reproduce the error:

import pydantic
import pydantic_settings

class SettingsNested(pydantic_settings.BaseSettings):
    my_var: str | None = pydantic.Field(default=None)

class Settings(SettingsNested):
    pass

m = Settings()
if m.my_var is None:
    print("hello there")  # mypy thinks we can never go inside this if

Here's the mypy error: test.py:14: error: Statement is unreachable  [unreachable]

Thanks @Greesb for reporting this 🙏
Could you please provide more information?

python version
pydantic version
pydantic-settings version
mypy version

@hramezani
python = 3.11.4
pydantic = 2.3.0
pydantic-settings = 2.0.3
mypy = 1.5.1

Thanks @Greesb  for the quick response.
Please provide you mypy settings as well.

[tool.mypy]
strict = true
warn_unreachable = true
exclude = [".venv"]
show_error_codes = true
plugins = ["pydantic.mypy"]

[[tool.mypy.overrides]]
module = [
       "deepdiff.*",
       "vcr.*",
       "voluptuous.*",
       "msgpack.*",
       "bs4.*",
       "graphql_utils",
       "markdownify",
       "asgi_lifespan",
       "authlib.*",
       "imia.*",
       "starsessions.*",
       "sqlalchemy_utils.*",
]
ignore_missing_imports = true

We have the same error with pyndatic.BaseModel. So, I've edited the example and replaced pydantic_settings.BaseSettings with pydantic.BaseModel and moved it to pydantic repo.


SheetsQuickstart  java.lang.NullPointerException

(Please fill out these details before submitting an issue)
SheetsQuickstart
(i.e. Gmail Quickstart)
Prints the names and majors of students in a sample spreadsheet
NullPointerException
Exception in thread "main" java.lang.NullPointerException
at java.io.Reader.(Unknown Source)
at java.io.InputStreamReader.(Unknown Source)
at ruhr.oracle.gclient.Auth.getCredentials(Auth.java:44)
at ruhr.oracle.gclient.Auth.main(Auth.java:68)
Specifications

Java version (java -version) : 1.8.0_181
OS (Mac/Linux/Windows) : Windows

Did you follow Step 2.b, and copy the credentials.jsonfile you downloaded into src/main/java src/main/resources?

I'm having this exact same issue, following the steps fully.

Are you using the gradle run command to run the same, or running it from your IDE?

I am running it from the IDE.

Please try running it using the gradle command. The credentials.json file needs to be added to the Java project as a resource, so alternatively ensure that it is in your IDE.


SQL query output as string and parameters.

For safety and to avoid SQL injection, it would seem beneficial to generate the SQL string as a string and all arguments as parameters.  Rather than just a single string.  This enables prepared statements where arguments are passed as placeholders with "?"
Example: For the following query

color:red AND NOT (type:"honey crisp" OR type:"granny smith") AND age_in_months:[5 TO *]

It would generate a condition like:
    (
		("color" = ?) AND
		(
			NOT(
				("type" = ?) OR
				("type" = ?)
			)
		)
	) AND
	("age_in_months" >= ?)

and a set of arguments
args = ["red", "honey crisp", "granny smith", 5]

That can be used as input to:
   db.Query("SELECT * FROM apples WHERE ...", args...)

As discussed here this provides additional security https://go.dev/doc/database/sql-injection that has been tested well.
It would probably also be good to validate the user-input fields (keys) and verify that they are intended to be accessible/filterable by the user.
Is it possible to achieve this with a custom driver possibly?

That is a good suggestion! More likely this could be a "mode" where you could render to parameterized input rather than a string. I'll chew on what that would look like and follow up here.

Ok I have a really rough implementation that works and has tests passing. It's pretty gross though so I am going to think of a way to clean it up. If you are curious you can see it at #23. I'll think about this some more and merge something in the next few days to add support here.
tl;dr is that during the render we build up a list of parameters and then replace literals with a ?. The tricky part that makes it gross is the type inference is lost for things like the range operator and general edge cases in the regular expression handling.

I just merged the PR so this is now exposed via the new lucene.ToParameterizedPostgres function. Thanks for filing the issue!


chore(main): release 0.4.49

:robot: I have created a release beep boop
0.4.49 (2024-04-10)
Continuous Integration

build: Fix fd install (55a856d)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/linrongbin16/cargo-ci-template.rs/releases/tag/v0.4.49 :sunflower:


Support for copy and paste

Copy and paste does not work currently. For example, when trying to paste a choreography task:
TypeError: self.canPaste is not a function[Learn More] ChoreoRules.js:213:11
	[38]</ChoreoRules.prototype.init/< ChoreoRules.js:213:11
	unwrapEvent/< CommandInterceptor.js:43:11
	<anonymous> self-hosted:978:17
	invokeFunction EventBus.js:480:9
	[172]</EventBus.prototype._invokeListener EventBus.js:356:18
	[172]</EventBus.prototype._invokeListeners EventBus.js:344:18
	[172]</EventBus.prototype.fire EventBus.js:304:18
	[167]</CommandStack.prototype._fire CommandStack.js:346:13
	[167]</CommandStack.prototype.canExecute CommandStack.js:178:15
	[278]</Rules.prototype.allowed Rules.js:41:14
	[201]</CopyPaste.prototype.paste CopyPaste.js:225:13
	[210]</EditorActions.prototype._registerDefaultActions/< EditorActions.js:101:6
	[210]</EditorActions.prototype.trigger EditorActions.js:173:9
	[221]</KeyboardBindings.prototype.registerBindings/< KeyboardBindings.js:107:6
	invokeFunction EventBus.js:480:9
	[172]</EventBus.prototype._invokeListener EventBus.js:356:18
	[172]</EventBus.prototype._invokeListeners EventBus.js:344:18
	[172]</EventBus.prototype.fire EventBus.js:304:18
	[220]</Keyboard.prototype._keyHandler Keyboard.js:94:19
	<anonymous> self-hosted:976:17

First of all we will have to add a canPaste function to ChoreoRules. Then BpmnCopyPaste.js which registers a function on 'element.paste' (element without 's', so every single element) looks like a candidate to overwrite/disable/subscribe.
It iteratively gets a context which contains some (should be possible to contain all) pasted elements. It could be possible to implement correct copy/clone and re-referencing here.

Closed in #81


Added support for 'enabled' parameter for monitors

I have several monitors what are used for debug purposes and I would like to default to disabled.  This isn't possible to do in the current implementation of the init() process because he streams aren't created when you call config.monitor() but rather after the init() function has returned.
See also zettajs/zetta-device#16

@kyork-cl That's for the PR. After discussing with the team we'd be happy to accept it if we can make a slight change to the option parameter.
Currently the second parameter is just a boolean. config.monitor('counter', false) we'd like to see this as an option object. Like so: config.monitor('counter', { disable: true }) this would allow us to expand on it later and follow a similar syntax to config.stream.

Sure, I'll update the PR

Went ahead and changed it to behave more like streams here and in the zetta-device PR.

Looks good to me.

We may want to change how we identify the disabled streams. In the PR you change the config.monitors to an object vs having it as an array.
If we accept the PR this will break any devices that are using previous version of zetta-device.

So it's not the cleanest solution but it would allow old devices with monitors to work and still support the options.disable.
If we leave this.monitors as an array and create a separate this.monitorsOptions object with the new options. https://github.com/zettajs/zetta-device/pull/16 would also need to be updated to the new format.
@kevinswiber @mdobson How much do we care about backwards compatibility for zetta devices at this time vs a nicer solution in the code? My vote would be for the former.
I think the bigger issue here is how zetta does the dependencies for zetta-scientist and zetta-device, zetta proper as well as zetta-scout both require and use zetta-scientist to init/config the devices using deice_config.js however what consumes the output of the device config is zetta-device which could be at an incompatible version with the output. The issue mainly comes into play with zetta proper requiring zetta-scientist as the device developer doesn't have control over the version zetta depends on.
A solution might be to have zetta-device either require or just contain the device_config.js so it's always at the correct version as when the device was built.

+1 to moving it into the zetta-device.  They are pretty tightly coupled together so this would prevent future changes from causing issues.

Closing this PR as i created #5 with backwards compatibility support. Going to table the device_config issue for now and create a change later. Filed an issue here on the main zetta project. https://github.com/zettajs/zetta/issues/290


perfect

App summary

App name: ?
App serial: ?
App version: ?

Vita3K summary

Version: v?
Build number: ?
Commit hash: https://github.com/vita3k/vita3k/commit/[commit]
CPU backend:
GPU backend:

Test environment summary

Tested by: ?
OS: Windows 10/macOS/Linux Distro, Kernel Version?
CPU: AMD/Intel?
GPU: AMD/NVIDIA/Intel?
RAM: ? GB

Issues

Screenshots

Log
Recommended labels

A?
B?
C?

invalid issue

What does any of this mean? You provided no details at all.


use with docker-compose

I have seen that you provide instructions to run kubefwd with docker. Is it also possible to use it with docker-compose and do you have an example?

Hey @jeff1985 . I had the same question, so I started playing around with it. Here is how I currently use it.
version: "2"
services:
  kubefwd:
    image: txn2/kubefwd
    volumes:
      - /Users/<username>/.kube/:/root/.kube/
    command: "svc -n default"
    privileged: true
  nginx:
    image: nginx:1.9
    ports:
      - "80:80"
    links:
      - kubefwd:php
    volumes_from:
      - php
    volumes:
      - ./default.conf:/etc/nginx/conf.d/default.conf
  php:
    image: php:latest
    network_mode: service:kubefwd
    volumes:
      - ./:/var/www/html


Failed for Archive.

**

/AFNetworking.framework/AFNetworking was built without full bitcode.

**
Xcode version 10.1 and using swift.
While running the app in the device its perfectly works but while archiving I am getting this error.
I using the AFNetworking via PaymentSDK framework.
I am adding the PaymentSDK for another swift framework.

Same issue

AFNetworking' does not contain bitcode. You must rebuild it with bitcode enabled (Xcode setting ENABLE_BITCODE), obtain an updated library from the vendor, or disable bitcode for this target.

Workaround : adding 'BITCODE_GENERATION_MODE' as key and value as 'bitcode' in user define build setting fixed the issue.


Fixed SV annotation issues and improved SV report

The frequency annotation bugged because inversions are somtimes missing the SVLEN field in the SVIM VCF. I updated sveval to use the END field in these cases.
Also debugged the report that had a problem when nothing was found in the genes of interest and split the two gene names for gene fusions.

@jmonlong Added you and @hbeale as collaborators - you're driving this as much (more) then I so feel free to just make commits directly vs. pull request - but totally up to you (or pull request and then approve if you'd like to keep that discipline)


Typo in workshop 17

Correct Openinig to Opening in https://virtual.acl2020.org/workshop_W17.html

Sent a PR in the data repo.
Ref: https://github.com/acl-org/acl-2020-virtual-conference-sitedata/pull/223