Performance issues

I've been using turndown in enex-dump for converting Evernote's notes to markdown. I've tried to optimize it's performance, but the main bottleneck comes from turndown, here you can check the actual function I'm using, I've tried removing all my custom regexes and extra rules and they barely make any difference.
Part, if not most, of the bloat I think comes from jsdom, which turndown imports. Just because of jsdom I'm seeing ~10MB of dependencies getting installed, which is definitely not good for my use case where I'm embeddeding enex-dump in an Electron application. Also jsdom provides an implementation of the DOM, which I'm not sure a project like turndown really needs, shouldn't a fast xml parser like fast-xml-parser get the job done with significantly less bloat?
Thanks.

One of the main benefits of using the DOM is that you get HTML parsing for free in browsers:
https://github.com/domchristie/turndown/blob/e2c64f121a170ad04363c7da3bbf978ebcde5f7f/src/html-parser.js#L41-L47
We also use methods defined on DOM nodes for querying. This makes it very easy for developers to extend the rules because the DOM Node/Element interface is familiar. Of course we could re-implement all these methods if we were to parse HTML as JSON, but that seems unnecessary given the above, and we'd lose the familiarity.
I'm not very familiar with Electron, so this could be wrong, but if it effectively just renders HTML to a webview, then it should be possible to use the built in parser, rather than importing JSDOM. For this you should import the browser version from lib/turndown.browser.es.js.
Hope that helps.

I don't understand the title change, enex-dump is a standalone application and has nothing to do with Electron, turndown is not bloated and slow only when used in Electron applications.

Sorry if I misunderstood. You mentioned your use case was "embeddeding enex-dump in an Electron application" so I thought that could be where the problem lies. I carried out some brief performance tests after launching v4, and it seemed to be significantly faster than other similar libraries. These tests were performed using the CommonJS build on a Node.js server. As I mentioned, I think the underlying issue is that you're using the non-browser build, and therefore embedding JSDOM within a browser, which is not its intended use :/ We may need to look into updating a few package.json fields in order to prevent this happening so much.

I get your point of view, after all the maintainers are the ones putting the actual work into this, and the project should be something they feel comfortable with. I just wanted to express a bit of frustration since the cause of slowness/bloat for my use case comes from this library.

I'm open to pull requests and feedback. You are obviously more than welcome to explore the source code, replace JSDOM with a parser of your choosing, or suggest improvements. However, the issues aren't really a place to vent frustration in this way.

Interesting, maybe I could instruct webpack to load the es version for browsers instead.

👍

it's generally better to do the heavy work on a separate (non-renderer, I guess) process

Technically true, but I'd recommend trying it and seeing how it feels. If you're converting a significant amount of HTML then you may want to perform this on the server.

Sorry if I misunderstood. You mentioned your use case was "embeddeding enex-dump in an Electron application" so I thought that could be where the problem lies.

My bad, I should have said more explicitly that I basically have 2 use cases for turndown: 1) using it in enex-dump 2) using it indirectly in an Electron app that uses enex-dump.

I carried out some brief performance tests after launching v4, and it seemed to be significantly faster than other similar libraries.

I don't doubt that, but in my tests turndown needs about 80ms to process a particular html string, which I can convert to json in about 2ms with fast-xml-parser. I'm kind of comparing apples to oranges here, but should converting the document to markdown take ~40x as much as parsing it? Maybe not, I think there's quite some room for improvement here.

As I mentioned, I think the underlying issue is that you're using the non-browser build, and therefore embedding JSDOM within a browser, which is not its intended use :/

I think we should consider my 2 use cases separately for this:

Standalone: I don't think the ~10mb of dependencies matter much in this scenario, the bad performance is still there though.

Embedded in Electron: ~10mb of extra dependencies are definitely important in this scenario, and if turndown is slow because it isn't using the native parser it's a problem also. I don't think I'm doing anything "unintended" here though, just importing turndown and using it.

However, the issues aren't really a place to vent frustration in this way.

I'm sorry, but I've opened this issue because turndown requires ~10mb of dependencies and because I think it's not very performant, aren't these good enough reasons for opening an issue? The fact that this frustrates me is irrelevant.

Technically true, but I'd recommend trying it and seeing how it feels. If you're converting a significant amount of HTML then you may want to perform this on the server.

The UI blocks for multiple minutes when converting a large enough number of html strings (100s to 1000s of html strings), on the renderer process. I'll definitely try to run this on another process, hoping to have access to the browser's native parser there.

I'm open to pull requests and feedback. You are obviously more than welcome to explore the source code, replace JSDOM with a parser of your choosing, or suggest improvements.

Unfortunately I don't have the time to make major changes (switching from a full-blown DOM implementation to an xml parser) to this unfamiliar codebase, but I think there might be a few things I can help with:

I have to check if within Electron turndown is still using jsdom, if that's the case it should be fixed.
jsdom should be loaded conditionally, in such a way that it's import can be removed entirely via dead code elimination.

I've opened this issue because turndown requires ~10mb of dependencies and because I think it's not very performant, aren't these good enough reasons for opening an issue? The fact that this frustrates me is irrelevant.

Opening an issue to discuss performance issues is fine. Stating that "Turndown is slow and bloated" probably isn't the best way to start one.
I try as much as possible to reduce the number of dependencies; dealing with frequent updates, incompatibilities and conflicts is never fun! It's why Turndown aims to use the simplest set up possible: virtually no transpilation—just ES6 modules compiled with Rollup using npm scripts. JSDOM was chosen early on because it is reliable and well maintained. It offers a solid implementation of the DOM allowing for Turndown rules to use familiar node/element properties, and for them to be shared across different environments: Node and browser. If there is a library that offers similar levels of reliability, with a nice API, that is well maintained, then I'd happily switch to it.

I don't think I'm doing anything "unintended" here though, just importing turndown and using it.

This is a key point. In a Node.js environment, the a node version (with JSDOM) will probably be used. For building JS for electron apps, this is obviously not what we want. There is some discussion around this issue (https://github.com/defunctzombie/package-browser-field-spec) but I've not looked into its status.
One thing to note re: enex-dump, you are using an old version of Turndown. Turndown v5 updated the escaping to be more aggressive, but almost certainly more efficient.

jsdom should be loaded conditionally, in such a way that it's import can be removed entirely via dead code elimination.

I believe this is already the case:
https://github.com/domchristie/turndown/blob/e2c64f121a170ad04363c7da3bbf978ebcde5f7f/src/html-parser.js#L27-L56
process.browser determines whether JSDOM is required or not. This is set in config: https://github.com/domchristie/turndown/tree/master/config

Opening an issue to discuss performance issues is fine. Stating that "Turndown is slow and bloated" probably isn't the best way to start one.

That was just a descriptive short title for the content of my message, I didn't mean to offend anyone with it, sorry if it did.

If there is a library that offers similar levels of reliability, with a nice API, that is well maintained, then I'd happily switch to it.

If you're looking for another DOM implementation I'm not sure there are significantly better ones. My argument is that using a XML parser instead would be better for performance, perhaps at the cost of (slightly?) larger bundles for browsers.

One thing to note re: enex-dump, you are using an old version of Turndown. Turndown v5 updated the escaping to be more aggressive, but almost certainly more efficient.

Cool, I didn't realize another version was out. I just benchmarked it and it seems the performance improvement is about 0.05% for me.

jsdom should be loaded conditionally, in such a way that it's import can be removed entirely via dead code elimination.
I believe this is already the case:

Kind of, you're completely omitting the import from the bundle, but if the right bundle isn't loaded (which seems to be the case for me) that doesn't matter. Maybe the browser property in package.json can solve that, I'm not very familiar with it 🤔

Those are a lot of bundles.
I did some tests and:

I couldn't find a way of instructing webpack to load the proper file, the following configuration changes nothing for me (the $ at the end is supposed to tell webpack "match only this and not turndown/foo for instance, but I also tried omitting it, same thing):

  resolve: {
    alias: {
      turndown$: require ( 'path' ).join ( __dirname, 'node_modules', 'turndown', 'lib', 'turndown.browser.cjs.js' )
    }
  }

If I patch package.json manually and force it to load the right file then the performance difference isn't really noticeable, so I guess jsdom's implementation of the DOM isn't significantly slower than chrome's, which would be quite surprising to me, or the cause of the slowness is elsewhere in the codebase.

By default the packaged electron app weighs 162.6mb (58.4mb zipped), if I manually remove the jsdom dependency from package.json then it weighs 154.6mb (56.9mb zipped). Notice how by making the main field point to the right file jsdom is still included in the bundled, I had to manually remove the dependency from package.json. Maybe webpack expects a "sideEffects": false in package.json, or it just doesn't perform this optimization for non-es2015 imports/exports, I'm not sure, here there are some details.

I see turndown already has a browser property in its package.json:
https://github.com/domchristie/turndown/blob/e2c64f121a170ad04363c7da3bbf978ebcde5f7f/package.json#L9-L11
which if my understanding is correct in this particular case should instruct the bundler not to include "jsdom" along with the bundle when bundling for the browser. But I'm not sure how webpack handles this, maybe it uses it if the target is web but not if it is electron-renderer.
Both turndown and jsdom weren't considered external dependencies during my tests.
I'll try to profile turndown and check if there's any low hanging fruit with regard to performance.

Thanks for taking the time to investigate this further!

Those are a lot of bundles.

The state of JS 😩
Would it be possible to do something like the following in your source:
require('turndown/lib/turndown.browser.es.js')

Perhaps something like this approach might be worth looking into: https://blog.patrickhulce.com/2016/09/15/exposing-multiple-exports-in-a-single-npm-package/?

If I patch package.json manually and force it to load the right file then the performance difference isn't really noticeable, so I guess jsdom's implementation of the DOM isn't significantly slower than chrome's, which would be quite surprising to me, or the cause of the slowness is elsewhere in the codebase.

Interesting. Previous performance issues were due to slow-running regular expressions (https://github.com/domchristie/turndown/issues/188#issuecomment-332216019). These issues should have been resolved by now, but it might be a good starting point?

I'm interested in the minidom branch. I see mention of it passing all tests so I assume it's stable enough. Is it compatible with the gfm plugin? Would you consider publishing this as a 'turndown-lite' version?

Another idea is just remove jsdom altogether and make the turndown service accept an optional html parser to replace a native parser. Of course when using in nodejs this would be required but I think anyone coming from a node background would be comfortable passing the dependency in.

I'm interested in the minidom branch. I see mention of it passing all tests so I assume it's stable enough. … Would you consider publishing this as a 'turndown-lite' version?

The minidom branch is experimental, so I'd say it's reasonably stable :) Unfortunately minidom uses an outdated version of parse5, and hasn't been updated in over four years. I'm also wary of releasing a "lite" version as this would require additional maintenance, and might be confusing to those choosing which one to use.

Is it compatible with the gfm plugin?

No, not yet. It'd have to be updated to use the set of DOM methods that minidom supports.

Another idea is just remove jsdom altogether and make the turndown service accept an optional html parser to replace a native parser.

Yes, this is something I'll definitely consider. Thanks!

I made a minidom compatible gfm branch if anyone's interested. https://github.com/domchristie/turndown-plugin-gfm/issues/14

I'm finding about 700ms require time improvement with turndown minidom

@domchristie I really believe the next step should be to remove any built-in parsers from the library and add a "parser" option instead, as I mentioned before, so that I can use the built-in parser when the library is running inside a browser environment and provide my own parser (minidom, jsdom, whatever) when running in a node.js environment.
I'm going to hack something like this starting from your minidom branch, but it'd be great if this was an official package, like perhaps it could be published as turndown-core and then imported by turndown proper or something?

perhaps it could be published as turndown-core and then imported by turndown proper or something?

Yes. My current thinking is along these lines:

This repo would contain both turndown-core and turndown packages to avoid confusion as to where to log issues, and to save me maintaining two repos. (Most people will be using turndown, but bugs will likely be due to code in turndown-core.)
turndown-core will only support parsed HTML as input but will include an option to hook into the conversation method (e.g. before). The result of this before hook will be converted to Markdown, which will allow for custom parsers
The turndown package will import the turndown-core and add a before option to parse with JSDOM

The turndown implementation might look something like:
import { Service } from 'turndown-core'
import parse from './html-parser'

default function TurndownService (options) {
  options.before = parse
  Service.call(this, options)
}

Let me know what you think.

Is this still being considered? The proposed updates sound great!

FYI - For those who need fast speed in markdown, I ended up writing a heavily optimized compiler built in TS for max speed centered around node (though it has browser compatibility as well).
The output format is visually very clean, also. It does some unique things like automatically numbering multi-level lists, not allowing excess running linefeeds, etc.
We're able to tear through gigabytes very quickly. I believe avg. file processing time is about ~4ms.
It's up and stable: node-html-markdown
We have a few enhancement issues open to round out some things, so if anyone is interested, I'd love to have a few extra hands!
PS. My intent isn't to denigrate this library or spam. turndown is a great library! This is simply to present an alternative for those who need extreme perf in a node environment.

FYI - For those who need fast speed in markdown, I ended up writing a heavily optimized compiler built in TS for max speed centered around node (though it has browser compatibility as well).

How does it compare to Turndown with node-html-markdown used as HTML parser? We were converting gigabytes of HTMLs too and the real bottleneck was actually in NodeJS startup, when used in "one CLI command per file" mode. The conversions were needed by a non-JS app, so we wrapped Turndown in a microservice in Node and it was lightning fast. The second bottleneck would have been probably JSDOM (now replaced with domino), but even with JSDOM it was faster than the database I/O.
I wonder what the conceptual difference is. Turndown is a library with a fixed HTML parser dependency. node-html-markdown is a library with a fixed HTML parser dependency too. And this opinionated dependency actually became the issue that @domchristie wants to address in the end. In practical terms, a user with their own DOM parser implementation would suffer from an unused dependency. node-html-markdown has apparently the same issue and it has just picked another HTML parser for Node. Or am I missing something?
Indeed, we can expect some performance penalty for the Turndown's flexible rule system, but I doubt it can make significant difference. Looking forward to some benchmarks.
Fingers crossed for your project anyway, it's good to have some competitors. :)

@martincizek I actually wrapped it up before the move to domino. Looks like you guys have made good strides in improving! Nicely done!

I wonder what the conceptual difference is.

My goal was two-fold...
First, I wanted to see how much performance I could get out of node by optimizing the code. That means things like:

Perf testing and compiling regex
Carefully considering every line of code, every operator, every function call, etc.
Minimize the amount of CPU operations performed, taking the shortest necessary path for each node
Using a shared output buffer and optimizing writes called to it
Other JS-related perf tips (using object access vs. map, switch, or conditional chains, leading decrement operators, etc)

(That said, there are likely ways to further improve it down the line!)
The second goal was to produce a human readable format. Due to the way we're doing our output buffer writes, we were able to do some very nice output improvements such as:

Proper indentation
Automatic list numbering
Produce a cleaner output in terms of consistent linefeeds

This makes for a very human-readable format without needing a markdown viewer.
We now have a database full of very readable, much smaller content! ☺

This opinionated dependency actually became the issue that @domchristie wants to address in the end. In practical terms, a user with their own DOM parser implementation would suffer from an unused dependency.

The big issue for me lines up with the title of this issue - namely, performance!
But I get where you're coming from. I would certainly be open to making the dependency swappable. In my perspective, though, given the benchmarks, I wouldn't see much benefit as the one we use seems to outperform the others and it also has a very small footprint.
Flattened out, it's simply 2 dependencies (one being he, which I believe we'd need anyway).
The total decompressed size is 57.3 kB, so I'm not overly worried about the extra dependency. Though I do plan to add a rollup version for the browser which omits it.
The only real benefit I can see to allowing a custom supplied parser would be maybe to do something like have rust handle the parsing. Though, considering that this parser averages 2ms, I'm not sure how much difference that would make.
Would be curious to hear if you have other thoughts on this, though!

Looking forward to some benchmarks.

Just wrapped up the benchmarking tool. It tends to center around a ~29% speed improvement, which is significant for our use.
Example:
node-html-makrdown (reused instance): 47.2261 ms/file ± 29.1069 (avg bytes/sec: 2.086)
node-html-markdown                  : 55.2654 ms/file ± 38.2978 (avg bytes/sec: 1.782)
turndown (reused instance)          : 66.8692 ms/file ± 30.9974 (avg bytes/sec: 1.473)
turndown                            : 68.9757 ms/file ± 35.0430 (avg bytes/sec: 1.428)

Comparison to fastest (node-html-makrdown (reused instance)):
  node-html-markdown: -14.55%
  turndown (reused instance): -29.38%
  turndown: -31.53%

Estimated processing times (fastest to slowest):

  [node-html-makrdown (reused instance)]
    100 kB:  0.05sec
    1 MB:    0.50sec
    50 MB:   25.14sec
    1 GB:    8min, 34.80sec
    50 GB:   7hr, 8min, 59.91sec

  [turndown (reused instance)]
    100 kB:  0.07sec
    1 MB:    0.71sec
    50 MB:   35.59sec
    1 GB:    12min, 8.92sec
    50 GB:   10hr, 7min, 26.09sec

Nice tests. It really seems that the performance is getting close when the parser performance is getting closer. But even if the difference was 30% with the same parser engine, it'd be "roughly the same" for me and other criteria will decide then. When we were doing our big migration project, the backtranslation stability was the most important -  i.e. no data and formatting loss, the output renders back to semantically equivalent HTML. The second was to have the output very similar to how a human user would write it.
So the other claims are now quite interesting for me:

Proper (unbroken) indentation in multi-line list items

Any example? The indentation in Turndown conforms to the CommonMark spec and its recommendation of having enough indenting keep the lists items. Although we've overriden the rule, as our users use minimalistic indenting. But minimalistic indenting has its drawbacks - the output may be reindented when a new item is inserted in an ordered list (the number can change from 9 to 10 and then a sublist will be indentet one more space, which can cause false diff if the output is version-tracked).

Automatic list numbering

This just works, even with sublist and list offset is respected too. What is the issue?

A clean output in terms of consistent linefeeds

Do you mean CR vs LF or creating softbreaks to keep the line length under e.g. 80 columns? The former is a quite a minor thing, the latter is quite difficult to achieve without eventually breaking the output.

Looks like I was tired last night. I goofed on my math. In terms of speed, the figure was actually 76% improvement in speed.

the output renders back to semantically equivalent HTML

Definitely important. Ours is reversible as well.

Any example? (indentation)

It's been a few months since I wrote this, during that time I tried several different libraries and found numerous gotchas in the various libraries out there. Among them were numerous indentation issues on multi-level lists and multi-line LI/OL items not doing proper leading indentation for new lines, which broke the lists.

This just works, even with sublist and list offset is respected too. What is the issue?

Most libraries I saw did not actually output a real number with ordered lists. Not sure on TD's status

Do you mean CR vs LF

In many libraries, repeating LFs was out of control. I can say that specifically for turndown, but that was using the cheerio branch which was broken in many other areas, so perhaps that isn't an issue. I recall having documents with 8+ blank lines
What I mean by a clean output is that because we use a shared output buffer, we track linefeeds to produce a 'smart' result. So, for example, say we have two block elements in a row, we don't want a bunch of extra whitespace between them. This library has the ability to determine line separation between specific elements, and is generally mindful about readability in the output. (not sacrificing reversibility)

even if the difference was 30% with the same parser engine, it'd be "roughly the same"

I'm not sure I understand you here. If you mean that your assume the primary difference is in the parser, I haven't tested domino, but I doubt that it's much of a factor. It's been awhile since I went through td source, but believe it or not, small things like regex make a huge difference.
In fact, I decided to drill down a little deeper last night into the individual sections of code, because the numbers weren't what I expected, given the considerable effort into JS optimization. There was one tiny function which calculated trailing whitespace using a regex. I replaced it with a proper for loop, and here are the new numbers:
Speed increase: 204%
-------------------------------------------------------------------------------

node-html-makrdown (reused instance): 22.8886 ms/file ± 12.7700 (4.1 MB/s)
turndown                            : 69.7362 ms/file ± 32.0228 (1.35 MB/s)

-------------------------------------------------------------------------------

Estimated processing times (fastest to slowest):

  [node-html-makrdown (reused instance)]
    100 kB:  24ms
    1 MB:    244ms
    50 MB:   12.18sec
    1 GB:    4min, 10sec
    50 GB:   3hr, 27min, 55sec

  [turndown (reused instance)]
    100 kB:  67ms
    1 MB:    689ms
    50 MB:   34.43sec
    1 GB:    11min, 45sec
    50 GB:   9hr, 47min, 34sec

-------------------------------------------------------------------------------

This is far more reflective of what I would expect, although I believe by re-writing the parser, I could probably get it down closer to 14ms

Alright, thanks for your insights! And please don't get me wrong, I indeed want to examine every claim that seems to be related to Turndown.


bug & fix: ion-split-pane not allow click in ion-menu (Vue.js)

Bug Report
Ionic version:
[x] 4.x
Current behavior:
When you use ion-split-pane with ion-menu inside can't click on elements and click on next layer instead ion-menu layer.
Expected behavior:
Element click as expected
Steps to reproduce:
Try to use ion-split-pane with ion-menu normally.
Related code:
<ion-split-pane class="indexTopMenu" id="main-content">
      <menu-lateral />
    </ion-split-pane>

MenuLateral.vue:
<template>
  <ion-menu class="menuOptions" type="overlay" side="start" content-id="main-content">
    <ion-content padding>
      /* Insert what you want here as button or anything else */
    </ion-content>
    </ion-menu>
</template>

Other information:
How to fix:
CSS:
.menu-content-open {
pointer-events: unset !important;
}
If z-index show behind main layer use this option:
.menu-content.menu-content-open {
z-index: 20;
}
Ionic info:

Ionic:

   Ionic CLI       : 5.2.7 (C:\Users\Pablo\AppData\Roaming\npm\node_modules\ionic)
   Ionic Framework : @ionic/vue 0.0.9

Vue:
    @ionic/vue: 0.0.9 => 0.0.9

Utility:

   cordova-res : 0.6.0
   native-run  : 0.2.8

System:

   NodeJS : v12.8.0 (C:\Program Files\nodejs\node.exe)
   npm    : 6.10.2
   OS     : Windows 10

Thanks for the issue. Can you provide a repo with the code required to reproduce this issue?

In Ionic React same happens with using just IonMenu and IonItem with button attributes. Buttons work if using split pane and screen wide enough so it wont overlap the page (changing type attribute in menu to other than "overlap" just breaks the whole menu so it cannot be even opened).

Do you know if this happens in Ionic Angular?

If You directed that question towards me: sorry no I don't if this happens with Ionic angular.
This was my first and rather disappointing venture with Ionic. Of course it could be I'm just using it wrong but does not explain why it works when the menu does not overlap the page. Or why changing the type attribute on menu breaks it.

Are you able to provide a reproduction of the issue?

I tried to reproduce again.
z-index fail as expected (you can see that menu is behind main layer) but i dont understand why is not loading CSS .menu.content-open in the example and load in my project with bug. I will check tomorrow.
Here you have github with z-index problem.
https://github.com/PabloGBarcelo/vue-fix-ionic/tree/master/test-ionic
Thanks!

As I suspected I was using menu wrong; checked the side menu example and my issue was caused by a wrong contentId on the menu. So You can disregard my comments related to this, thank you :)

@PabloGBarcelo Thanks for the follow up. I took a look at your repo, and it looks like this bug is occurring because you are using ion-split-pane incorrectly. I changed the following things, and the app is now working correctly:

ion-menu-controller does not need content-id.
You need to wrap your main content in a div and assign it the ID main-content.
ion-split-pane needs to have content-id, not id.
Your main content should be inside of the ion-split-pane component.

The new template should look like:
<template>
  <ion-app>
    <ion-menu-controller></ion-menu-controller>

    <ion-split-pane content-id="main-content">
      <ion-menu class="menuOptions" type="overlay" side="start" content-id="main-content">
        <ion-content class="fullWidth" padding>
          <button @click="close">close</button>
        </ion-content>

footer

      </ion-menu>

        <ion-button @click="openStart">Open Menu</ion-button>
        <list-component></list-component>

    </ion-split-pane>
  </ion-app>
</template>

You will likely need to add width: 100% to your main content as well. For further help implementing your app, I recommend looking at the following resources:
ion-split-pane Documentation
Ionic Forums
Ionic Slack
I am going to close this issue. Thanks!

Thanks for resolve my problem.
Vue documentation its a bit confusing in this component, can you add this example (with ion-split-pane) for people if its possible.
I will update my repo with right way.
Thanks for your time!


Performance on Pi Zero Isn't Great

On the Pi Zero, it's possible to hit 100% CPU usage during faster ramps.
The culprit here is two-fold:

apscheduler adds a lot of overhead to the work being done. This becomes very noticeable on the Pi Zero, when compared to the Pi 3.
The update frequency caps out at 100 times a second. This is a bit excessive, and not really needed.

These combine to create a pretty poor situation for the Pi Zero.

First change I made is to cap the update frequency to 24 times a second. This brings the CPU load down to about 28% on the Pi Zero. That means at 100 times a second, we were trying to use nearly 120% CPU.
I've broken down and did the work to remove the dependency on apscheduler. It's disappointing, but for the best.
After making that change, CPU load peaks out at about 13-14%. At least on paper, I could do 10 ms intervals again at under 60% CPU load, but this still seems excessive.
Once I've finished testing the new scheduler, and checked it in, I'll close this issue.


AboutStrings. Add step about  C# 6 string interpolation.

It would be nice to add a step about string interpolation.
Something like
var firstName = "Matt"; var lastName = "Groves"; var str = $"My name is {firstName} {lastName}";
May I do a PR please?

Sorry I just snoticed this is mplemented as teh last step.


Send Wallet / Vault loader

Sending is taking more time than usual need a loader like we have in Tribe.

https://xd.adobe.com/view/489bcd8a-f669-4ecb-9d11-bd705d75a996-1dc8/screen/8767194b-6ff6-412e-a1f3-0bd8f1ff91af/

@Deveshshankar Also works fine in Vault.

Verified this issue on keeper app v 1.1.8(271)


Improve flash speed by skipping bootloader and partition tables when possible

See: https://github.com/esp-rs/esp-hal/issues/803#issuecomment-1727958673
But essentially, we could make espflash smart enough to flash the things that have changed, which is usually just the main app.

If it is possible to read the hash of a particular flash region, it should also be possible I believe to implement selective reflashing: for each flash block, read the hash, compare, and only reflash if differs.
Also, cc https://github.com/esp-rs/espflash/issues/259 as this issue is basically the meat of flash content verification.

There is   ESP_FLASH_VERIFY_MD5 command ( https://github.com/espressif/esptool/blob/8538b8983e5e04ad5d6f63620d609bfa63412eb3/flasher_stub/include/stub_flasher.h#L53 )
I think it's not yet supported by the Rust esp-flasherstub but by the esp-tool stub and in ROM (not for ESP-8266)

Started to look into ESP_FLASH_VERIFY_MD5 ... as mentioned, we don't have it supported by the Rust esp-flasherstub which is not that good
Tried without a flasher stub and it needs quite a few changes to espflash internals because of the way we currently parse responses into CommandResponse (and how we currently cannot return something that is not u32)
Additionally, there are at least two different response formats for that command


Feat/enable multi gpu

This PR does the following:

Sets up the package for multi-GPU benchmarking, where only the main process is logging. A guide to how to run scandeval with multiple GPUs is included in the readme. Basically, you simply replace scandeval with accelerate launch -m scandeval when benchmarking. This is possible due to the added __main__.py module. If you want to benchmark very large models it is beneficial to also use deepspeed - simply pip install deepspeed and call accelerate launch --use_deepspeed --zero_rank 3 -m scandeval instead of scandeval in this case.
When not in a distributed setting, only the first GPU will be used. This is mentioned to the user, so that they're able to swap to the accelerate setup instead, if needed.
Uses 8bit AdamW when a GPU is available, to save memory
Previously, when CUDA OOM errors occured, the batch size would be reduced but the gradient accumulation was not increased. This is because auto_find_batch_size was used. Instead, this is done manually now.

This closes https://github.com/saattrupdan/ScandEval/issues/37

@peter-sk noted that the multi-GPU functionality doesn't yet work for decoder-models, and he's correct! Turned the PR into a draft and will look into this.


Programming Guide Nomenclature Updates

Update the nomenclature in the Programming Guide to reflect the upcoming changes:
Game Manual Part 1 & 2 -> Competition Manual
Driver Station -> Operator Console
Driver Control -> TELEOP
Autonomous -> AUTO

@rachmo can you review the "File changed" to see if the use of Operator Console is correct.  I think we may need a clearer example of what the Operator Console is and how it is used when describing the Android devices.


Sample: Build and Publish from a multi-project solution to Azure Artifact Feed

For anyone going through the doc https://docs.microsoft.com/en-us/azure/devops/pipelines/targets/nuget?view=vsts&tabs=yaml
and finding it hard to figure out how to do the above, maybe the following sample would prove helpful:
Premise:
You have your Package configuration setup in the csproj (through Visual Studio, for example) and do not need to separately execute nuget pack - The reason you may want to do this is if these packages are part of a larger build and you don't need to increment each package version on every build. This way you can decide to increment locally and let VSTS deploy packages only if the version had been upgraded by you explicitly in the source code.
# ASP.NET Core
# Build and test ASP.NET Core projects targeting .NET Core.
# Add steps that run tests, create a NuGet package, deploy, and more:
# https://docs.microsoft.com/azure/devops/pipelines/languages/dotnet-core

variables:
  buildConfiguration: 'Release'

steps:
- script: dotnet build --configuration $(buildConfiguration)
  displayName: 'dotnet build $(buildConfiguration)'
- task: NuGetCommand@2
  inputs:
    command: push
    packagesToPush: '$(Build.SourcesDirectory)/**/*.nupkg;!$(Build.SourcesDirectory)/**/*.symbols.nupkg'
    publishVstsFeed: 'your-nuget-feed-name'
    allowPackageConflicts: true

Things to note:

Using allowPackageConflicts:true only works on Windows. Therefore, I had to use the default hosted Visual Studio pool (rather than a Linux VM).
packagesToPush has to be set because, the MSBUILD configuration (i.e. the csproj settings), does not push the packages to Build.ArtifactStagingDirectory which is the default.

Probably useful for anyone viewing the following issues:
https://github.com/MicrosoftDocs/vsts-docs/issues/2162
https://github.com/MicrosoftDocs/vsts-docs/issues/653

Thanks for taking the time to write this up and share!


[SUGGESTION] Console command

To make it so that people do not have to download a client-side, pull a command from the console. The game allows for binds to be created. So a simple !pprise in the console would compulsively count.
Then people only have to do something along the lines of   'console.bond Shift+KeypadPlus !pprise'
Along for people to be able to use the mod without having to install the addon client side.

That is a really good idea! Thank you for the suggestion.

The in-game console is sadly client-side - so there is no way to implement this. I hope this will change in the future as it would enable a lot of cool stuff for vanilla clients.
Next update will include a chat command for quick stacking (defined in in config, default "!qs"). This is the best I can do for now...


Generate MockMvc tests

currently, from the contracts, we generate JUnit or Spock tests that use RestAssured. It would be great to start generating MockMvc tests too. That way we can leverage RestDocs too, and we could generate all sorts of documentation out of the box.

I'll start checking this/working on this in order to fix/close:
#613 - Generate rest docs from contracts
#617 - Generating http-based client from contract

We've decided to pick another flow. The ultimate goal was to use RestDocs. To do that initially we thought that MockMvc tests will need to be generated. That's actually not the case since it's enough to  use the RestAssured & RestDocs integration. You can check the docs for more info https://cloud.spring.io/spring-cloud-contract/single/spring-cloud-contract.html#_generating_spring_rest_docs_snippets_from_the_contracts


store cached translations

Pull request fixing a bug in this issue https://github.com/argosopentech/argos-translate/issues/337
Created a global storage array of CachedTranslation objects by unique keys.
I tried to make as few changes as possible and as safe as possible. Perhaps this approach is not the best in terms of code style and in the future it may be refactored, but for now the main thing is that it completely eliminates a huge part of the memory leak due to the fact that objects are not created each time new ones, but once created from memory are taken.

Video demonsrtation https://youtu.be/UoMVIKUTWJ0

Wow! Thanks for submitting this
I had heard reports that there might be a memory leak in LibreTranslate but had no idea where.
https://community.libretranslate.com/t/libretranslate-under-load/482
https://community.libretranslate.com/t/memory-leak-in-argos-translate/638


Long inference time for single prompt (2s for encoding)

Here is my code, just adding the time check on the demo code. It seems the encoding process is far more time-consuming than expected.
from mobile_sam import sam_model_registry, SamAutomaticMaskGenerator, SamPredictor
import torch
import cv2
import numpy as np
import time

model_type = "vit_t"
sam_checkpoint = "./weights/mobile_sam.pt"

device = "cuda:1" if torch.cuda.is_available() else "cpu"
print(device)

mobile_sam = sam_model_registry[model_type](checkpoint=sam_checkpoint)
mobile_sam.to(device=device)
mobile_sam.eval()

predictor = SamPredictor(mobile_sam)

image = cv2.imread('./0000_color.png')
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
print(image.shape)
box = np.array([100,100,400,600])
time_s = time.time()
predictor.set_image(image=image)
time_e1 = time.time()
masks, _, _ = predictor.predict(box=box)
time_e2 = time.time()
print('encoding time:',time_e1-time_s)
print('decoding time:',time_e2-time_e1)

When using a GPU (RTX 3090), the output is:
cuda:1 (480, 640, 3) encoding time: 2.325988531112671 decoding time: 0.018665313720703125
When using cpu, the output is:
cpu (480, 640, 3) encoding time: 0.8602027893066406 decoding time: 0.08456754684448242
The cpu even runs faster on encoding

I met the same problem on a single RTX 3090, have you solved the problem?

I met the same problem on a single RTX 3090, have you solved the problem?
I didn't try this code later because I found some alternatives.
If you want to use MobileSAM, the implementation here https://github.com/IDEA-Research/Grounded-Segment-Anything/tree/main/EfficientSAM#run-grounded-mobilesam-demo is fast, and I tried on single RTX4090.

I met the same problem on a single RTX 3090, have you solved the problem?

I didn't try this code later because I found some alternatives. If you want to use MobileSAM, the implementation here https://github.com/IDEA-Research/Grounded-Segment-Anything/tree/main/EfficientSAM#run-grounded-mobilesam-demo is fast, and I tried on single RTX4090.

Which of EdgeSAM, FastSAM, LightHQSAM, Mobile SAM, and RepVitSAM has the fastest running speed and the best performance?


Visual issue in cover calculator settings

The size in the Token Cover Size page is showing as a javascript object:

That's interesting. Can you share your Foundry and (I assume) DnD system versions?
And also (if in chrome) press F12 to open developer tools and have a look at the console to see if you have any errors?

Foundry: Version 11 Stable 315
DnD: 3.0.3
Yes sorry should have mentioned this was in the D&D system!
No errors in the console at all

That is so weird. Running the same versions on mine without issue. For interest sake, was it a new install on DnD 3.0.3? Meaning you had you system on 3.0.3 and then installed the module?

So that was a world that was upgraded from a previous version to Dnd 3.0.
But just to check, I just went and created a brand new DnD 3.0 world, added cover calculator and Athenaeum only as modules and still getting the same issue:

BTW, Cover Calculator version is 2.0.4

I'm also getting this on DnD 3.2.1 Foundry 11 Simbul's 2.0.4. not sure if it is helpful but the reason I created a new world is because in my existing world (with more modules) Simbul's suddenly started recognizing all modules as dead, when they werent...
has the DnD system been updated to change the way token size/dead status has been determined?


Ignore files eror

https://github.com/homerjam/angular-ui-router-anim-in-out/blob/master/bower.json shows the css and scss files from their directories not to be removed, but they're removed after install. If I set the ignore list to this, it works:
    "ignore": [
        "*",
        "!anim-in-out.js",
        "!css/",
        "!scss/"
    ],

Fixed by #219.


Problem with cyrillic symbols

I installed the rails3-jquery-autocomplete and everything works well. But when I enter сyrillic characters, I see encoding problem:
JSON:
[{"id":"16","label":"\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd","value":"\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd"}]
Please, help, how to use encoding, or what must i do.
autocomplete :country, :name do |items|
CustomJSON::Encoder.encode(items)
end

The page encoding is UTF8. Mysql and Rails configs are set to UTF8.
ruby 1.9.2
rails 3.1.1

The problem was with the mysql encoding.
If I use the pgsql there are no problems with cyrillic symbols.
Sorry and Thanx!


Kansa Core Bug: Convert-CSV invalid Cmdlet

PowerShell v4- got an error when trying to convert output to TSV indicating that Convert-CSV is not a valid cmdlet.  I believe the correct cmdlet should be ConvertTo-CSV instead.  Changing in my fork and sending a pull request.  Below is the snippet:
# save the data
switch -Wildcard ($OutputFormat) {
...
      "*tsv" {
         $Outfile = $Outfile + ".tsv"
          # LogParser can't handle quoted tab separated values, so we'll strip the quotes.
          $Recpt | Convert-Csv -NoTypeInformation -Delimiter "`t" | ForEach-Object { $_ -replace "`"" } | Set-Content -Encoding $Encoding $Outfile
      }
...

Damn. Nice catch Juan. I've fixed it. This demonstrates the need for something like Pester.


group_vars: rm Infernalis comment

As of Infernalis, the Ceph daemons run as an unprivileged "ceph" UID, and this is by design.
Commit f19b765f7995dab4aca46e1885320dc440485ead altered the default civetweb port from 80 to 8080 with a comment in the commit log about "until this gets solved"
Remove the comment about permissions on Infernalis, because this is always going to be the case on the Ceph versions we support, and it is just confusing.
If users want to expose civetweb to s3 clients using privileged TCP ports, they can redirect traffic with iptables, or use a reverse proxy application like HAproxy.

Oops sorry, can you rebase on top of https://github.com/ceph/ceph-ansible/pull/1325? Thanks!

This change should have been done in ceph-common/defaults/main.yml because it'll be overwritten next time somebody runs generate_group_vars_sample.sh.

I am getting the sense that PRs are merging too quickly without enough review


(MAINTENANCE) update promoninja prizes

out:
video_interview
rollups
earlybird
left:
meetups
vip
blog
brand_highlight (maximum position: 1 instead of 4)
leaflets

after an update - please confirm its OK @KarolinaTEH

thanks,
just one thing - meetups:

max: 300 (as agreed with PS it should not be limited)
level: 20 (50 invitations for the first 20 points)

max: 300 - OK
"level: 20 (50 invitations for the first 20 points)"
does Peter know that we have different solution for counting views? (via fingerprinting library)?
i would wait with that decision till:

we see if we would have better visitor RSVP panel (sending invites and not having responses is worse that not sending invites)
check the dynamics with new pageviews counting solution

another thing is that they see (without imiting) LINKEDIN profiles so...they may try to use it and we do not have to pollute communication on our side
one can always change some parameteres during the contest and it is nice motif to another reminder msg

related https://github.com/ecommerceberlin/exhibitors/issues/183

yes, he knows.
level: ok - totally understand. So I'll adjust translations (delete the 20 points part)

btw: .... that linkedin profiles and company_role should be accessible as a "prize" for (even micro-engaged ones)... what do you think?
we could add another prize like "profiled list of visitors".... and that could be accessible for those with min 20pts :)

participation_type could stay as a standard ofcoz....

sounds like a good idea. I'll just talk it through in a minute with Mark and Peter

yep... we always looked for more virtual prizes and it seems perfect fit for this purpose ....
for company_role the would see the label "click here for position info" (info how to get it via overlay).... for LI they would get an overlay with the info how to make it visible....also the company_role filtering would be disabled

ok, so we came to the conclusion that it'd be really difficult to communicate.
exhibitors were not eager to use this tool before and blocking 2 features would not make it easier to convince them to do so
but we could just hide linkedin, leaving filters visible for everyone -> easy to communicate and not that confusing for exhb


Build GUI with CMake

What has to be done
Switch over the build toolchain of the GUI to use CMake instead of QMake. It appears that this is possible, as a cursory web search reveals a number of pages that talk about how to do this, at least in principle... for example from Qt themselves:
https://doc.qt.io/archives/QtQuickCompiler/qquickcompiler-building-with-cmake.html
Resulting functionality should not change, only how the makefiles are generated.
How do you know it has to be done
While QMake is relatively capable at doing "usual" things, we are trying to be quite rigorous with all our test coverage and static check requirements. QMake is just not as flexible at specifying various compiler settings at different scopes, for example - adding some include paths for third-party liberaries with -isystem only to suppress static checks on code we did not ourselves write, or perhaps silencing specific checks on that code. This is where a more mature and flexible system like CMake would come in useful.
Don't forget

[ ] Put the issue in a milestone
[ ] Assign it to one of the projects
[ ] Add appropriate labels
[ ] Mention dependent/blocked issues if any

Closed since #1185


vkGetSwapchainImagesKHR is different form the official api description

##Description
I capture a frame on a Mate 30 Pro mobile(android), the graphics api is Vulkan,  I double click to open the RDC file and export as zip.xml file。When I analyze the trace file of zip.xml, I found the api vkGetSwapchainImagesKHR different from khronos spec description。Here is the log:
<chunk id="1024" index="5" name="vkGetSwapchainImagesKHR" length="28" threadID="529413422416" timestamp="52914285" duration="0">
    <ResourceId name="device" typename="VkDevice" width="8" string="ResourceId::2455">2455</ResourceId>
    <ResourceId name="Swapchain" typename="VkSwapchainKHR" width="8" string="ResourceId::2537">2537</ResourceId>
    <uint name="SwapchainImageIndex" typename="uint32_t" width="4">0</uint>
    <ResourceId name="SwapchainImage" typename="VkImage" width="8" string="ResourceId::2539">2539</ResourceId>
</chunk>
<chunk id="1024" index="6" name="vkGetSwapchainImagesKHR" length="28" threadID="529413422416" timestamp="52914307" duration="0">
    <ResourceId name="device" typename="VkDevice" width="8" string="ResourceId::2455">2455</ResourceId>
    <ResourceId name="Swapchain" typename="VkSwapchainKHR" width="8" string="ResourceId::2537">2537</ResourceId>
    <uint name="SwapchainImageIndex" typename="uint32_t" width="4">1</uint>
    <ResourceId name="SwapchainImage" typename="VkImage" width="8" string="ResourceId::2540">2540</ResourceId>
</chunk>
<chunk id="1024" index="7" name="vkGetSwapchainImagesKHR" length="28" threadID="529413422416" timestamp="52914311" duration="0">
    <ResourceId name="device" typename="VkDevice" width="8" string="ResourceId::2455">2455</ResourceId>
    <ResourceId name="Swapchain" typename="VkSwapchainKHR" width="8" string="ResourceId::2537">2537</ResourceId>
    <uint name="SwapchainImageIndex" typename="uint32_t" width="4">2</uint>
    <ResourceId name="SwapchainImage" typename="VkImage" width="8" string="ResourceId::2541">2541</ResourceId>
</chunk>

Purpose
The API description:
// Provided by VK_KHR_swapchain
VkResult vkGetSwapchainImagesKHR(
    VkDevice                                    device,
    VkSwapchainKHR                       swapchain,
    uint32_t*                                     pSwapchainImageCount,
    VkImage*                                    pSwapchainImages);

Parameters
device is the device associated with swapchain.
swapchain is the swapchain to query.
pSwapchainImageCount is a pointer to an integer related to the number of presentable images available or queried, as described below.
pSwapchainImages is either NULL or a pointer to an array of VkImage handles.

Purpose trace as follow:
vkGetSwapchainImagesKHR({"device":527600705536, "pSwapchainImageCount":3, "pSwapchainImages":[2500,2535,2566], "swapchain":527600705538})
Question

Why the renderdoc save the swapchain image parted？

Why not save the swapchain Image in an array?

The last Question:  are you planning to fix this problem and consistent with the official api description?

Environment

RenderDoc version:  1.9
Operating System: Windows 10 and Android 9
Graphics API: Vulkan 1.2.141
GPU Device: Mali-G76
ADB version: 1.0.41

RenderDoc's zip.xml export is the raw conversion of its internal serialised data. It is not a 100% exact representation of API calls.

The zip.xml export is confused, how many this different in the RenderDoc ? if I want to keep with the API, i must modify the serialised method? thank you for your answer.

RenderDoc's zip.xml export is the raw conversion of its internal serialised data. It is not a 100% exact representation of API calls.

The zip.xml export is confused, how many this different in the RenderDoc ? if I want to keep with the API, i must modify the serialised method? thank you for your answer.

I'm sorry but I don't understand what you're asking.
If you want an exact list of every API call and parameter then you should use a different tool, like the api_dump layer or vktrace.

I'm sorry but I don't understand what you're asking.
If you want an exact list of every API call and parameter then you should use a different tool, like the api_dump layer or vktrace.

yes you are right, I want an exact list of every API call and parameter. But I want to use renderdoc, this is a condition. can you give me a suggestion? thank you very much again!


New CDK version released: v1.76.0

A new version of CDK has been released! :tada:Version: v1.76.0Release: https://ift.tt/2JBUrIp@mrgrain Please release a new version of @mrgrain/cdk-esbuild

Released<EMAIL_ADDRESS>


Feat/results

Pull Request Test Coverage Report for Build<PHONE_NUMBER>8
Details

32 of 33 (96.97%)  changed or added relevant lines in 4 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.3%) to 90.132%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

baseballhub/src/fragments/Game/GameStory/GameStory.tsx
20
21
95.24%

Totals

Change from base Build<PHONE_NUMBER>3:
0.3%

Covered Lines:
1373

Relevant Lines:
1491

💛  - Coveralls


Way to use in SvelteKit and Vite bundler

Hi,
We are currently using Sveltkit and Vite bundler.
Is there any way to use Scichart in Sveltkit and Vite bundler environment?
If so, I'd be grateful if you could provide sample code.
Or are there any plans to support Vite bundler?

@ki93 You probably are able to find some pointers on how to integrate the library with vite in this example.


Add padding to fix ui being hidden

Just adding a little fix to move the ui down a bit from the edge. On Safari it was on on the edge and was not fully visiable.

For reference, here’s the bug report: https://github.com/TokamakUI/Tokamak/issues/270


Feature Request: Add Bounding Box info to LSTM choices

Newly made enhancements by @noahmetzger provide accurate bounding box info at the character level as well as the LSTM choices for the character.
https://github.com/tesseract-ocr/tesseract/pull/2554
https://github.com/tesseract-ocr/tesseract/pull/2576
An earlier commit by Nick White had added the option to include character bounding boxes in hocr output.
https://github.com/tesseract-ocr/tesseract/commit/06b7a7b188b2ed21a101cd179b4dd3cfc13aaf30
Currently it is possible to get HOCR output with both options as follows:

 tesseract -l eng --psm 6 --dpi 300 -c lstm_choice_mode=4 -c lstm_choice_amount=0  -c hocr_char_boxes=1  --tessdata-dir ~/tessdata_best choices.png choices40 hocr

The output is as follows:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

  <title></title>
  <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
  <meta name='ocr-system' content='tesseract 5.0.0-alpha-315-g5a47' />
  <meta name='ocr-capabilities' content='ocr_page ocr_carea ocr_par ocr_line ocrx_word ocrp_wconf'/>

             B
             A
             S
             O
             B
             C
       B
       A
       S
       O
       B
       C

             6

       6

This feature request is for combining the output from both options with accurate bounding boxes and confidence values at character level.

The formatting can also be improved. @noahmetzger, is the calculation of the confidence values correct? See my code comment.

yeah the calculation of the confidence values differs slightly to the original confidence because my values are using the new bounding boxes to evaluate the confidence while the old algorithm relates the old character boundaries.

@kba, I just tried to find the right form from the hOCR spec.
Why does the example for x_bboxes have 8 parameters there? Shouldn't there be exactly 4 parameters?
And what would be the recommended form for character choices with bounding boxes?

Why does the example for x_bboxes have 8 parameters there? Shouldn't there be exactly 4 parameters?

The example is for two characters.

And what would be the recommended form for character choices with bounding boxes?

http://kba.cloud/hocr-spec/1.2/#segmentation would seem the most appropriate mechanism, using  and  in a . However, I have not come across that construct in the wild.

So the first word from the image above could be encoded like this?

   B
   A
   S
   O
   B
   C

    B
    R

    A
    a

    S
    5

    O
    0

    B

    C
    (

I added the required nlp property to the alternatives and wonder how many digits are reasonable for float values in hOCR. Maybe the spec should suggest 4 or 5 digits.
@noahmetzger, I just noticed that the characters for the alternatives still need to be escaped (HOcrEscape). How can I get the bounding boxes of the different alternatives, or are they all identical?

The example with nlp in https://github.com/kba/hocr-spec/blob/master/1.2/spec.md only uses single digits after decimal point.
hello

If I am not mistaken, 1.7 would be a recognition probability of 18 %. That's not really good. The other values are even worse, so that seems to be a bad example.

@stweil Shouldn't it be formatted as below, considering that the alternatives are for each character?
Then ins class indicates the character that was chosen and del class indicates the choices that were discarded.

    B
    R

If I am not mistaken, 1.7 would be a recognition probability of 18 %. That's not really good. The other values are even worse, so that seems to be a bad example.

http://kba.cloud/hocr-spec/1.2/#segmentation has a different (obsolete?) example.

hello
hallo

nlp 0.3 would be 74 % (exp(-0.3) * 100), nlp 1.1 would be 33 %.

@noahmetzger Does your new code regarding choices implement this?

@Shreeshrii Yes the new code should be compatible to hocr_char_boxes now.

Thanks. I tried it now using lstm_choice_iterations instead of lstm_choice_amount. The output is better formatted than before, but the confidence levels in both are still different.
tesseract -l eng --psm 6 --dpi 300 -c lstm_choice_mode=2 -c lstm_choice_iterations=0  -c hocr_char_boxes=1  --tessdata-dir ~/tessdata_best choices.png -  hocr
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

  <title></title>
  <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
  <meta name='ocr-system' content='tesseract 5.0.0-alpha-381-g4257' />
  <meta name='ocr-capabilities' content='ocr_page ocr_carea ocr_par ocr_line ocrx_word ocrp_wconf'/>

       B

         B

       A

         A

       S

         S

       O

         O

       B

         B

       C

         C

       6

         6

Also, shouldn't the class='ocrx_cinfo' for each character from both commands be combined together?

yeah the confidence levels will stay different as they are different rating procedures. The second rating is the rating procedure which is also used to evaluate the choices. It is based on den confidence Levels inside the beamsearch finding the best path. It is mainly there to be compared with the other choices.

Ok.
So, how do I get the bounding boxes with new values ( not using old hocr_char_boxes=1).

Example in reply to https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/tesseract-ocr/WX4yZUMUsYQ/Sp5QcN9hBQAJ
 tesseract -l eng --psm 6 --dpi 300  -c hocr_char_boxes=1  --tessdata-dir ~/tessdata_best choices.png choices hocr

HOCR output

       B
       A
       S
       O
       B
       C

       6


proposal: streaming reader as Searcher param

https://github.com/bytedance/sonic/blob/d2711a6af0e3bcc2015f9058297c2eb013e621e2/ast/search.go#L33
say response.Body  is a stream as io.Reader, shall we just parse the stream to avoid large memory allocation on searching?
example:
resp, err := httpClient.Do(req)
// handle err
defer resp.Body.Close()
searcher := NewStreamSearcher(resp.Body)
node, err := searcher.GetPath("foo", "bar", "largePayload")
// handle error
io.Copy(writer, node.GetStream())

Not very possible.
One of the reason Sonic is so fast, is that it uses SIMD instructions to accelerate string processing, which require contiguous memory with known size. Streams work with bytes, and do not have predetermined length, which is not very compatible with SIMD instructions.
You can use a pool of some kind (like sync.Pool) to manage the JSON string buffer for you, it also helps with memory allocation.
Also, because of the delimiter-based nature of JSON, it is not a very Stream-friendly format per-se.

got it.
while,  in fact sync.Pool does little help, especially on IoT device with few memory.
thanks anyway.


Using Kubecost with an Openstack infrastructure (custom prices)

Hello,
At my company, we are starting to implement a Kubecost, however part of our infrastructure is based on Openstack.
Kubestack does not support flavor billing, which results in correct usage patterns, but incorrect pricing.
I know that cloud providers offering openstack (like infomaniak: https://docs.infomaniak.cloud/user-guide/Billing_Metering_and_Rating/), are based on calculation modes similar to Kubecost (using metrics returned via openstack gnocchi), and convert to a fictitious currency, then to a real currency.
Are there any developments planned in kubecost to support flavor pricing? Possibly CSV loading of flavors or different network/disk/S3/Volume costs? (this seems to be complementary to #1095).
Best regards
gz#1498

Curious @maxime-sourdin is the flavor of the node written anywhere in the node label?  If so I think this becomes a lot easier to support, or even already possible with the CSV provider.

From what I see on the infrastructure, I have labels like this:
beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
failure-domain.beta.kubernetes.io/is-baremetal=false
failure-domain.beta.kubernetes.io/region=eu-west-0
failure-domain.beta.kubernetes.io/zone=eu-west0b
kubernetes.io/arch=amd64
kubernetes.io/availablezone=eu-west0b
topology.kubernetes.io/zone=eu-west-0b
type=small-node
node.kubernetes.io/instance-type=s3.4xlarge.4

with indeed the flavor indicated in labels
Translated with www.DeepL.com/Translator (free version)

In that case, we should already be able to use custom pricing with flavors. Take a look at how the pricing CSV works:
https://github.com/kubecost/cost-model/blob/develop/configs/pricing_schema.csv
You can set InstanceID to the flavor and InstanceIDField to the label where you're reading flavor, eg:
InstanceID: small-node
InstanceIDField: spec.metadata.label.type

@maxime-sourdin were you able to make custom pricing work with flavors?

Hello @ivankube,
I was able to create a csv file with the flavors (approximately 200 lines!!!). As expected, there are some internal problems with the labels that made it difficult to create the csv file.
I have not yet been able to test the integration due to lack of time, sorry

@maxime-sourdin it sounds like you can theoretically make custom pricing work with flavors. Thus, I'm going to close this issue. Please feel free to re-open if you'll face any issues.


Screenshot is dated

[Enter feedback here]
Hi,
Could you please provide an updated screenshot?  It looks very old and doesn't match what the current Marketplace looks like.
Thanks,

Stephen

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 90d78351-7bde-2224-ae47-1c567c65a05d
Version Independent ID: cf1c0055-aca8-17e0-73ca-9b5688c25909
Content: Private offers in Microsoft commercial marketplace
Content Source: articles/marketplace/private-offers.md
Service: marketplace
Sub-service: partnercenter-marketplace-publisher
GitHub Login: @vikrambmsft
Microsoft Alias: vikramb

@sgel-msw Thanks for your comment! We will review and provide an update as appropriate.
@vikrambmsft FYI

@sgel-msw Thanks for your comment! We will review and provide an update as appropriate.
@vikrambmsft FYI

Working with vikrambmsft; #35522.

#reassign:ShaiSandMSFT

ShaiSandMSFT is not a valid GitHub ID, or is not a collaborator on this repo.

We updated this screen on 2/22. Sorry for not notifying of this sooner. If this does not resolve your question(s), please provide additional information, or submit a support ticket at https://docs.microsoft.com/en-us/azure/marketplace/support. Thank you for contributing! Task 35522. #please-close


How to use it, where to insert your file?

Hello, sorry you are in an early release,  this repo has been put public recently to perform some test on a custom OS.
Many things are incomplete, but if you want take a look you can see how this project is build here:
DJGPP build
This project is relatively simple to build since no real dependency is required. (Maybe only the libstd/libc)
If you are under Windows, it's highly recommended to use the Cwc Compiler
Then you just need to double click on a .cwMake file
I put some working/in progress example here if you want to try:
XE-Loader-bin


[Story]: Empty State platform selection

Goal
Create a view to select the model serving platform
Dependency issue

https://github.com/opendatahub-io/odh-dashboard/issues/1941

Itemized goals

Add a title and description to reflect the selection
Add the card for single model serving platform
Add the card for multi model serving platform
Add a warning label to tell the user that there's no way to revert the selection
Add the logic to determine if the project is using wether kserve or modelmesh

Aspects continued elsewhere
No response
Mocks

https://www.sketch.com/s/113593f8-5970-49d6-a352-709b07639127/a/mPM4ngw

I think the easiest way to do this until we worry about resets is just set the modelmesh-enabled flag to true or false based on their selection. No label is "choice time" (assuming feature flags back that up)
When we support the switching inside a project... we just ignore the label all together when there is no ServingRuntime objects in the project.

@andrewballantyne Yes, that makes total sense.


Can not use the function "placeCaretAfterNode".

version:
"ngx-tagify": "0.7.6"
Can not call placeCaretAfterNode, below is my code snippet:
addTagsToForm() {
    let value = "My Tag";
    let tag: TagData = { value: value };
    let tagElement = this._tagifyService.get("MixedTagArea").createTagElem(tag);
    this._tagifyService.get("MixedTagArea").injectAtCaret(tagElement);
    let elementAfterTag = this._tagifyService
      .get("MixedTagArea")
      .insertAfterTag(tagElement, "");
    this._tagifyService
      .get("MixedTagArea")
      .placeCaretAfterNode(elementAfterTag);
  }

The reason not use addTags is that I used a button to trigger addTagsToForm() and I want to place caret after insert tags.
Please help.

You are basically using the core library @yaireo/tagify. With the latest release v4.17.9, placeCaretAfterNode() moved to helpers file and is not exposed anymore (https://github.com/yairEO/tagify/commit/1d41a6980507de07d7161846125aea26a842aa7e). You should report there that this is a breaking change, and he should expose this function again. Or you can manually install<EMAIL_ADDRESS>(which would however be just a workaround).

I see. Thank you so much!


Cannot Find Dynamic Objects in FlaUI

How to access/Find dynamic elements that can be found with FLAUIInspect  but are inaccessible through code. I have two TreeItems in the application , Shared and Local

With FlaUIInspect both are found

The only difference between them in the name.
When I am trying to interact with "Local" element, this can not be found through code. I have tried a lot of approaches by finding all children, put some waiting times, used treewalker , nothing can find this second element "Local", only Shared can be found

There is something that we can do to find it?

I have same issue with you and looking for the way


Arte.tv no longer works

Please follow the guide below
Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2017.04.17. If it's not read this FAQ entry and update. Issues with outdated version will be rejected.

[X] I've verified and I assure that I'm running youtube-dl 2017.04.17

Before submitting an issue make sure you have:

[X] At least skimmed through README and most notably FAQ and BUGS sections
[X] Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

[X] Bug report (encountered problems with youtube-dl)
[X] Site support request (request for adding support for a new site)

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue
Arte.tv used to work but does no longer after they redesigned the website.
Example:
$ youtube-dl -v "http://www.arte.tv/de/videos/048696-000-A/der-kluge-bauch-unser-zweites-gehirn"

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'http://www.arte.tv/de/videos/048696-000-A/der-kluge-bauch-unser-zweites-gehirn']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.04.17
[debug] Python version 2.7.13 - Darwin-16.5.0-x86_64-i386-64bit
[debug] exe versions: avconv 12, avprobe 12, ffmpeg 3.3, ffprobe 3.3, rtmpdump 2.4
[debug] Proxy map: {}
[generic] der-kluge-bauch-unser-zweites-gehirn: Requesting header
WARNING: Falling back on generic information extractor.
[generic] der-kluge-bauch-unser-zweites-gehirn: Downloading webpage
[generic] der-kluge-bauch-unser-zweites-gehirn: Extracting information
ERROR: Unsupported URL: http://www.arte.tv/de/videos/048696-000-A/der-kluge-bauch-unser-zweites-gehirn

Yes I can confirm this ... used version 2017.04.17 on Mac OSX 10.12.4 (16E195). Thanks a lot for the great tool!

I also have the same issue with this version of youtube-dl.
The website just have change the URLs of their webpages I think.


add net451 support to signalr.redis

net451 framework was missing from the SignalR.Redis project

Hi @searus, I'm your friendly neighborhood .NET Foundation Pull Request Bot (You can call me DNFBOT). Thanks for your contribution!
You've already signed the contribution license agreement. Thanks!
We will now validate the agreement and then real humans will evaluate your PR.
TTYL, DNFBOT;

Thanks. I ordered usings, rebased and merged as 4158dfed8ff090927761466b062de3db5aed946f

Do you know when this will be pushed as a NuGet package?

It already is https://dotnet.myget.org/feed/aspnetcore-ci-dev/package/nuget/Microsoft.AspNetCore.SignalR.Redis

you guys are awesome 👍
Thanks!


Fix - add content security policy to info panel web view

Attempting to resolve #1199.
Currently unsure if content="default-src 'none' is the correct security policy to use.
https://code.visualstudio.com/api/extension-guides/webview#security

Thanks a lot!


通过playground扫描vue文件生成的二维码显示不出来

写了一个简单的vue格式的文件，通过weex test.vue 可以运行起来，并且在浏览器中也看得是正常的。但是通过playground扫描二维码的时候就显示不出来，我的网络没问题，因为同样的we文件就可以显示出来。

请确认你 playgroud 中 WeexSDK 的版本，需要 0.9.5 或以上才支持 Vue。

This PR/issue doesn't received response since Weex migrated to Apache two years ago and this repository is no longer active yet.
Feel free to ask question in new repository and thanks for your contribution.

This issue is outdated for a long time and will be closed now. You can create a new one if you still have questions.


stream join is not work

stream1 content

stream2 content

select * from stream_1 as a left join stream_2 as b on a.assetCode = b.assetCode group by TUMBLINGWINDOW(SS, 20);

Why  no result returned？I'm sure I have the same assetcode
am I misunderstood？

Hi nick,  the stream/table alias is not supported in join. Try not using alias. Thanks.

Hi nick, the stream/table alias is not supported in join. Try not using alias. Thanks.

Alias is not supported. An error message should be reported

Yeah, you're right. This is actually a known issue #943.  We are going to support this so not validating it


[WIP] enable xunit tests

Fix Issue #1511
The xUnit Analyzer detected these test methods that are missing the [Fact] attribute.
If these tests were intended to be disabled, they should use the Skip property and provide an explanation. Ex: [Fact (Skip = "specific reason")]

[ ] I ran Unit Tests locally.
[ ] CHANGELOG.md updated with one line description of the fix, and a link to the original issue if available.

For significant contributions please make sure you have completed the following items:

[ ] Design discussion issue #
[ ] Changes in public surface reviewed
[ ] CHANGELOG.md updated with one line description of the fix, and a link to the original issue.

The PR will trigger build, unit tests, and functional tests automatically. Please follow these instructions to build and test locally.
Notes for reviewers:

We support comment build triggers

/AzurePipelines run will queue all builds
/AzurePipelines run <pipeline-name> will queue a specific build

I'm waiting on #1517 to merge first

@cijothomas, last PR this year :)


Error out / better description when there is no components listed.

Ran into this earlier today when I didn't have an application set:

github.com/redhat-developer/ocdev  create-components-same-name ✗                                                                                3h40m ⚑
▶ ./ocdev application list
ACTIVE   NAME
         app
         foobar

github.com/redhat-developer/ocdev  create-components-same-name ✗                                                                                3h40m ⚑  ⍉
▶ make bin && ./ocdev update -v --git https://github.com/openshift/nodejs-ex.git

go build -ldflags="-w -X github.com/redhat-developer/ocdev/cmd.GITCOMMIT=b99df98" -o ocdev main.go
DEBU[0000] getOcBinary - searching for oc binary
DEBU[0000] envKubectlPluginCaller =

DEBU[0000] envOcBin =

DEBU[0000] using oc from PATH
DEBU[0000] using oc from %s/usr/local/bin/oc
DEBU[0000] Trying to connect to server <IP_ADDRESS>:8443 - %!v(MISSING)
DEBU[0000] Server https://<IP_ADDRESS>:8443 is up
DEBU[0000] isLoggedIn err:  <nil>
 output: "system:admin\n"
DEBU[0000] myproject-
DEBU[0000] Getting BuildConfig:
unable to update the component: unable to get the BuildConfig file: unable to get BuildConfig : resource name may not be empty

We should be checking to see if the component is actually available (not blank / is available) and error out better / notify the user that no component is actually selected / there are none / there is no actual application selected either.

@cdrage @kadel @containscafeine Should the error message display one error message or multiple depending upon the type of the error like no application exists or no application is set as active?

In the situation shown above, the message should warn the user about the component.
It should show display that there is no active component.

@kadel What if no application or component is present and the user executes the command?

no component also means no active component, so that is the same.
If there is no active application then it should display a message informing the user about that.
But currently that is harder to check, you will probably need changes in https://github.com/redhat-developer/odo/pull/358/files for that.

@kadel and what about a empty project with no applications in it?

@kadel Ok got it but should the error message in both cases be same or different?

different

@kadel Can we close this?

fixed in #374


Lilu v1.2.0

Please update for Lilu v1.2.0

Please update for Lilu v1.2.0

Update ath9kfixup.kext according to v1.2.3


Color switcher UI glitch

The color switcher widget for customizing TimeStyle colors doesn't work quite right--the user must first tap the old color before being allowed to select a new color

Never seen that before. What phone are you on?

Closed, can't reproduce


[VS Code] Cannot read property 'sessionId' of undefined

Error:
Cannot read property 'sessionId' of undefined
Steps to Reproduce:
1.
2.

Version Data

extensionName
VSLS

extensionVersion
1.0.614

protocolVersion
2.2

applicationName
VSCode

applicationVersion
1.36.1

platformName
Linux

platformVersion
4.15.0-20-generic

@DefGh thanks for reporting the issue. Can you please let us know when are you seeing this error? Also can you please attach logs when you hitting this error? You can access them via the Live Share: Export Logs command from the command palette (ctrl + shift + p) and attach them to this issue.

Sorry, but i don't remember the moment the bug appears. I can give you
access to my PC, if you need.
пн, 5 авг. 2019 г., 7:56 Nandini<EMAIL_ADDRESS>
@DefGh https://github.com/DefGh thanks for reporting the issue. Can you
please let us know when are you seeing this error? Also can you please
attach logs when you hitting this error? You can access them via the Live
Share: Export Logs command from the command palette (ctrl + shift + p) and
attach them to this issue.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/MicrosoftDocs/live-share/issues/2609?email_source=notifications&email_token=ABK4UTYOQC6WYOUGCVJXR63QC6XHBA5CNFSM4IJDF7U2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3QVYIQ#issuecomment-518085666,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABK4UT2RBUOLHS6FV37TCMTQC6XHBANCNFSM4IJDF7UQ
.

Closing this issue due to inactivity. Please re-open it if you're still encountering an issue. Thanks!


Preserve the coordinate reference system when returning geodataframes

This sets the coordinate reference system to be the same as the source coordinate reference system of the trajectories.

Thank you for this contribution, Aaron! Could you please also include a test that checks that this is working as expected?

Codecov Report

Merging #66 into master will increase coverage by 12.30%.
The diff coverage is 96.96%.

@@             Coverage Diff             @@
##           master      #66       +/-   ##
===========================================
+ Coverage   80.98%   93.28%   +12.30%
===========================================
  Files          10       13        +3
  Lines        1404     1459       +55
===========================================
+ Hits         1137     1361      +224
+ Misses        267       98      -169

Impacted Files
Coverage Δ

movingpandas/trajectory_aggregator.py
90.51% <83.33%> (+74.36%)
:arrow_up:

movingpandas/tests/test_trajectory_aggregator.py
100.00% <100.00%> (ø)

movingpandas/trajectory.py
87.25% <0.00%> (-2.01%)
:arrow_down:

movingpandas/trajectory_collection.py
91.37% <0.00%> (-0.50%)
:arrow_down:

movingpandas/tests/test_trajectory.py
100.00% <0.00%> (ø)

movingpandas/tests/test_trajectory_collection.py
100.00% <0.00%> (ø)

movingpandas/trajectory_generalizer.py
96.47% <0.00%> (ø)

movingpandas/tests/test_trajectory_generalizer.py
100.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 74645fa...d307329. Read the comment docs.

@anitagraser I updated the branch with new tests for this


[BIA-420] Initial commit - Source code.

The docs-out directory contains files used for creating PDF, which was useful for the Exchange release of version before 2.0.0, but no longer necessary. Can you remove that directory from the prior commit and force push an updated branch?


Empty Profile page looks sad

An empty User Profile Page looks sad :-(

I wonder if we can do something a bit more welcoming?
Like maybe add a message like, "It's lonely here...  try these Skillmaps to earn a badge!"

@Jaqster hour of code badge will be there once we fix the bug. Lets close it when we verify.

I didn't file this specifically for Hour of Code.  This is for any user who clicks on their Profile going forward and sees this page.  Maybe should run User Profiles generally through the Customer Development process to see how we can refine and improve this experience.  Will add it to that project.

I meant this won't ever be empty as we show locked badge here starting with hour of code.

I can still see empty in /beta. Shouldn't it have atleast one greyed out icon  - hour of code


pythonArgs are not taking effect

My launch.jason
{
// Use IntelliSense to learn about possible attributes.
// Hover to view descriptions of existing attributes.
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
"version": "0.2.0",
"configurations": [
{
"name": "Python: Current File",
"type": "python",
"request": "launch",
"program": "${file}",
"console": "integratedTerminal",
"justMyCode": true,
"python": "C:\Users\dtu\AppData\Roaming\Python\Python38\Scripts\ipython.exe --ipython-dir=.\",
"pythonArgs": [ "--ipython-dir=.\"],
"cwd": "C:\Dropbox\dtu_HP_desktop2\PythonLab_beryllium\PythonLab"
}
]
}
Command that VS Code generated:
PS C:\Dropbox\dtu_HP_desktop2\PythonLab_beryllium\PythonLab> & c:/Users/dtu/AppData/Roaming/Python/Python38/Scripts/ipython.exe c:/Dropbox/dtu_HP_desktop2/PythonLab_beryllium/PythonLab/pythonlab/syseng/perf/hamilton/hamilton_rx_test_top.py
no pythonArgs at all!!

/extPython

It looks like this is caused by the Python extension. Please file the issue to the Python extension repository. Make sure to check their issue reporting template and provide them relevant information such as the extension version you're using. See also our issue reporting guidelines for more information.
Happy Coding!


Access token likely to expire when changing time zone from PST

Hi folks,
When deploying the OVA it'll have the time zone set to PST.  A check during the deployment will advise of the time being too far out considering the time zone.  This is likely because the time is correct (VMware tools pulling time from NTP server on ESXi host).  Correcting the time zone will have an impact when finalizing the deployment as the access token will have potentially expired.  Fixed by logging out/back into Azure.  Worth adding a little note on this around the "Validating appliance configuration" step.
Thanks in advance.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 7a699efb-c78c-2dc6-2d0d-6ffc0a2918ce
Version Independent ID: 36a048d5-71b5-ce1d-75d7-bd5182f4386b
Content: Deploy the configuration server for VMware disaster recovery with Azure Site Recovery
Content Source: articles/site-recovery/vmware-azure-deploy-configuration-server.md
Service: site-recovery
GitHub Login: @mayurigupta13
Microsoft Alias: ramamill

@fixx220 Thanks for bringing this to our attention. Your feedback has been shared with the content owner for further review.

Adding @Rajeswari-Mamilla for the doc enhancement.

@fixx220 It takes 5 min for the changes to be reflected. So, in case of this error, you can wait for 5 min and then retry the operation. Logging out and logging in is not necessary. I am adding this suggestion to troubleshooting documentation. Thank you for sharing your input.
#please-close


feat: created optional usdMode prop to configure default mode

Description
Created optional usdMode prop to configure default mode for EtherInput & Balance component.
Additional Information

[X] I have read the contributing docs (if this is your first contribution)
[X] This is not a duplicate of any existing pull request

Related Issues
Closes #677
Note: If your changes are small and straightforward, you may skip the creation of an issue beforehand and remove this section. However, for medium-to-large changes, it is recommended to have an open issue for discussion and approval prior to submitting a pull request.
Your ENS/address: 0xfC001B20Db6195b148cfc4F7685091931D93cD93

Thank you, @technophile-04 🙌


Pixel scale in pyKLIP JWST.py

Unfortunately, another update of pyKLIP JWST.py is needed. Currently, the pixel scale is taken as the square root of the PIXAR_A2 header keyword, but this header keyword does not exist if the photometry step of the JWST pipeline has been skipped. Skipping this step is common practice for AMI and KPI data. Hence, doing KLIP RDI with KPI data is currently not possible. Tagging @semaphoreP and @AarynnCarter.

Recommend grabbing XSciScale and YSciScale from pysiaf aperture object if header keyword doesn't exist. The equivalent would then be np.sqrt(XSciScale * YSciScale).

Agreed! I have implemented it already and can push it. It makes pysiaf a pyKLIP dependency though @semaphoreP.

Isn't this just grabbing some header keyword? I just want to understand why we need a whole package for this, unless this is quite complicated.

@semaphoreP Unfortunately not! Pixel scales are not in the headers by default. The pixel area is added when running the JWST pipeline photometry step, but if this step is skipped, then the current implementation will not work.
I have updated pyKLIP so that it only tries to import pysiaf when the photometry step has been skipped, so that the package is not required by default. See https://bitbucket.org/pyKLIP/pyklip/commits/0be44828dda8e7c22c465409d6a2decf18750eb6.

Can this be a step in spaceKLIP to import pysiaf and insert the relevant keywords into the header? I'd like to both avoid forcing all pyklip uses to install jwst specific packages, but I also rather avoid not installing dependencies when possible.

Jason's recommendation could work, but we should be careful about implementation. Jens' solution also looks feasible as it assumes the only reason you would be running this portion of the code is for JWST-specific tasks, so the user would already have pysiaf installed as it's a spaceKLIP dependency. As long as you put the import inside the JWST-specific function call, then there should not be any conflict with other portions of the code being used for other purposes. One potential hiccup might be for automated tests and coverage (but pysiaf could be a requirement in a separate requirement-tests.txt file).

Both for testing and coverage, but also to keep JWST-specific things in spaceKLIP, it would be nice to do this in spaceKLIP. We could just have a step that checks for the platescale, does nothing if it's already there, or uses pysiaf to add it into the header. I'd like to keep the JWST interface in pyklip as simple as possible, so we don't have to remember if some pre-processing step is done in spaceKLIP or pyKLIP (it should always be done in spaceKLIP).

Yeah, I agree. I had earlier advocated for moving much of the JWST stuff from pyklip to spaceklip, but it's been a while since I've looked at the pyklip code to see what makes sense.

Ok, I have implemented a utility function in spaceKLIP which populates the PIXAR_A2 SCI header keyword if it is not already available before running pyKLIP (here). I have also reverted pyKLIP back to the previous state which did not need pysiaf as a dependency. On the way, I have also added a fallback option for when SVO is unavailable to the JWST class in pyKLIP (here). I hope this concludes our work here @semaphoreP and the pyKLIP jwst branch can now be merged into main.

For completeness, I wanted to note that with NIRCam LW for instance, the pixel area in arcsec^2 is 0.003918172030450555 when populated by the JWST pipeline photometry step, but 0.003896778872914225 (i.e., slightly different) when using the pixel scale from pysiaf. Any ideas why this could be the case @mperrin?

My first guess for what might cause that difference in PIXAR_S2 values was probably sightly different choice of apertures, or different versions... But upon looking a bit closer at the code in the pipeline, I see that the JWST pipeline photometry step is not getting values directly from pysiaf, but instead is retrieving reference files from CRDS for pixel area.  Looking at the history comments of those, it seems like the difference is due to including the geometric distortions, and in particular the slight non-orthogonality of the X and Y axes. Example history text from https://jwst-crds.stsci.edu/browse/jwst_nircam_area_0042.fits:

The nominal pixel area that is stored in PIXAR_A2 and PIXAR_SR was calculated by multiplying the nominal pixel scale in the x and y directions by the sine of the angle between the x and y axes. This angle was calculated using the arctangent of the Science to Ideal coefficients in the y and x directions. This file was created using create_pam.py

Follow up question for @JarronL and @juliengirard, maybe you can confirm if the above interpretation is correct?  It looks like the pixel area reference files for NRC_CORON were all derived from commissioning data, so I presume may eventually get updated with more precise values as the geometric distortion knowledge improves for coronagraphy modes?


Support more than one yara file as input

right now yara forces you to use yarac to create a new yara file to use it as input for yara, but yarac doesnt supports precompiled yara files as input, so it is not possible to generate a combined one in case you want to use more than one rule.
So the only option here is to run yara twice. This looks to me like a limitation of the tool, because r2yara supports loading several yara files using the API.

yara doesn't force you to use yarac, that's optional. yarac is used mainly when you want to use the same rules over and over again, so you don't need to compile the rules multiple times. If I understood right you have rules in more than one files and you want to be able to scan a file with all the rules contained in those multiple files, right?

Yes, the thing is that you cant use the compiled yara files as sources for yarac or even pass more than one rule to yara. I fixed this by running the tool more than once. But this is far from optimal because of the lack of ortogonality

On 7 Apr 2017, at 17:44, Victor M. Alvarez<EMAIL_ADDRESS>wrote:
yara doesn't force you to use yarac, that's optional. yarac is used mainly when you want to use the same rules over and over again, so you don't need to compile the rules multiple times. If I understood right you have rules in more than one files and you want to be able to scan a file with all the rules contained in those multiple files, right?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

@trufae How I solve this is to merge the rules I create into one 'superrule', not sure if that would fit your needs however.

No that doesn't works for me because i have several groups of yara rules and i want the user to use N of them of choice, so if i do what you want i have the following options:

distribute the sources (not an option)
build all NxN combinations of all rules to match user's choice (impractical)
run yara N times (slow)

In fact r2 yara plugin can do that, i dont see what's the limitation in yara to not do so.

Implemented in 632a6ab2411ab9e296ce05224199f1e3927f487a.

thanks!

Is there a way to recursively pick all the rule-files under specified folder? and create a single compiled yara binary file?


Bug: doc(...).snapshotChanges does not return DocumentChangeAction

Version info

Angular: 6.0.3
Firebase: 5.1.0
AngularFire: 5.0.0-rc.11
Other: Windows 10, browser (chrome)
How to reproduce these conditions
Subscribe to any DOCUMENT with "snapshotChanges". Eg:
this.afs.collection('myCollection').doc('myDocId')
  .snapshotChanges()
  .pipe(tap(res => console.log(res))
  .subscribe()

Debug output

Expected behavior
According to documentation snapshotChanges should return Observable<DocumentChangeAction<T>>.
Actual behavior
What it does return is Observable<Action<DocumentSnapshot<T>>>. type field is always "value" even if I delete record and payload does not seem to contain actual data (I was not able to find it).
Remarks
snapshotChanges works well on collections returning Observable<DocumentChangeAction<T>[]>. Problem occurs only when using it on document.

same issue on ubuntu
 this.db.collection('polls').doc(id).snapshotChanges().subscribe(poll => {})
poll.payload  is not a DocumentChange, so you can't call .doc property to retrieve le document

@Bielik20 @Noext We are changing doc.snapshotChanges() to return a Observable<DocumentSnapshot<T>> in the next release. The type is useless on it because it is always a value update regardless of the type of change.

How can I know which change occurred if the type field is always "value"?
Thanks.

Hi @davideast thanks for all the hard work on this great project, love the way it auto-syncs data.  Favor: could one of the contributors update the docs on the doc.snapshotChanges() method?  I'm using @angular/fire 5.0.0 and this change is in there, but I have next to no idea how to use this method now.
THANKS!

The problem persists to the present date. I just met it.

@davideast what release were you referencing? I couldn't find anything finished or a pull request and the unusable type causes quite dirty code down the line.

Still having this issue to this day

Having same issue as author, the type on Action type is always returning value no matter what. Maybe we must use snapshotChanges() on a collection-document chain instead?

I have same issue, is imposible to know what type of changes was performed over one specific document using the snapshotChanges, someone know some workaround to resolve this issue?

closing in favor of #2108


fix: not to throw error when no txs in block

Description

I followd this cosmos-sdk/issues/12040 and cosmos-sdk/pull/12108
our version 0.45.x does not have GetBlockWithTxs in x/auth/tx/service, so we create x/auth/tx2.
It's hard to cherr-pick because the directories are different. So I reflected the changes directly into tx2.
Motivation and context

When query through GetBlockWithTxs, invalid request error occurred when no txs are included in block.
The correct behavior is to return the block with empty list of txs.
How has this been tested?

tested no error occured and return 0 length of tx list
Screenshots (if appropriate):
before thsi PR, when tx does not exist in block
{
  "code": 3,
  "message": "out of range: cannot paginate 0 txs with offset 0 and limit 100: invalid request: invalid request",
  "details": []
}

when tx does not exist in block
{
  "txs": [],
  "block_id": {
    "hash": "tMvoEOdixC8Kt9YNwb6vGDsYJO/euSW3TRuDfzLijPk=",
    "part_set_header": {
      "total": 1,
      "hash": "ykh4pBbXq5ehsj2zyFRv0UdOFnf/8rVwjVl5MXN+pFM="
    }
  },
  "block": {
    "header": {
      "version": {
        "block": "11",
        "app": "0"
      },
      "chain_id": "simd-testing",
      "height": "115",
      "time": "2023-04-18T08:51:57.136998Z",
      "last_block_id": {
        "hash": "2mVthS8/78uoXxmNaQaO43UivRU142JD42xE/d3ps4I=",
        "part_set_header": {
          "total": 1,
          "hash": "ucOMMUTAhQtMR8sREYbbyG6RKKIDb3bOxjh5D/j1s0M="
        }
      },
      "last_commit_hash": "sUmrUkJM+xlOfckEN7yXVJX9zFySepZa2Gnl670tzwQ=",
      "data_hash": "47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=",
      "validators_hash": "wUFwitwdiD2JQIkDe4MofWg7YqGlxwofCZBWzI7H5Mk=",
      "next_validators_hash": "wUFwitwdiD2JQIkDe4MofWg7YqGlxwofCZBWzI7H5Mk=",
      "consensus_hash": "BICRvH3cKD93v7+R1zxE2ljD34qcvIZ0Bdi389qtoi8=",
      "app_hash": "39H866EPWyvnODRPEYFHFVXqdH+EJj03QpKADY12BGI=",
      "last_results_hash": "47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=",
      "evidence_hash": "47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=",
      "proposer_address": "VwWa2x8sslGSe+6MjgugQo0iGBA="
    },
    "data": {
      "txs": []
    },
    "evidence": {
      "evidence": []
    },
    "last_commit": {
      "height": "114",
      "round": 0,
      "block_id": {
        "hash": "2mVthS8/78uoXxmNaQaO43UivRU142JD42xE/d3ps4I=",
        "part_set_header": {
          "total": 1,
          "hash": "ucOMMUTAhQtMR8sREYbbyG6RKKIDb3bOxjh5D/j1s0M="
        }
      },
      "signatures": [
        {
          "block_id_flag": "BLOCK_ID_FLAG_COMMIT",
          "validator_address": "VwWa2x8sslGSe+6MjgugQo0iGBA=",
          "timestamp": "2023-04-18T08:51:57.136998Z",
          "signature": "lsvVrOiJqC8HzBR98fExYb32m4a+Hx2BtpeS8xu98EArSt7xM9Fni55jNujet85uWczriUFhUdPOf2zDEGyRBQ=="
        }
      ]
    },
    "entropy": {
      "round": 0,
      "proof": "A1AeRTirM3lC5QXH0DaCevHadrGw5WTRwOkOMytT5rDqUG19978T9wGAoR94pYQBYQ2DIHCI5QzyYAwTB6P8hqfULXBvsb8FAxvCKfjO6yQK"
    }
  },
  "pagination": {
    "next_key": null,
    "total": "0"
  }
}

Checklist:

[x] I followed the contributing guidelines and code of conduct.
[x] I have added a relevant changelog to CHANGELOG.md
[x] I have added tests to cover my changes.
[ ] I have updated the documentation accordingly.
[ ] I have updated API documentation client/docs/swagger-ui/swagger.yaml

Codecov Report

Merging #982 (ea6b6d8) into rc/v0.47.0-alpha1 (1c841d9) will decrease coverage by 0.01%.
The diff coverage is 100.00%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
Additional details and impacted files

@@                  Coverage Diff                  @@
##           rc/v0.47.0-alpha1     #982      +/-   ##
=====================================================
- Coverage              62.08%   62.08%   -0.01%
=====================================================
  Files                    655      655
  Lines                  79041    79041
=====================================================
- Hits                   49076    49075       -1
- Misses                 27283    27284       +1
  Partials                2682     2682

Impacted Files
Coverage Δ

x/auth/tx2/service.go
69.44% <100.00%> (ø)

... and 1 file with indirect coverage changes