Does this work?

Hi, I have been trying to implement VDM as well, and am just wondering whether you have had any success with this?
I am working in PyTorch with the discrete-time version, but I am having trouble with numerical stability. It is not directly comparable, but most of our code seems to match up.
One difference I notice is that you constrained the biases of logSNR function to be positive - I think you can get away with positive weights and unconstrained biases (to satisfy monotonic constraint), but I'm willing to be corrected here. (Not sure if this makes much of a difference).
Otherwise, this is a really awesome implementation! I'm just curious if it works because I am struggling to make my own version work!
Cheers

Hi, Turner. Thank you for your participation.
First, I fail my all experiments, and I'm estimating the reason that the expressivity of wavenet is not enough to model the continuous-time loss. So I try to alter the model with HiFi-GAN generator nowadays.
Second, if the bias you refer was the variable gamma_gap, I think it should be positive to satisfy the monotonic constraint, since the normalized -logSNR is in range [0, 1]. Otherwise, if it was the variable bias of PosDense, I think it should be too, but I can't be sure that I don't have an intuition about that proof.
Precisely, it works and loss converges. But it could not recover the spectrogram, and I thought the loss should be converged at lower points. This was the basis for my intuition, lack of expressivity of the generator model.

Thanks, these training plots are quite helpful!
For the monotonic constraint, I was referring to the bias of PosDense - I think this should be unconstrained. PosDense will be monotonic as long as it has a positive derivative with respect to the input timestep. The bias terms do not affect this derivative. (I am not 100% sure, but I have also seen other examples doing unconstrained bias, such as this Neural Autoregressive Flow implementation.) Maybe this is important for the flexibility of the learned logSNR function?
The other question I have is about the calculation of the loss terms? For example, with the prior and reconstruction, I wonder what happens if you use the NLL and KL instead of MSE? For example, maybe this could affect the relatively scales of the different loss terms?
For example, if you do something likerec_loss = -Normal(q_mu_z0, q_logvar_z0).log_pdf(x) ? The scale will be quite different to using MSE, which might affect learning? (And likewise, for prior loss compute analytic KL from q(z1|x) and N(0, 1) ).
Anyway, these are just some ideas. Interested to follow along if you get things working!

I agree with your opinion that constrained bias can hurt the expressivity of logSNR, since model use sigmoid activation. I'll test the bias-unconstrained PosDense in near days.
(1. constrained PosDense would generate only positive numbers from input time [0, 1], 2. then it will be fed to the sigmoid, 3. the output range [0, 1] limits to [0.5, 1]. 4. cannot utilize whole ranges)
The earlier versions of this repo use the NLL and KL, like your referred -Normal(q_mu_z0, q_logvar_z0).log_pdf(x).
Analytically, for z = alpha x signal + std x noise, reconstruction loss and posterior loss can be rewritten as nll(z, mean=signal, stddev=1) \propto (z - signal) ** 2 + C and KL(z || N(0, 1)) \propto z ** 2 - 2logstd + C. So after version omits 2logstd of posterior loss and simply use square loss only.
(KL(z || N(0, 1)) = nll(z, mean=0, stddev=1) - nll(z, mean=alpha x signal, stddev=std) \propto z ** 2 - 2logstd - (z - alpha x signal) ** 2 / std ** 2 + C = z ** 2 - 2logstd - noise ** 2 + C = z ** 2 - 2logstd + C)
Of course, this is theoretically incorrect, but in experiments, it helps the convergence of the prior and reconstruction loss. I erased the earlier loss plots before, so I can't show you how this loss affects the training curves. So I think it would be nice if you think of it as just heuristics.

Sorry, it was not so different (red for unconstrained posdense, orange for original one). It could make a small difference, not the solution for the major early-convergence problem.
I'll close this issue. Please reopen the issue if you have another questions.
Thanks

I agree with your opinion that constrained bias can hurt the expressivity of logSNR, since model use sigmoid activation. I'll test the bias-unconstrained PosDense in near days.
(1. constrained PosDense would generate only positive numbers from input time [0, 1], 2. then it will be fed to the sigmoid, 3. the output range [0, 1] limits to [0.5, 1]. 4. cannot utilize whole ranges)
The earlier versions of this repo use the NLL and KL, as you referred -Normal(q_mu_z0, q_logvar_z0).log_pdf(x).
Analytically, for z = alpha x signal + std x noise, reconstruction loss and posterior loss can be rewritten as nll(z, mean=signal, stddev=1) \propto (z - signal) ** 2 + C and KL(z || N(0, 1)) \propto z ** 2 - 2logstd + C. In posterior loss, z ** 2 is derived from N(0, 1) log-likelihood and 2logstd acts like preventing posterior collapse. But in optimization perspective, square loss tries to reduce both alpha and sigma, during 2logstd tries to increase the sigma. It causes the training instability in previous experiments, and so after version use (z - noise) ** 2 to increase only sigma and simply use square loss for reconstruction loss too.
(KL(z || N(0, 1)) = nll(z, mean=0, stddev=1) - nll(z, mean=alpha x signal, stddev=std) \propto z ** 2 - 2logstd - (z - alpha x signal) ** 2 / std ** 2 + C = z ** 2 - 2logstd - noise ** 2 + C = z ** 2 - 2logstd + C)
Of course, this is theoretically incorrect, but in experiments, it helps the convergence of the prior and reconstruction loss. I erased the earlier loss plots before, so I can't show you how this loss affects the training curves. So I think it would be nice if you think of it as just heuristics.

In your brief explanation of the reconstruction loss, which @angusturner referred as -Normal(q_mu_z0, q_logvar_z0).log_pdf(x), you firstly replaced it with: nll(z, mean=signal, stddev=1), why the standard deviation item is default to 1? I can't find any clue of the reconstruction loss in the paper. However in my opinion, stddev should be 0 (q_logvar_z0 to be negative infinity) or a small number that is very close to 0, i.e. from z0 to x should be as deterministic as possible.

E_{q(z_0|x)}[-log p(x|z_0)] should form -Normal(mu_{x|z_0}, sigma_{x|z_0}).log_pdf(x),
as they noted in paper:

where we choose p(xi|z0,i) \propto q(z0,i|xi). With sufficiently large SNR(0), this becomes a very close approximation to the true q(z0|x), as the influence of the unknown data distribution q(x) is overwhelmed by the likelihood q(z0|x).

it seems that they tend to use the distribution of q(z0|x) for reconstruction estimation, thus -Normal(mu_{x|z_0}, sigma_{x|z_0}).log_pdf(x) \approx -Normal(q_mu_z0, q_logvar_z0).log_pdf(x), which as you @revsic mentioned is E_{p(x)}[-log q(z_0|x)].
So, I think -Normal(q_mu_z0, q_logvar_z0).log_pdf(x) should be the author's way of reconstruction loss calculation. Of course it was just my one-sided guess.

Right, I misunderstood the formula and agree with the form -Normal(mu_{x|z0), sigma_{x|z0}).log_pdf(x). I will fix it and retest the experiment but I also think it is not a solution to the major working problem.
Thanks for your contribution.

Actually I have done a few trials with some toy datasets, and found it hardly to replicate the noise schedule curve (titled by log SNR vs time t in Figure 4)  learned in that VDM's paper. In my case, SNR_max increases as optimization going on and doesn't seem to converge.
However, maybe I was wrong in some hidden corners. Hoping you @revsic would get a good result.

I also did a re-implementation of VDM using diffwave and I can share some results I got.
@angusturner I would suggest you check the open review version of VDM paper. With Eq. 14 you should get rid of the numerical issue.

So, I think -Normal(q_mu_z0, q_logvar_z0).log_pdf(x) should be the author's way of reconstruction loss calculation. Of course it was just my one-sided guess.

@rardz I can confirm this works well in my implementation.
Some plots I got when training with T = 1000:

The final negative VLB was around ~= -5.75.
The learned noise schedule:

Although my implementation can work quite well with discrete-time loss, it failed on training with continuous-time.
I have tried to disable the variance minimization trick or detach the gradients of the derivative of logSNR when computing loss,  but in both ways, the learned noise schedule curve still bends in the opposite direction. I didn't check very carefully, but most of the codes are matched with @revsic.

continous-time modeling is so hard.. thank you for your comment @yoyololicon

Hi, I know it has been a long time, but I have revisited this recently and found that I can get VDM to work with the correct scaling of reconstruction and prior terms.
I will admit, I have not tried it on raw audio, but on toy problems and another task I am trying for work I can get reasonable results.
For the KL / prior term I use the analytic solution from the official VDM code:
https://github.com/google-research/vdm/blob/main/model_vdm.py
For the reconstruction, I have continuous data, and set it up as follows:
p(x|z0) = q(z0|x)p(x)/p(z0)
p(x|z0) is proportional to q(z0|x) (with p(x) as unknown constant, p(z0) also unknown),

Therefore, I simply optimize rec_loss = -Normal(q_mu_z0, q_logvar_z0).log_pdf(z_0). (i.e. minimize the entropy of the first forward step).
The loss scale looks weird visually (I presume because we are missing the normalization terms), but the training seems stable and the learned values are sensible (actually quite close to the learned values in Kingma's code).
In the case of discrete data it seems you can actually compute the normalization p(z_0) = sum_x{q(z|x_0} for all 256 pixel values. I imagine you could probably make an estimate here with monte carlo sampling too. I'm going to try some more things. But for now this seems to work...
tldr; it kind of works with justrec_loss = -Normal(q_mu_z0, q_logvar_z0).log_pdf(z_0)

The official repo now updates with a colab example:
https://colab.research.google.com/github/google-research/vdm/blob/main/colab/SimpleDiffusionColab.ipynb
In that case,  their use a simple resnet to predict the output (in their case it's discrete logit).
So this may be a very suitable trick for calculating the final reconstruction loss, by adding some simple layers for the output distribution prediction.


Allow psr/http-message v2

Q
A

Bug fix?
no

New feature?
yes

BC breaks?
no

Deprecations?
no

Related tickets
-

Documentation
-

License
MIT

What's in this PR?
Adds support for psr/http-message v2, similar to https://github.com/php-http/message-factory/pull/42.
Why?
Latest version.
Example Usage
N/A
Checklist

[ ] Updated CHANGELOG.md to describe BC breaks / deprecations | new feature | bugfix
[ ] Documentation pull request created (if not simply a bugfix)

To Do

[ ] If the PR is not complete but you want to discuss the approach, list what remains to be done here

I have not updated the changelog because it would conflict with another PR.


Restrict Intellisense

While using javascript, ~90% of the suggestions I feel are useless. I seem to only use suggestions based on the project (ex: variables, methods, etc...) and snippets.
Is it anyway to restrict Intellisense to remove the generic suggestions?
I turned off quickSuggestions to prevent seeing the tons of suggestions but, I realized I still would like to see the suggestions specific to the open project.

Have you tried to configure the "editor.wordBasedSuggestions"?

@weinand I'm not exactly sure what that's for. The description just says // Enable word based suggestions
I just set it to false and I'm not sure if anything changed.

What version of VS Code are you using?
For me the setting looks like this:

I'm using  1.10.2 (1.10.2) on Mac
is other the all of the extra suggestions i mentioned?

This is what I have...

Yes, we'll expand this option in 1.11.0 (current Insiders).
Setting editor.wordBasedSuggestions to false eliminates all generic (text based) suggestions. These are the ones with the 'abc' icon.

I'm not seeing any 'abc' icons. Something new in 1.11.0 also?
This is what I see when I type a. In this case, I only care about the first instance of api since I have an import for it.

No, 'abc' suggestions exist since the beginning of time. They are what the "editor.wordBasedSuggestions" setting controls. But it seems that they are disabled for JS/TS automatically.
You can seem them in plain text files:

I've assigned the issue to someone from the VS Code JS/TS team.

The word based suggestion issue sounds related to #21611
As for the original question, I suspect that the suggestions that you are seeing are coming from the DOM or from the standard library. Try creating a jsconfig.json in the root of your project with the content:
{
  "compilerOptions": {
      "lib": ["es2015"]
   }
}

This instructs our language support to only include typings from the es2015 standard library in your suggestions. To disable all builtin suggestions, use "lib": []


merge current repository with remote updates

Can you do the update from the remote branch in a different clean branch so that your changes are clear and look different from the changes pulled in from the stepinterns repository.

Yes. Sorry, I didn't think about that when updating the working branch.  I think the changes pulled from the remote repository are mixed in with my own changes, and I am not sure how to separate them.

Can you do the update from the remote branch in a different clean branch so that your changes are clear and look different from the changes pulled in from the stepinterns repository.

Yes. Sorry, I didn't think about that when updating the working branch. I think the changes pulled from the remote repository are mixed in with my own changes, and I am not sure how to separate them.

If you create a new independent branch and pull in the changes from the remote stepinterns repo then you can create a PR with just those changes. Then when you update this branch, the extra diffs should dissapear.

Can you do the update from the remote branch in a different clean branch so that your changes are clear and look different from the changes pulled in from the stepinterns repository.

Yes. Sorry, I didn't think about that when updating the working branch. I think the changes pulled from the remote repository are mixed in with my own changes, and I am not sure how to separate them.

If you create a new independent branch and pull in the changes from the remote stepinterns repo then you can create a PR with just those changes. Then when you update this branch, the extra diffs should disappear.

Okay, sounds good.  I will do that right now; I'll cancel this pull request, make a new pull for the new branch, and then work on the changes you mentioned.


memory-profile.robot: type errors

"Get Option State" Keyword returns True/False if the value is a boolean value you "check" or "uncheck" and String if there is some text inside. Relying on the type of returned value may be dangerous and it caused the memory-profile test suite to fail because of type errors.
But the memory profile is a list of values, not a checkbox! Now it turns out if an option value starts with an "X" it is treated as a boolean because the letter "X" is used as a checkmark. That is the case with options like "XMP#1...".

@philipandag Please fix pre-commit. Do you have it installed locally?

@philipandag I love your detailed explanation in the comment.
Next time, can you please try to include it in the commit description instead? Please note that GH will automatically copy it over to MR comment as well. But this way, we store this knowledge in independent git repo, not only in the GH UX.


Request body length mismatch after v2.2.0

The latest release (v2.2.0) introduced a bug that causes the action to fail with the error: Error: Request body length does not match content-length header.
This error occurs in our workflow when uploading assets to a release. This issue was not present in prior versions of the action.
Steps to reproduce:

Set up a GitHub Actions workflow with the following snippet:

- name: Upload Asset
  uses: softprops/action-gh-release@master
  with:
    target_commitish: ${{ inputs.sha }}
    tag_name: ${{ inputs.name }}
    files: ./path/to/assets.zip
    append_body: true
    body: |-
      ### Artifacts
      * `Release asset` - assets.zip

Trigger the workflow with the appropriate inputs.

Expected Behavior:
The workflow completes successfully, and the asset (assets.zip) is uploaded to the GitHub release with the specified body content.
Actual Behavior:
The workflow fails with the following error:
Error: Request body length does not match content-length header
GHA Output:
Run softprops/action-gh-release@master
Found release *** (with id=190203584)
⬆ Uploading assets.zip...
Error: Request body length does not match content-length header

I have the same problem

I suspect this is related to bumping the type command
https://github.com/softprops/action-gh-release/commit/92dffe6c28ac278510d01d723882bf82bd907f89
whose underlying cause would be https://github.com/DefinitelyTyped/DefinitelyTyped/commit/f53c4ca31debf2beabee7569dbaf43c5dd71b210#diff-74a6a13d9653e8d64c70649a6d2bbd5e5c90c28361fa76f921df61bc565ad8ab
Raising an issue there

Heads-up: I've made a fix at https://github.com/softprops/action-gh-release/pull/562, waiting for merge and release. Sorry for the breakage!


Timezones issue

Daterangepicker uses 00:00:00 and 23:59:59 time marks for start and end dates, and uses browser timezone to represent them. String (JSON) representation of these dates are in UTC. And this leads to a problem when timezone is different between clients (or event client/server)
For example, start and end dates are '2015-08-01' and '2015-08-05'. In case of tz to be GMT-4, JSON representation of dates will be "2015-08-01T04:00:00.000Z" and "2015-08-06T03:59:59.999Z". Say, we store this timestamps and display to client. If client has browser in the same timezone, he is ok. But if not, and his timezone is, let's say, GMT/UTC, he will get '2015-08-01 - 2015-08-06' date range instead of correct '2015-08-01 - 2015-08-05'.
Probably date range should explicitly converted to UTC, because we don't deal with time here, only with date ranges.
Also this problem occurs when backend code wants to store date range as dates (not as timestamps) and need to convert it from browsers' timezone, but there is no info about it.
Any thoughts?

I've always dealt with timezones client side before passing the dates to ui modules. This is because there is so much work in dealing with timezones and every company I have worked for has dealt with them differently.
I will refer you to this video to hopefully entertain while also talking illustrating why dealing with timezones is a pain.
https://www.youtube.com/watch?v=-5wpm-gesOY
If you have any suggestions about how this module should deal with them please suggest. I am more than willing to add something if it is warranted.

You're right, timezones are tricky, and that's, actually, the point in case of daterangepicker - we don't need timezones at all here. Correct me if I'm wrong, but use case for date range picker is to handle date ranges - say, when you book a hotel. We don't work with exact timestamps (well, maybe check-in/check-out times need to be specified, but that's more of application level logic).
If I select range "Sep 10 - Oct 10 ", I want exactly this date range to be stored and shown to all clients, regardless of their timezone. That fact that 'Sep 10 00:00:00' in my timezone is 'Sep 09 18:00:00' in another timezone should be irrelevant to date range picker.
My suggestion would be to always store startDate and endDate as UTC with 00:00:00 time, but it would be backward incompatible with people who use it already. So I'd suggest to make new option, something like 'utc' or 'forceUTC' and implement it this way, stepping out of time/timezone calculations completely and let user deal with their particular case.
Currently I use this ugly code to get 'correct' date timestamp:
var end = $scope.dateRange.endDate.toDate();
year = 1900 + end.getYear();
month = end.getMonth();
day = end.getDate();
var end_date = new Date(Date.UTC(year, month, day, 12, 00, 00));

But it's kinda sad :)


Changed base commit definition

:green_heart: Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Duration: 6 min 42 sec

:robot: GitHub comments
Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

:green_heart: Build Succeeded

Buildkite Build
Commit: ee5c9afa8d9c0115fd776859556455cfe8e4f426


Fixed types in tests + improved inference of TEvent passed to TransitionConfig

@davidkpiano used AnyEventObject approach, should be ready to merge now

Merged master, tested locally, good to go!


Feature: Sagemaker compilation jobs

Adds support for Sagemaker compilation jobs create, describe, list, and stop.

@bblommers this one should be merged now, I'll move on to the next one after this is merged.


[1.13.0] Patching webpack_public_path?

I've got a webpack application that I'm deploying to Aerobatic, who provide a CDN as part of their platform. Aerobatic provides some tools to allow JS running in the page to find the CDN root, which allows me to fetch assets from the CDN using code like
((window['__aerobatic__'] || {})['staticAssetPath'] || '') + "/path/to/some/asset.png";

Not my real code, but you get the idea: when developing, assets are fetched from the same origin server as the page itself, while on Aerobatic, they're fetched from Aerobatic's CDN. Aerobatic also rewrites HTML so that asset references in the page (such as <script> tags) refer to the CDN, instead of fetching them directly from the origin.
Is there any way to take advantage of this in Webpack's chunk loader? Inspecting my compiled bundle, I see Webpack uses
…
/******/ 			script.src = __webpack_require__.p + "chunks/" + ({}[chunkId]||chunkId) + "." + {"1":"5ee8ddc71a2ddb623a23","2":"6a7511be760e094d38ad","3":"cbf7d406633e0cb6805d"}[chunkId] + ".chunk.js";
/******/ 			head.appendChild(script);
…
/******/ 	// __webpack_public_path__
/******/ 	__webpack_require__.p = "/js/";
…

to load chunks. The resulting script elements load chunks from the origin directly, even though the entry point came from a CDN. If I could replace the literal "/js/" (which comes from my webpack.config.js) with a function call, or if I could reliably modify __webpack_require__.p after the fact, I would be able to lazily load code while also using the CDN for those chunks. Since lazy chunks contain the majority of my app, that'd be a pretty big win for me.
Is there any way to do that? I'm willing to write plugins if necessary. If not, would some kind of support for this be a worthwhile patch?

The first thing I tried (reassigning require.p in my entry points) made Webpack fairly unhappy:
'use strict';

import domReady from 'detect-dom-ready';
import asset from './App/asset';

// Use the Aerobatic CDN for loading assets. This must happen before calls to
// require.ensure.
require.p = asset(require.p);

domReady(() => {
    require.ensure([], (require) => {
        // use explicit module index as a concession to Mac OS developers. Stops "require('./App')" from resolving to
        // this very file.
        require('./App/index');
    });
});

Lead to this output:
Hash: a2779ed2f9e570813d32
Version: webpack 1.13.0
Time: 928ms
            Asset     Size  Chunks             Chunk Names
    app.bundle.js   898 kB       0  [emitted]  app
app.bundle.js.map  1.05 MB       0  [emitted]  app
  [80] ./js ^\.\/.*$ 622 bytes {0} [built] [2 warnings]
    + 243 hidden modules

WARNING in ./js/app.js
Critical dependencies:
15:0-7 require function is used in a way in which dependencies cannot be statically extracted
15:33-40 require function is used in a way in which dependencies cannot be statically extracted
 @ ./js/app.js 15:0-7 15:33-40

WARNING in ./js ^\.\/.*$
Module not found: Error: a dependency to an entry point is not allowed
 @ ./js ^\.\/.*$

WARNING in ./js ^\.\/.*$
Module not found: Error: a dependency to an entry point is not allowed
 @ ./js ^\.\/.*$

This is, somewhat notably, missing most of my chunks.

@ojacobson you can do this from backend like this:
assetsPath = process.env.DEBUG? 'local' : 'http://cdn.com/xxx';

<script src="{{assetsPath}}/index.js"></script>

@helloyou2012 I don't think there are official plans related to this. I marked the issue as "enhancement" if someone wants to try to tackle it, though.

You can set the __webpack_public_path__ at runtime:
__webpack_public_path__ = "http://cdn.example.com";

@sokra I should've realized that would be sufficient. It's not in the current docs, but it is mentioned in the Github docs - thanks for pointing me in the right direction!

@helloyou2012 Thanks for the suggestion. Unfortunately, Aerobatic provides their own server, and I don't have the option of running my own code on it. That's a pretty good workaround, otherwise!


C lexer comments + function definition/declaration bug fix

(Original issue 352 created by delroth on 2008-08-01T23:59:10+00:00)
This patch solves a bug with comments parsed as Token.Text with C lexer, when placed in function declarations or definitions.
Test case :
{{{
void f(void) /* comment KO */
{}
void g(void) /* comment KO */
{/* comment OK */}
void h(void) /* comment KO */;
f(5) /* comment OK */;
void f(void); /* comment OK */
int z /* comment OK */ =0;
enum /* comment OK */
{A,B};
}}}
Diff is attached, as usual.

(Original issue was assigned to gbrandl)
(Original issue 352 last updated on 2008-08-02T12:19:05+00:00)
(Issue automaticaly closed due to status in Bitbucket: resolved)

(Original comment by gbrandl on 2008-08-02T12:19:05+00:00)
Thanks, committed to main.


Okta support

This updates djangosaml2 to 0.15.0 to support okta without crashing. I've also updated the docs and changed all sample URLs to sal.example.com so it is easier to follow the README and settings.py, plus we can now do a find all and replace.
I have not tested this update with OneLogin.

So I signed up for a OneLogin dev account as I was gonna test this. Then I got lost in all the options OneLogin has...so I'm going to allow someone else test that.
You can use the following container if you want to test the above commit:
clburlison/sal-saml-test tag 3.1.1-okta.
I verified this on my local machine and a AWS production box so the Okta stuff should be 👍

Confirmed working in my development environment using Puppet deployments.

I've updated the instructions and tested the Crypt flavor of these changes. Seeing as how the only real different is djangosaml2 to 0.15.0. I'm pretty sure Sal will work with OneLogin as well.
Feel free to merge at your convenience. I know you've got a busy month.

I'm gonna give this a poke since @erikng tested this change at PSU last week. He took a current docker container and did a pip install djangosaml2==0.15.0 then restarted the container. Everything was 👍 for OneLogin.


IE, Edge multiple mentions

Hello there,
When trying to add multiple mentions in IE, Edge it works fine for 1-2 mentions, 3 mention then has wrong position and removes typed text.
https://www.dropbox.com/s/b89jr44llf316bl/bandicam 2015-08-31 14-39-36-808.mp4?dl=0
Seems like ranges problem.
Thanks.

I have the same issue. Any updates with this?

I don't know if this gets fixed by updating the Caret plugin. I see a merge there but that change hasn't been released. Are there any future plans to release it?


Backward compatibility between 3.0.0 release and beta 2 version

Hi.
It is possible there is some discussion about it. But i didn't find it.
Initialize method for objects in litecollection has been changed as shown below:
For beta 2 version:
Expression.Lambda(Expression.New(type)).Compile()
For release version
var dynamicMethod = new DynamicMethod("_", type, (Type[])null);
var il = dynamicMethod.GetILGenerator();
il.Emit(OpCodes.Newobj, type.GetConstructor(Type.EmptyTypes));
It is very well, but it is fatally  for internal or non-public types. Exception occured during reading collection from litedb.

Hi @Vakterman, changed because there are some problems using Expression in .net35 (I can't remeber now). Now I have 2 implementations: one for net35 (using emit) and one for netstandard (using expression).
I will review this and test if it's possible use CreateClass/CreateStruct use only Expression.

Hi @Vakterman, current HEAD version now complete remove Emit from solution in .net35 (I made a single solution for .net35 and netstadnard). I forgot to link to his issue. Take a look there

Thank you very much!
It works fine!!!


Read defaults from a .env file or an environment variable

Is your feature request related to a problem? Please describe.
When working on the project using Lagoon, I work from the directory of that project which sets some sort of context.
It would be great if I could avoid the need to provide the project name as -p every time I need to perform an operation in Lagoon. The project name could be read from a .env file or environment variable if it exists.
Describe the solution you'd like
Set LAGOON_PROJECT in .env and use that as default value for -p CLI option for all commands.
Describe alternatives you've considered
No alternatives to set the default value of the project that I am aware of.
Additional context
N/A

There is a feature for this that is disabled by default because it reads the .git directory information. It is not really recommended to use it as we may end up getting rid of it at some stage.
We have discussed though being able to use env variables to set some of these things instead of flags. But we would not be reading any files, only evaluating env vars


Converting path to posix style to prevent error on windows

Overview
I know that taskcat currently is not supported in Windows but I do use it on windows and only thing which currently stops me using it on windows is it creates wrong s3 URL for template after uploading it on s3 due to windows path format.
Eg. c:\user\project\template\my-template.yml when uploaded on s3, URL becomes https://s3.console.aws.amazon.com/s3/object/your-bucket-name/cf-project/templates\my-template.yml and hence give error that it is not correct s3 bucket URL.
In above example correct URL should be, https://s3.console.aws.amazon.com/s3/object/your-bucket-name/cf-project/templates/my-template.yml
Testing/Steps taken to ensure quality
I have run above change in my windows machine as well on mac to make sure it works

Any chance that submodules are still having this issue?

Shouldn’t be. I reopened the tracking issue so we can dig further.

#469 => #611


Examples are hard to recreate

 Issue by robbla
Wednesday Nov 29, 2017 at 10:38 GMT
Originally opened as https://github.com/ballerina-lang/composer/issues/4757

When we look at the examples, e.g. the JMS producer it is quite hard to figure out the steps as to get to the sample, in other words rebuilding it.
This is particularly so in:

which function first (dependencies)
use of predefined sub functions
dependencies and errors (e.g. the return statement needs a return value)

For instance how do we create this function from the graphical overview?
function getConnectorConfig () () {
     // We define the connection properties as a map. 'providerUrl' or 'configFilePath' and the 'initialContextFactory' vary according to        //the JMS provider you use.
     // In this example we connect to the WSO2 MB server.
    jms:ClientProperties properties = {   initialContextFactory:"wso2mbInitialContextFactory",
                                          configFilePath:"C:/ballerina-tools-0.95.0/bre/conf/jndi.properties",
                                          connectionFactoryName: "QueueConnectionFactory",
                                          connectionFactoryType : "queue"};
    return properties;

Sorry for taking so long to reply.
At the time of this issue is reported we had a drag and drop graphical editor. As of now, we have moved away from considering the graphical editor as a fully fledge program editor. Now the focus is that graphical editing will only allow you to create an initial flow of the program. ie. add the connectors, defining the basic flow of the code etc.
We will be closing this issue since it is no longer relevant, please do try new editing capabilities in Ballerina VSCode plugin and give us your feedback.


Unable to access iOS Photo Library

Godot version:
3.2 Beta 2
OS/device including version:
Pop OS 19.1
Issue description:
Unable to access iOS Photo Library
Steps to reproduce:
in iOS export settings, add a PhotoLibrary usage description

OS.request_permissions()
OS.get_system_dir(6) for user photos directory (same code works with android and desktop)
I have also tried using
'/private/var/mobile/Media/DCIM'
When I open a file dialog with that path, I get this path instead:
'/private/var/mobile/Containers/Data/Application/.......'

Minimal reproduction project:

Neither OS.request_permissions nor OS.get_system_dir is implemented on iOS.

I not sure if it's at all possible to access Photo library as filesystem, there's PhotoKit API for this (which is not used by Godot).

Ok, thanks for the clarification.  Is there a way to read / write external storage?  I'd be fine with documents or wherever, doesn't have to be photos.

Is there a way to read / write to external storage? I'd be fine with documents or wherever, doesn't have to be photos.

Add Support opening documents in place (LSSupportsOpeningDocumentsInPlace) = YES and Application that supports iTunes sharing (UIFileSharingEnabled) = YES to the {name}-Info.plist in the Xcode project.

And write/read your files to the user://Documents/ (create directory first, it won't be there for newly installed app).
Files should be available in macOS Finder/iTunes share (when connected via USB), and Files app on the device.

Thank you!!  I've been searching for this for months, trying it now!

It worked!  I can see the files in Files app on iOS.  I'm still having trouble using godot's file dialog though.  I'm using the same path (user://exports) but godot file dialog doesn't see it.  It goes to /private/var/mobile/Containers/Data/Application/... instead.  Is there a way to get file dialog working? Or a way to use iOS native file dialog

There's access property in the file dialog, set it to User Data to restrict it to user://...

Ah!  right.  I had it on file system for Android / Desktop.  Thank you!!!  This made my day

I'm able to see generated files in user:// with the iOS files app, but not from godot's file browser with access set to User data, and path set to user://  Is there something else I'm missing?
Also, ideally I would like to put files somewhere external/permanent so user paintings don't get deleted when the app is uninstalled.  Is there another info.plist option for external storage?

I'm able to see generated files in user:// with the iOS files app, but not from godot's file browser with access set to User data, and path set to user:// Is there something else I'm missing?
Also, ideally I would like to put files somewhere external/permanent so user paintings don't get deleted when the app is uninstalled. Is there another info.plist option for external storage?

use share module
IOS does not allow you to directly work with the file system.
This is against the rules of the apple
You can save your pictures to the photo library through a special API or this module
There is a special API PhotoKit

Can anyone still reproduce this bug in Godot 3.2.3 or any later release?


Tree Spirit animation with animation smoothing in NMZ

Tree Spirit glitches while using animation smoothing in NMZ possibly on quest too but not sure

This animation is no longer smoothed with the plugin as of client patch<PHONE_NUMBER>.80.


Create a bunch of example CFUs operations

Here is a list of example CFU operations the we thought would be interesting;

[x] Count of number of 1s
[ ] Integer Multiple

[ ] 32bit×32bit == 32bit
[ ] 32bit×32bit == 64bit

[ ] Integer Divide

Rounding modes?

[ ] Multiply + Accumulate - rs1×rs2+ra  → rd, ra
[ ] Divide + Remainder
[ ] Alpha blending of 32bit argb colors
[ ] SWAR / MMX style operations

[ ] two 32-bit integers,
[ ] four 16-bit integers, or
[ ] eight 8-bit integers at once

[ ] Saturation arithmetic

We should create implementations of each of the CFU operations and check they work with our CFU interface.

Added Popcount
Added MulAcc (acc += 32b x 32b) which is different from the above
32b x 32b -> 64b might be done with CFU_REQ_DATA_W=32 where CFU_RESP_DATA_W=64, or might be done withCFU_RESP_DATA_W=32 and CFU_RESP_OUTPUTS = 2.
This begs the question of how to handle CFU_RESP_OUTPUTS>1. Does the CPU accept the tuple of two outputs in one cycle, or does it receive them sequentially one at a time?


Allow leading underscore for unused variables

Fixes #14

Can we drop support for Node 4? Apr 30, 2018 was the official end of life date.

@mike-north Yeah it can be dropped 👍


Make CTX('api') available in UI context

The OXI::Template Plugins need access to API functions and use CTX('api') at the moment. This fails if those plugins are called from the Client::UI classes.
We need a transparent mapping so the CTX('api')->call is translated to a "send_command" using the socket. Fallback solution - provide API method to do TT processing on the server.

The new API provides a render_template call


i have an issue

this does't work with the https://duoling.com/ website bad plugin 0/10 wouldn't recommend

lol


fix: Correct random password lifecycle by eagerly creating random password value

Description

Correct random password lifecycle by eagerly creating random password value. Instead of linking the random password resource generation to whether or not the instance is a replica in the root, just let the random password eagerly create a value; whether or not the password value generated is used is handled by the db_instance sub-module regardless https://github.com/terraform-aws-modules/terraform-aws-rds/blob/master/modules/db_instance/main.tf#L8

Motivation and Context

Fix lifecycle dependency issue causing errors when trying to use replicas
Closes #386

Breaking Changes

No

How Has This Been Tested?

[x] I have tested and validated these changes using one or more of the provided examples/* projects

This PR is included in version 4.1.3 :tada:


Automatically apply schema changes on sinks when the parent table changes schema.

See #16834 for the background.
This proposal aims to enable the automatic synchronization of schema changes from source tables to destination tables, such as a StarRocks table, as illustrated below.
CREATE TABLE users (...);

CREATE SINK sink_users FROM users WITH (
  connector = 'starrocks',
  starrocks.table = 'sr_users',
  auto.schema.evolution = 'true'
);

-- The SR table `sr_users` will adapt this change.
ALTER TABLE users ADD COLUMN address VARCHAR;

When auto.schema.evolution is set to true, the sink_users sink will handle schema change events emitted from the users table by executing ALTER ADD COLUMN on sr_users in StarRocks.
Specifically,

When the schema change is ADD COLUMN, the sink will execute ALTER TABLE ADD COLUMN.
When the schema change is DROP COLUMN, the sink will update its local metadata by tagging the column as "dropped", but not removing it. Instead, it fills the column with null.
Other events like type changes will be omitted for now, since we don't yet support them.

When auto.schema.evolution is enabled, the downstream table is expected to be fully managed by RW. We need to warn users to be cautious when executing manual schema changes. From this viewpoint, we can also let auto.schema.evolution auto-create the table (see #15586) if absent, thus offering users a fully-automated ELT experience.
It should be noted that the E2E auto schema change is not allowed for SINK AS SELECT or SINK FROM <mv>, given that queries are not able to change schema. It's only available to SINK FROM <table> or <source>.

Generally I think this is doable but let's talk about one specific case: not all databases support instant ALTER TABLE ADD/DROP COLUMN like we do. If ALTER TABLE ADD/DROP COLUMN takes a long time (due to locking / large data volume / ...) in the external sink, is auto schema evolution still beneficial or should we automatically pause/resume the stream for schema evolution?

We don't have to provide this option for databases where ALTER TABLE is time-consuming.
Let's support it connector-by-connector. In the worst case, users can simply disable the option if the latency is not desired.

Generally I think this is doable but let's talk about one specific case: not all databases support instant ALTER TABLE ADD/DROP COLUMN like we do. If ALTER TABLE ADD/DROP COLUMN takes a long time (due to locking / large data volume / ...) in the external sink, is auto schema evolution still beneficial or should we automatically pause/resume the stream for schema evolution?

Discussed offline, auto schema change can be supported case by case. For example, if the downstream system is slow in ALTER statement, we won't support auto schema evolution on it. Note that for DROP COLUMN, we don't necessarily need to issue a ALTER DROP COLUMN statement and instead we can just sink NULL for the dropped column.

For better understanding, this task can be split into 2 parts.

The support of doing schema change (i.e. ADD COLUMN) in downstream.
The support of altering a streaming job, which is a SINK here.

For the 1st point, as mentioned in Patrick's comment, this work is largely case by case.

If the downstream is structured, had better to check the ADD COLUMN can be done instantly
If the downstream is semi- or unstructured e.g. JSON, no need to do anything
If the downstream is semi-structured and managed by schema registry, need to update the schema registry.

For the 2nd point, note that even for CREATE SINK sink_users FROM users, which might be the simplest form I can tell, it's still a streaming job comprising 2 or 3 executors:
MaterializeExecutor --> [ExhcnageExecutor]   -->    SinkExecutor
(table)          (only if sink_key != table_pk)

When a new column is added, all the channels and executors in the pipeline need to be aware and change accordingly.
This problem is exactly altering MV (no rebuilding) that we have discussed a lot before, but the boundary of the problem can be well-defined and fairly limited. cc. @BugenZhao

@fuyufjh  Do you think it's technically feasible?
Since the problem scope is greatly reduced, I think it is quite different from the alter-mv that we discussed before.

@fuyufjh Do you think it's technically feasible? Since the problem scope is greatly reduced, I think it is quite different from the alter-mv that we discussed before.

From my understanding, the essential work is similar: changing the definition of the exsiting actors


FranklinDAO Research Proposal

Penn Blockchain/FranklinDAO Research
Grant category
Please select one:

Educational material

Description
We hope to research a robust analysis of Flow that highlights its fundamental workings and future opportunities or research areas. Our findings would then be published in long-form format, Twitter threads, as well as in media formats such as podcasts and social media video clips, by FranklinDAO.
The first part of our analysis would be a quantitative and qualitative dive into how Flow works. This would begin with a high-level overview of the history, goals, and basic functionality of Flow, before launching into a rigorous explanation that explores each detail of functionality. Quantitative details through on-chain analysis and data oracles like Dune would provide additional support to the impact of Flow and the design choices made by developers. The goal of this section is to go beyond the original white paper by providing an additional analysis of how design choices have shaped out in practice once deployed on-chain.
The second part of our analysis would be a gaze into the future of Flow. We would give recommendations for areas of improvement in future versions of Flow and findings for how Flow fits into the broader picture of Web3—in particular, how it is important to trends and innovations in the spotlight at the moment, such as developer and user-friendliness for Web2 onboarding. This portion of our analysis is slightly more subjective, and we would tailor it to highlight the strengths of members of our research team. For example, in the past, we have produced research on the philosophy between App-specific Rollups versus Chains for scaling; this is something that we could connect to the future of Flow and the benefits it offers to consumer-wide applications in a sustainable manner.
The final section of our project is releasing the publication in a variety of media formats to maximize the traction that it gains. Like our previous research, we typically release in the format of a long-form piece, Twitter thread, podcast, and social media clips. These are published by our FranklinDAO accounts, as well as the accounts of our individual researchers.
Problem statement
One of the biggest bottlenecks to growth is the bootstrapping of new users and earning the trust of users in the underlying technology or governance. This is especially profound in Web3, as many Web3 users are onboarded through very niche channels — we haven’t seen waves of the average Web2 user become actively involved in this space — and due to previous exploits in Web3 that have left consumers rightfully doubtful of certain technologies. Being an EduDAO school, our target audience is undergraduate innovators at top universities (Harvard, MIT, Berkeley, Oxford) around the world and students looking to adopt new technology. We hope that our work will compel these groups to become more actively involved in Flow, whether as a developer or user.
About
Penn Blockchain/FranklinDAO is the student-run Blockchain organization at the University of Pennsylvania. Our team regularly publishes research, investment theses, and governance recommendations and hosts hackathons and conferences to promote the education of Web3 across campus and in the greater Philadephia area. Many of our past & present leadership, both undergraduate and graduate students, have been inspired by their involvement in FranklinDAO to work on Web3 ventures full-time and are prominently involved in the ecosystem.
In particular, our research team is comprised of students from both the School of Engineering and Wharton School with diverse interests in Web3. This past semester, we published long-form pieces on crypto regulation, the thesis behind app chains versus rollups, undercollateralized lending, maximum extractable value, and decentralized identity. We have also become more involved with promoting our work to the mainstream public by recording podcasts and social media clips.
Here is our DAO Substack: https://franklindao.substack.com/ (consider subscribing!) and work samples: https://tinyurl.com/2p9wcddr
The FranklinDAO Research team is led by Erik Zhang and Cindy Jiang, who are both undergraduate students at Penn. Erik is a Pear VC Fellow and incoming summer analyst at Bridgewater and, and Cindy previously interned at Galaxy Digital’s ventures team.
Other members of the team have majors ranging from Computer Science, Mathematical Economics, Data Analytics, Politics, Philosophy, and Economics, as well as Finance and Business Analytics. We’re proud to have a team that can produce both rigorous quantitative and qualitative research.
Impact
The goal of our project is two-fold. The first goal is to provide a meaningful analysis of the state and future of Flow As stated above, our team will rigorously dive into Flow and its analytics to paint a multi-dimensional picture of its design and provide recommendations for why Flow is important and where it should focus on in the future. We take pride in the quality of our analysis and are confident in the quantitative and qualitative soundness of our findings, and we believe that our research will be helpful to developers and the governance community moving forward as Flow continues to grow.
The second goal is to promote Flow from both an engineering and investment standpoint, given the skillset make-up of our research team. By producing a technically advanced explanation of how Flow works and justification for the design choices in the protocol, we hope that this will increase the confidence of users in the ecosystem to rely on Flow or find opportunities unavailable on other platforms. Our analysis into why Flow relates to key trends in Web3 supports investment theses into Flow and provides additional support into Flow’s utlization in the future.
Beyond the quality of our analysis, we believe that our position as a fully-student run organziation puts us in a unique position to make an impact on those that follow our content. As a member of EduDAO, our biggest audience are other college blockchain enthusiasts at EduDAO schools (Berkeley, Harvard, MIT, Oxford) or Penn students. Both of these populations are rife with talent in a multitude of skillsets, and given the age demographic, are most likely to be first mover adopters of decentralized applications.  As such, we have an impact to reach not only other Web3 innovators, but also students who are not familiar with Flow but would be able to make a solid contribution to the community.
Milestones and funding

Note: Please consider adoption and/or maintenance milestones at the end of your project.

Milestone
Deliverables
Timeline
Risks
USD proposal

1 - Sourcing Areas of Interest
One-pager rundown
1 week
-
100

X - Technical + Quantitative Research
Draft of long-form paper
2.5 weeks
-
1000

X - Future Opportunity Research
Pitch Deck with Recommendations for Flow moving forward
2.5 weeks
-
1000

X - Media Creation and Finalization
Podcast, Social Media Clips, Twitter Threads, Final Draft
1.5 weeks
-
500

Total funding proposed: $2750

Team

Name
Role
Bio
Contact

Erik Zhang
Co-director
Pear VC Fellow, Incoming Summer Analyst at Bridgewater
<EMAIL_ADDRESS>

Cindy Jiang
Co-director
Formerly @ Galaxy Ventures Team
<EMAIL_ADDRESS>

Hi @austin-yao, we reviewed your proposal in detail and decided to pass because it is unclear if the impact on the developer ecosystem will be high enough.
Thanks for taking the time to submit your proposal and feel free to reach me directly on Discord in case you have further questions: alx-flw.find#6198.


Memory usage

I noticed that if I opened a sample PDF, closed it, opened a different sample, and repeated the process that the memory usage continually increased.
So I used the PdfiumViewer.Demo project and the only change I made was to make the project use .NET Framework 4.6.1 rather than the default so that I could use the built in memory diagnostic tool in Visual Studio.  The same problem happened there as well.
Is there something I should do to properly dispose items once finished viewing a given file?

The problem most likely is that the PdfDocument isn't disposed. The viewer doesn't do this for you and you're supposed to do this yourself. Easiest way to do this is to add the following line in the constructor of your form:
Disposed += (s, e) => pdfViewer1.Document.Dispose();

Or to make it a little bit nicer:
Disposed += (s, e) =>
{
    var document = pdfViewer1.Document;
    if (document != null)
        document.Dispose();
};

Of course, whenever you assign to pdfViewer1.Document, you also have to make sure you dispose the existing one:
private void OpenDocument(PdfDocument document)
{
    if (pdfViewer1.Document != null)
        pdfViewer1.Document.Dispose();

    pdfViewer1.Document = document;
}

still not disposing when i load from stream
Dim pdf_stream As Stream = stream
Dim pdfdoc = PdfDocument.Load(pdf_stream)
_pdf = pdfdoc
Dim f As New PdfViewer
f.Parent = pnlPreview
f.Dock = DockStyle.Fill
If f.Document IsNot Nothing Then
f.Document.Dispose()
End If
f.Document = PdfDocument.Load(pdf_stream)
f.Visible = True
even i use the dispose you provided still not working..this only works if i clear the parent panel..but i don't want to clear the panel, because the viewer is already loaded, please help


Fetch Service Manager plans in multiple goroutines

When getting plans from Service Manager for particular broker, service offerings IDs are used as query parameters. If there are many offerings in one service broker this query may become very long and break the functionality.
Also the requests for each broker can be done in parallel.

Fixed by #129


Show names of entered ECO codes

After ECO codes have been added, show the codes and the actual names so that the user can verify the codes were correctly entered.

This was intended for CTD2.


[Parameter Request] - codeunit 7322 "Create Inventory Pick/Movement" OnBeforeFindFromBinContent

Could you please add the parameters  DefaultBin: Boolean and WhseItemTrackingSetup: Record "Item Tracking Setup" to the event OnBeforeFindFromBinContent?
local procedure InsertPickOrMoveBinWhseActLine(NewWhseActivLine: Record "Warehouse Activity Line"; BinCode: Code[20]; DefaultBin: Boolean; var RemQtyToPickBase: Decimal; WhseItemTrackingSetup: Record "Item Tracking Setup")
var
    FromBinContent: Record "Bin Content";
    QtyToPickBase: Decimal;
    QtyAvailToPickBase: Decimal;
    IsHandled: Boolean;
begin
   ...

    with FromBinContent do begin
        SetCurrentKey(Default, "Location Code", "Item No.", "Variant Code", "Bin Code");
        SetRange(Default, DefaultBin);
        SetRange("Location Code", NewWhseActivLine."Location Code");
        SetRange("Item No.", NewWhseActivLine."Item No.");
        SetRange("Variant Code", NewWhseActivLine."Variant Code");

        ...

        //+Change
        //-OnBeforeFindFromBinContent(FromBinContent, NewWhseActivLine, FromBinCode, BinCode, IsInvtMovement, IsBlankInvtMovement);
        OnBeforeFindFromBinContent(FromBinContent, NewWhseActivLine, FromBinCode, BinCode, IsInvtMovement, IsBlankInvtMovement, DefaultBin, WhseItemTrackingSetup);
        //-Change
       ...

end;

[IntegrationEvent(false, false)]
//-local procedure OnBeforeFindFromBinContent(var FromBinContent: Record "Bin Content"; var WarehouseActivityLine: Record "Warehouse Activity Line"; FromBinCode: Code[20]; BinCode: Code[20]; IsInvtMovement: Boolean; IsBlankInvtMovement: Boolean) //OLD
local procedure OnBeforeFindFromBinContent(var FromBinContent: Record "Bin Content"; var WarehouseActivityLine: Record "Warehouse Activity Line"; FromBinCode: Code[20]; BinCode: Code[20]; IsInvtMovement: Boolean; IsBlankInvtMovement: Boolean; DefaultBin: Boolean; WhseItemTrackingSetup: Record "Item Tracking Setup") //NEW
begin
end;

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.


fix: update to deployment to serial flow

:tada: This PR is included in version 0.2.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


WIP [DNM]: generate arch.xml from verilog templates including timing info

This PR is to track and discuss the timing data import into arch.xml file. The idea is to update all the BEL sim.v files to include timing templates (see https://github.com/antmicro/symbiflow-arch-defs/blob/25cb1d212d4890fb2a22a5576a56ec3a0e50fd05/xc7/primitives/common_slice/Nlut/ntemplate.Nlut.sim.v)
The script (https://github.com/antmicro/symbiflow-arch-defs/blob/25cb1d212d4890fb2a22a5576a56ec3a0e50fd05/utils/ntemplate_timings.py) can update such file with data read from sdf file.
Next, v2x tool can be used to generate xml file.
Initial tests are promising. The script called with:
ntemplate_timings.py --template ntemplate.Nlut.sim.v --sdf CLBLL_L.sdf --npattern a alut.sim.v  --bel LUT5 --bel LUT6 --process slow --delay max --instance SLICEL/A5LUT --instance SLICEL/A6LUT

the following verilog file - https://pastebin.com/TSffCRKZ
v2x generated the following xml - https://pastebin.com/SmbucBn9

It might be better / more flexible to do this at the XML level rather than the verilog level?

It might be better / more flexible to do this at the XML level rather than the verilog level?

There was some brief discussion(logs here) on which way to take on this.

Relevant conversation

user
msg
time

sf-slack
 @litghost @mithro I'm looking at the possibilities of adding the timing information to architecture definition and there seem to be two options
14:05

sf-slack
 one is to implement a script to read the arch.merged.xml file and add/update timing entries basing on sdfs generated with timining fuzzer
14:06

sf-slack
 this option is pretty straight forward - it will add one step between xml merging and carry chains specialization
14:07

sf-slack
 I'm not sure if this scales, though
14:07

sf-slack
 the second option is to generate all the arch xmls from verilog (this is something @mithro mentioned some time ago)
14:08

sf-slack
 this is much more work, because we'd need to reimplment all the arch.xml generation
14:09

sf-slack
 but this would allow us to create verilog templates for every BEL and CLB structure
14:09

sf-slack
 a separate script can use the verlilog templates and merge them with timings from sdf
14:10

sf-slack
 in the end v2x tool can generate arch xml
14:10

sf-slack
 this approach is much more versatile, but requires more work
14:11

sf-slack
 what do you think
14:11

sf-slack
 I wonder whether it is possible in the second option to include the timings in Verilog models and use them for simulation. (just a random thought)
14:16

litghost
hackerfoo: What changed?  I'm also working on writing a testbench for the FF SR/CE test should we should probably commonize some things, at a minimum an assert of some kind
14:55

litghost
kgugala: I think we should proceed with a v2x solution, with the follow caveat: Let's write tests as we god
14:58

litghost
go*
14:58

litghost
kgugala: I am working on some FF tests, which also does a little combintorial testing
14:58

I can see the value in both approaches. Using v2x may mean expanding the attributes to support the different timing information and will require a lot more work. Also it's not clear to me what we will do with interconnect timing information.
@kgugala What is the concern with scaling in patching the xml? If we have to do that during routing import/generation, may be we should try to be consistent.

We can patch XMLs using similar approach like I did for verilog - create a bunch of templates and just fill them with data extracted from sdfs.
@elmsfu  v2x claims it can handle a lot of different parameters, so it shouldn't be a problem (see: https://github.com/SymbiFlow/symbiflow-arch-defs/blob/master/utils/vlog/vlog_to_pbtype.py)
I think interconnects will have to be updated directly in xmls.


Accuracy drop on pole mounted view images

@chufengt  i tested the pre-trained model on eye-level captured videos/images and the results are acceptable but with little change, in the camera view ie pole-mounted view images the accuracy drops considerably through RAP dataset has surveillance images captured at the pole-mounted view, is re-training the only approach to get the best accuracy ??

It seems the generalization between different scenarios is not good.
I think re-training on your custom dataset may work better.


docs: add info for webhook signature validation

This was requested by Radu. He and Max may be good resources if there are questions.
If an ASE wants to use a signature, the SIGNATURE_SECRET environment variable is optional, so the ASE can opt in or opt out. We should point this out in the Admin API(s) and somewhere in the Webhook Events page, rather than making signatures its own page.
For the webhooks, Radu suggested something like this as well as code snippets.

Go to https://docs.stripe.com/webhooks?lang=node&verify=verify-manually#verify-official-libraries
Click the Verify Manually tab

The steps in the Stripe doc are:

Extract the timestamp and signatures from the header (in our case the timestamp is in the Rafiki-Signature header)
Prepare the signed_payload string (in our case, the payload string is the request body [the data Rafiki sends to the ASE])
Determine the expected signature
Compare the signatures

Adding this content to the Webhook Events page before adding to any of the API pages. Adding @raducristianpopa and @mkurapov to the PR to verify this information first.
Need to learn more about how (or if) signatures are used across Rafiki for other services and if the logic is the same. Looks like Auth Admin API has its own signature variables as well.


Add batch processing support for large collections ...

Perhaps, something like collection.EachAsync(callback func, batchSize int) where it cuts up the work in chunks and invokes the callback as a go routine with appropriate wait groups.

Implemented as of release v1.0.9.


Recentre map when adding a new marker

@bdvholmes I believe this can be closed - didn't put a closes/fixes on my PR.

Good spot, thanks.


what is "treshold" work for?

filt_mask = logits_filt.max(dim=1)[0] > threshold
The provided code filters out the boxes detected by GroundingDino (by prompt) before passing them to SAM for detailed masking. Specifically, it creates a binary mask (filt_mask) based on the condition that the maximum value along the second dimension of the logits_filt tensor is greater than the specified threshold. Boxes meeting this condition are retained in the filtering process.


Prepare for release v0.8.1

@spikebrehm @lelandrichardson

LGTM!

@spikebrehm
Got this
I try to remove node_modules and reinstall by npm i, but got the same
Failed to build DependencyGraph: @providesModule naming collision:
  Duplicate module name: String.prototype.es6
  Paths: /Users/stereodenis/work/stereopark/PshPshMobile/node_modules/react-native/packager/react-packager/src/Resolver/polyfills/String.prototype.es6.js collides with /Users/stereodenis/work/stereopark/PshPshMobile/node_modules/react-native-maps/example2/node_modules/react-native/packager/react-packager/src/Resolver/polyfills/String.prototype.es6.js

This error is caused by a @providesModule declaration with the same name across two different files.
Error: @providesModule naming collision:
  Duplicate module name: String.prototype.es6
  Paths: /Users/stereodenis/work/stereopark/PshPshMobile/node_modules/react-native/packager/react-packager/src/Resolver/polyfills/String.prototype.es6.js collides with /Users/stereodenis/work/stereopark/PshPshMobile/node_modules/react-native-maps/example2/node_modules/react-native/packager/react-packager/src/Resolver/polyfills/String.prototype.es6.js

This error is caused by a @providesModule declaration with the same name across two different files.
    at HasteMap._updateHasteMap (/Users/stereodenis/work/stereopark/PshPshMobile/node_modules/react-native/packager/react-packager/src/node-haste/DependencyGraph/HasteMap.js:155:13)
    at module.getName.then.name (/Users/stereodenis/work/stereopark/PshPshMobile/node_modules/react-native/packager/react-packager/src/node-haste/DependencyGraph/HasteMap.js:115:31)

@stereodenis oh no! My bad.  I'll publish 0.8.2 to fix this.
https://github.com/lelandrichardson/react-native-maps/pull/558

Just published 0.8.2.


mayactl should show if the volume has multiple clients connected to it

Refer: #1278
There was a corner case (bug) in the earlier versions of the K8s cluster where the same target was being logged in from multiple clients (k8s nodes) as the pods associated with the target shifted across the nodes. For better debuggability/diagnosing it will help to have the mayactl provide information on the clients connected to it.

i am working on this


signpsbt: update to new signpsbt RPC

Update the use of signpsbt to proposed update on CLN.
Requires CLN PR https://github.com/ElementsProject/lightning/pull/6858

Thanks, looks like a make fmt is missing

There are any progress on this?

Ok looks like that this is not longer needed? https://github.com/ElementsProject/lightning/pull/6858#issuecomment-1957741658


Unable to train with custom dataset

Hello! Thank you for your great work with this impl. I am trying to train a model on a custom dataset (different language).
I didn't use MFA, I prepared data in LJSpeech format and edited dataset.py and preprocessing scripts. I didn't change symbols.py, however I use transliteration_cleaners before.
Original Traceback (most recent call last):
  File "/home/george/anaconda3/envs/reface_tts/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/home/george/anaconda3/envs/reface_tts/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/george/work/FastSpeech2/model/fastspeech2.py", line 81, in forward
    ) = self.variance_adaptor(
  File "/home/george/anaconda3/envs/reface_tts/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/george/work/FastSpeech2/model/modules.py", line 121, in forward
    x = x + pitch_embedding
RuntimeError: The size of tensor a (79) must match the size of tensor b (620) at non-singleton dimension 1

seems that something wrong with the pitch embedding and duration. As I didn't use mfa I set duration to some hardcoded value and didnt trim pitch array at preprocessing stage. How to properly setup data for training?
Thank you

@thepowerfuldeez  Maybe you have to make sure that the sum of the durations equals the length of the mel-spectrogram

@ming024 you are right, but where can i get these durations from?

@thepowerfuldeez Using MFA is one possible solution, or you can try to use the alignment of another TTS model (e.g. Tacotron 2) trained on your desired dataset, just like proposed in FastSpeech. You can refer to this implementation

@ming024 okay, i might try to use tacotron2 for alignment. Thank you! just thought maybe for ground truth durations there is a better way.

Hello! Thank you for your great work with this impl. I am trying to train a model on a custom dataset (different language). I didn't use MFA, I prepared data in LJSpeech format and edited dataset.py and preprocessing scripts. I didn't change symbols.py, however I use transliteration_cleaners before.
Original Traceback (most recent call last):
  File "/home/george/anaconda3/envs/reface_tts/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/home/george/anaconda3/envs/reface_tts/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/george/work/FastSpeech2/model/fastspeech2.py", line 81, in forward
    ) = self.variance_adaptor(
  File "/home/george/anaconda3/envs/reface_tts/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/george/work/FastSpeech2/model/modules.py", line 121, in forward
    x = x + pitch_embedding
RuntimeError: The size of tensor a (79) must match the size of tensor b (620) at non-singleton dimension 1

seems that something wrong with the pitch embedding and duration. As I didn't use mfa I set duration to some hardcoded value and didnt trim pitch array at preprocessing stage. How to properly setup data for training?
Thank you

Add phonemes/IPA in cum_dict.py file.

I have checked the sum of mel and durations, all of are same, total also same.
Found 22832 matching duration and mel files
Validating files: 100%|█████████████████████████████████████████████████████████████████| 22832/22832 [00:11<00:00, 1942.88it/s]
Validation Results:
Total files checked: 22832
Files with mismatches: 0
No mismatches found! All durations match mel-spectrogram lengths.
But the issue is actual
File "/home/george/work/FastSpeech2/model/modules.py", line 121, in forward
x = x + pitch_embedding
RuntimeError: The size of tensor a (79) must match the size of tensor b (620) at non-singleton dimension 1


[$250] mWeb - Compose box- Selected text isn't applied format

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: 9.0.67-0
Reproducible in staging?: Y
Reproducible in production?: Y
If this was caught on HybridApp, is this reproducible on New Expensify Standalone?: Y
**If this was caught during regression testing, add the test name, ID and link from TestRail:**N/A
Email or phone of affected tester (no customers<EMAIL_ADDRESS>Action Performed:

Open https://staging.new.expensify.com/
At compose box of any conversation, type 1 line
Select all text, choose Format: Bold and send message
Type another line
Select all text, choose Format: Italic and send message
Type another line
Select all text, choose Format: Underline and send message

Expected Result:
Selected texts should apply choosing format
Actual Result:
Selected texts isn't applied choosing format
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android: Standalone
[ ] Android: HybridApp
[ ] Android: mWeb Chrome
[ ] iOS: Standalone
[ ] iOS: HybridApp
[x] iOS: mWeb Safari
[ ] MacOS: Chrome / Safari
[ ] MacOS: Desktop

Screenshots/Videos

Add any screenshot/video evidence

https://github.com/user-attachments/assets/40872591-76a4-4b44-9383-0249f7fd97a4
View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~021861616540479439769
Upwork Job ID:<PHONE_NUMBER>479439769
Last Price Increase: 2024-11-27

Issue OwnerCurrent Issue Owner: @mananjadhav

Proposal
Please re-state the problem that we are trying to solve in this issue.
Selected text isn't formatted when selecting format option from the context menu.
What is the root cause of that problem?
When we have an element with contenteditable, the context menu will show the "Format" option. However, the live markdown currently doesn't handle the format command.
What changes do you think we should make in order to solve the problem?

There are 3 "Format" options, Bold, Italic, and Underline. Currently, we don't have a markdown for Underline. To handle the Bold and Italic format, we need to add new cases for bold and italic format command here.
https://github.com/Expensify/react-native-live-markdown/blob/cec2feae7463398b8d377a231be8671e1698e4a0/src/MarkdownTextInput.web.tsx#L349-L358
case 'formatBold':
  newInputUpdate = parseText(
    divRef.current,
    `${parsedText.slice(0, contentSelection.current.start)}*${parsedText.slice(contentSelection.current.start, contentSelection.current.end)}*${parsedText.slice(contentSelection.current.end)}`,
    processedMarkdownStyle,
    newCursorPosition + 2,
    true
  );
  break
case 'formatItalic':
  newInputUpdate = parseText(
    divRef.current,
    `${parsedText.slice(0, contentSelection.current.start)}_${parsedText.slice(contentSelection.current.start, contentSelection.current.end)}_${parsedText.slice(contentSelection.current.end)}`,
    processedMarkdownStyle,
    newCursorPosition + 2,
    true
  );
  break

The way the format logic work is by "wrapping" the selected text with the markdown (* for bold, _ for italic). The new selection will be added by 2 because we add 2 * or _.
But the Underline format still won't do anything because we don't have a markdown for that, and because the live markdown also handles the styling manually, so the alternative is to disable the "Format" option. We can do that by setting the contenteditable to "plaintext-only"
https://github.com/Expensify/react-native-live-markdown/blob/cec2feae7463398b8d377a231be8671e1698e4a0/src/MarkdownTextInput.web.tsx#L724
contentEditable: disabled ? 'false' : 'plaintext-only',

Based on the doc, "plaintext-only" is available on most browsers except Firefox, but this Format context menu is only available on Safari, so I think it's not a problem.
https://developer.mozilla.org/en-US/docs/Web/HTML/Global_attributes/contenteditable#browser_compatibility

Bumped @mananjadhav for review here.

I checked @bernhardoj's proposal. I think we can go ahead with this.  But before we finalize I want to understand if we could remove the Underline option from the Format menu? (Atleast on Whatsapp I don't see it), whereas Slack removes the format option for me.
Nevertheless we can use @bernhardoj's proposal as they've mentioned the alternative for removing the Format menu as well.
Tagging @Expensify/design for the input as well.
🎀 👀 🎀 C+ reviewed.

Oh interesting, I didn't even know you could do this from iOS. I suppose we could make it so that whatever style you picked from the native menu (bold, italic, etc) was applied via our markdown styles, as in we'd prepend and append a * if you picked bold. Is something like that possible?

That'd be great if that's possible. If not, I wonder if we could hide or turn off that formatting menu altogether.

I'm fine with making just bold and italic work or hiding that format menu item altogether. I also didn't even realize it was there haha.

Thanks for the confirmation. Opened the PR on live markdown repo here: https://github.com/Expensify/react-native-live-markdown/pull/556
Waiting for the review from the live markdown team.

However, this PR only implements Format menu on web. I believe the same feature is missing on native iOS. Are there any plans to implement this missing feature also in native iOS?

@bernhardoj That's a good question raised here. Do we not need it for native?

Replied on the PR

Keeping the conversation here to understand the scope.  @bernhardoj I can see Format option in the native iOS Whatsapp. When clicked these are the options I see. This is what I also meant if we could hide just the underline. This translates to markdown btw.

But it's not available on our app, so I'm not sure how we can handle it.

Issue is assigned, automation didn't work quite right for some reason. Conversation is active in the PR.

The conversation is the in Expensify/react-native-live-markdown repo. May be that's why.

Looks like we're waiting on reviews on the PR @bernhardoj @mananjadhav?

Yes, I bumped on the PR

Not overdue

Waiting on reviews, bumped the agency reviewers in the PR.

Still waiting on reviews. Holidays are slowing down the review process for this one I think.


patch 2.4.0-palantir.7 to make spark-submit in-process safe

@mccheah @robert3005 hmm are the circle tests not running because i'm not merging to master? or because i'm pushing from my fork? i currently don't have permissions to push branches to palantir/spark :(

it's a fork thing.

closing in favor of https://github.com/palantir/spark/pull/378


Dynamic ports opened on loopback address by kafka consumer

Description
Dynamic ports opened on loopback address by kafka consumer. I have created a sample application to consume some messages from kafka. One observation is that it is opening multiple tcp connections on loopback address <IP_ADDRESS> as below
TCP    <IP_ADDRESS>:64599        <IP_ADDRESS>:64600        ESTABLISHED
TCP    <IP_ADDRESS>:64600        <IP_ADDRESS>:64599        ESTABLISHED
Here is the sample using dotnet kafka client
var config = new ConsumerConfig
{
BootstrapServers = "myservername:9092",
GroupId = "foo",
AutoOffsetReset = AutoOffsetReset.Earliest
};
        List<string> topics = new List<string>();
        topics.Add("mytopicname");
        using (var consumer = new ConsumerBuilder<Ignore, string>(config).Build())
        {
            consumer.Subscribe(topics);
            while (true)
            {
                var consumeResult = consumer.Consume(10);

                // handle consumed message.
            }

            // consumer.Close();
        }

How to reproduce
Code provided in description
Checklist
Please provide the following information:

[ ] A complete (i.e. we can run it), minimal program demonstrating the problem. No need to supply a project file.
[ ] Confluent.Kafka nuget version.
[ ] Apache Kafka version.
[ ] Client configuration.
[x] Operating system.
[ ] Provide logs (with "debug" : "..." as necessary in configuration).
[ ] Provide broker log excerpts.
[ ] Critical issue.

I believe librdkafka does this on Windows as a mechanism to enable polling of network and internal event queue simultaneously.

Do we have an idea how many connections are created per topic? As I can see for above code shared in post, 3 unique connections were created on loopback address.
In our actual environments, ratio per topic is even more.

Hi @mhowlett , @edenhill , is the same mechanism happening on producer side as well? We see a lot of such connections at localhost, when we produce large amount of messages. And they seem to be in peer. Such as below.
  TCP    <IP_ADDRESS>:62503        <IP_ADDRESS>:62504        ESTABLISHED     5016  [dotnet.exe]   TCP    <IP_ADDRESS>:62504        <IP_ADDRESS>:62503        ESTABLISHED     5016  [dotnet.exe]   TCP    <IP_ADDRESS>:62505        <IP_ADDRESS>:62506        ESTABLISHED     5016  [dotnet.exe]   TCP    <IP_ADDRESS>:62506        <IP_ADDRESS>:62505        ESTABLISHED     5016  [dotnet.exe]   TCP    <IP_ADDRESS>:62507        <IP_ADDRESS>:62509        ESTABLISHED     5016  [dotnet.exe]   TCP    <IP_ADDRESS>:62508        <IP_ADDRESS>:62510        ESTABLISHED     5016  [dotnet.exe]   TCP    <IP_ADDRESS>:62509        <IP_ADDRESS>:62507        ESTABLISHED     5016  [dotnet.exe]   TCP    <IP_ADDRESS>:62510        <IP_ADDRESS>:62508        ESTABLISHED     5016  [dotnet.exe]   TCP    <IP_ADDRESS>:62511        <IP_ADDRESS>:62512        ESTABLISHED     5016  [dotnet.exe]   TCP    <IP_ADDRESS>:62512        <IP_ADDRESS>:62511        ESTABLISHED     5016  [dotnet.exe]   TCP    <IP_ADDRESS>:62514        <IP_ADDRESS>:62515        ESTABLISHED     5016  [dotnet.exe]   TCP    <IP_ADDRESS>:62515        <IP_ADDRESS>:62514        ESTABLISHED     5016  [dotnet.exe]   TCP    <IP_ADDRESS>:62516        <IP_ADDRESS>:62517        ESTABLISHED     5016  [dotnet.exe]   TCP    <IP_ADDRESS>:62517        <IP_ADDRESS>:62516        ESTABLISHED     5016  [dotnet.exe]   TCP    <IP_ADDRESS>:62518        <IP_ADDRESS>:62519        ESTABLISHED     5016  [dotnet.exe]   TCP    <IP_ADDRESS>:62519        <IP_ADDRESS>:62518        ESTABLISHED     5016  [dotnet.exe]
We are running producer on Windows platform.
There are so many of them, so that we have local ports exhausted. Is there any way to improve the situation?
Thanks for your help.

That's fixed in the upcoming 1.9.0 release, which replaces the internal loopback connections with proper WSA scheduling.

Hi @edenhill great to see this will be addressed in 1.9.0 release, before that is there any producer config that can mitigate it? for example, queue size, batch size, or lingerMs, ...? thank you.

no. except that the number of broker poll loops in librdkafka is a function of the number of brokers in the cluster, so less brokers will result in less local tcp connections.

got it, thanks

hi @mhowlett @edenhill ,when will 1.9.0 be released?


Integrate Remote Config

Integrate Firebase engagement tool remote config

I would like to work on this issue.

@DevilAJ 'll assign this to you as soon as the Firebase integration is merged.

What's the update on this @himanshusharma89 ?

We can work on this now.

What's the update on this @himanshusharma89 ?


[SETUP] Using Poetry to manage the dependencies

Poetry is a dependence manager.
Poetry allows the same development environment for all developers and organizing the project on a pyproject.toml file.
After install Poetry you can run the following command to install all dependencies on a Python's virtual environment:
$ poetry install

To activate the virtual environment you can run:
$ poetry shell

To add a dependency you can run:
$ poetry add [PACKGE_NAME]

To add remove a dependency you can run:
$ poetry remove [PACKGE_NAME]

isnt a standard option use VENV @felipemarkson ? i never used this...

isnt a standard option use VENV @felipemarkson ? i never used this...

Actually Poetry use a virtual environment to manager the dependencies, and it is a option to Pipenv and follows the PEP 517

awesome,  lets discuss with the others...


draft: [FLINK-17971] [Runtime/StateBackends] Add RocksDB SST ingestion for batch writes

WIP
I was hoping to get a first round of feedback for this implementation. This branch is currently passing tests, but there was additional work to clean it up:

[x] More testing of the new implementation.

There are plenty of tests in the branch that all pass! We could do testing with an application if necessary, but this seemed sufficient.

[ ] Add benchmarking to RocksDB for the two implementations used here.
[x] Add more configuration parameters to pass through to the writers.

The configuration for which writer to choose was added.
[ ] Now that there is an optionsContainer availalbe, the other RocksDB options should be passed through via that.

[ ] Compare performance of save-point recovery on our production instanace, as it has a large state.

What is the purpose of the change
This adds a new mode of bulk writing data into RocksDB, via RocksDBSSTIngestWriter. This implementation should provide a considerable performance improvement to operations such as save-point recovery and state migrations (but, has yet to be perf tested).
This is in reference to the discussion on the users maillist I brought up, here, as well as the previously reported ticket: FLINK-17288. I created FLINK-17971 for specifically SST ingestion bulk loading.
Brief change log
(commit 1)

Refactored the use of RocksDBWriteBatchWrapper into using a factory (RocksDBWriterFactory) and interface (RocksDBWriter), in preparation of adding a second implementation.

(commit 2)

Added RockSDBSSTWriter, which is a basic wrapper for SstFileWriter in order to create sst files.
Added RocksDBSSTIngestWriter, which uses the RockSDBSSTWriter, and provides a write-interface for batch writing k/v into RocksDB. This includes flushing and handling multiple column-families.
Added new configuration for opting into the writer, as well as tuning parameters.
Configuration was passed through RocksDBWriterFactory. When the configuration is set to use sst ingestion (WriteBatchMechanism.SST_INGEST), the factory's defaultPutWriter will provide an SST ingestion writer rather than RocksDB's WriteBatch.
Write paths doing bulk loading (full savepoint recovery, state migration) are changed to use defaultPutWriter, so they will use RocksDBSSTIngestWriter if configured.

Verifying this change
This change is already covered by existing tests, such as:

All existing RocksDB savepoint and checkpoint tests. I added all the code behind a configuration option so both paths can continue to be executed. The RocksDB tests are parameterized to test both the WriteBatchWrapper (current) and SstFileWriter (new) ways of writing bulk put operations, and for a few tests duplicated them to trigger both paths.

This change added tests and can be verified as follows:

Added new tests for the RocksDBSSTWriter and RocksDBSSTIngestWriter paths for the new writing pathway.
[ ] TODO Add more rigourous tests for the new implementation.
[ ] TODO Extend existing tests to test both writer implementations. (I'd manually changed the defaulting to test the new configuration).
[ ] TODO Manually verified the change running on a cluster.
[ ] TODO Write benchmarks for https://github.com/facebook/rocksdb to compare the two writing methods.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): yes
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: yes
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? yes
If yes, how is the feature documented? TODO: not yet documented

@lgo  Thanks for your contribution. As I suggested in mail threads, please create another JIRA ticket and link this PR to that one.
BTW, I noticed this PR cannot pass the CI due to some new files lacking of license. Maybe you could tag this PR as "draft" before it could pass the CI and then reviewed.

@Myasuka actually this is doing exactly what the bulk loading ticket is describing. The RocksDBSSTIngestWriter is a drop-in repacement for RocksDBWriteBatchWrapper where only RocksDB#put is being used, so this is implementing bulk loading.
Right now, the PR has 2 locations changed to use it in full savepoint recovery and state migrations (migrateStateValues). This happens via RocksDBWriteFactory#defaultPutWriter, which will provide the new RocksDBSSTIngestWriter if it has been configured.
Sorry, I'll update the description to be a bit more descriptive of this. This is definitely a draft, so I'll go ahead and add that tag (:

Oh sorry, I just realized what you meant. The ticket is quoted a bulk loading FAQ, and I assumedi t was always about the SST ingestion loading. I'll create a new one!

@lgo Have you ever completed the PR yet?

@lgo Have you evaluated the performance improvement of the change?

Sorry about having dropped the progress on these changes @qinjunjerry @Myasuka (life got the better of me for quite a while). I did not end up getting a good production benchmark for the changes here.
I recently had the time to rebase these changes to the latest master. I also carved out the change to use deleteRange into a separate PR (https://github.com/apache/flink/pull/14893) and an early benchmark for that.
While I was doing a pass on that PR, I stumbled on an earlier ticket and discussion where @sihuazhou previously investigated using the external ingest API to speed these paths up. This is documented in FLINK-8845.
Summarizing the details from there, @sihuazhou's initial work and investigation found that the RocksDB's Java API for the SST writer had some key performance issues. Specifically, the interface was limited to put(byte[] key, byte[] value) and internally copied memory for constructing the RocksDB DirectSlice. This added a non-trivial overhead causing the Java SstFileWriter to have poor performance. As a result, they implemented the RocksDBWriteBatchWrapper for bulk writes rather than SST file ingestion.
I found the RocksDB issue with a detailed write-up outlining this problem with SstFileWriter performance: https://github.com/facebook/rocksdb/issues/2668.
Now, since then there was a PR, https://github.com/facebook/rocksdb/pull/2283, made to address this issue but it was hanging open from 2017. It had only just gotten merged in Feb 2020! This change was released as part of the Java API in RocksDB 6.8.0, see the 6.8.0.
Provided @sihuazhou's earlier investigation, I suspect this branch may not have much of an improvement without upgrading RocksDB to 6.8.0. Given the state of the RocksDB upgrade in https://issues.apache.org/jira/browse/FLINK-14482, I suspect it'll be quite some time (& work) before we get to there.

@lgo don't worry about the performance issue as we plan to upgrade RocksDB to at least 6.10 in Flink-1.13. We could ensure the SST generator works as expected first. Have the current commits could work well except possible performance issue?

It passes all of the tests in flink-statebackend-rocksdb module. I have not done any other testing, like setitng up an app and experimenting.
I added all the code behind a configuration option so both paths can continue to be executed. To test them both, I parameterized RocksDB tests to test both the WriteBatchWrapper (current) and SstFileWriter (new) ways of writing bulk put operations, and for a few tests duplicated them to trigger both paths.

@lgo Do you have plans to update this as we already upgraded to RocksDB-6.20.3?

Do we have some plan to drive this feature forward ? We have applied partly of this patch to improve the performance in our internal version, and thanks for your effort @lgo . I think the code is already in a good shape expect lacking of some benchmark of the testing. I'am glad to do a favor to complete this part, If need any help, please ping me  @lgo @Myasuka  :)


fix garbage collector log

Why do we need this change? :thought_balloon:
Logs are not in the correct format
Documentation update? :memo:

[ ] Yes
[X] No

Security Checklist :lock:
Upon raising this PR please go through RedHatInsights/secure-coding-checklist
:guardsman: Checklist :dart:

[X] Bugfix
[ ] New Feature
[ ] Refactor
[ ] Unittests Added
[ ] DRY code
[ ] Dependency Added
[ ] DB Migration Added

Additional :mega:

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Files
Coverage Δ

ros/processor/garbage_collector.py
73.52% <ø> (ø)

:loudspeaker: Thoughts on this report? Let us know!.

/retest


[WIP] Rework SSH key management UI to add GPG

** work in progress **
Complete GPG implementation #710 and #1150

** work in progress **
TODO :

[ ] Rename SSH keys to keys
[ ] List GPG Key
[ ] Display key validity in time
[ ] Add GPG Key
[ ] Add translatation

An early, and minimal, suggestion: change the key icon to a lock icon for the GPG/PGP portion. Really love, and appreciate the work you are doing to get this in!

Awesome work! :+1:

@Fastidious I was thinking of making it like github that use also a key for both but add a label under. But it would be better to just have a different icon like you propose.
For the next 5 days, I will not have much time but I will finish it just after.

Other icon possibility : http://fontawesome.io/icon/key/, http://fontawesome.io/icon/pencil/ (for signing), http://fontawesome.io/icon/check-square-o/, http://fontawesome.io/icon/paperclip/

Hey, I just tested the gpg Validation and really like it!
Is it intended, that the Validation (green box + icons) are not shown in the Code-Overview ?
This is something that GitHub doesn't have, because they are not showing the commit-ref in the Code-Overview, but since Gitea shows that, it would be really awesome to see the verified "Checkmarks" there too

@Schizopriest  It could be included in most part easily now with just passing commit list to ParseCommitsWithSignature like https://github.com/go-gitea/gitea/pull/1150/files#diff-da47b1b0b02ed3ac1a84db3edc178279R71 this will add information about validity of signature in template and use it like https://github.com/go-gitea/gitea/pull/1150/files#diff-7a50b55ffb3c01e4e467c74b23e0863bR44).

Woooh! Awesome 😄 Few conflicts though.
Some notes:

Maybe break the template down, see #1275
Not a fan of any of the suggested alternative icons. TBH just go with the same key for both, potential icon-changes can be discussed in a later issue 🙂

I broke myself this PR with a other PR removing extra formatting in settings pages. ^^
I will put time on this one soon since it is the last part of sign commit with GPG key.

This PR work now, I will look into optimization of verification since I found that hash class are consume and not re-usable so I fix it by regenerating hash before verifying against commiter keys but this is not really efficient way. Maybe I will put it under a other PR since this only a overhead for signed commit and relative to the number of key of the signing user (and stop at first validation).

@sapk great job! I will test this.

500 when I add a GPG key with a non-exist email address.

Thanks for the feedback.
@lunny I added a error type to catch to be display.
I think that when integration test are ready, this functionality will need some testing scenario.

@sapk another consideration. After I added a gpg key, then I remove the email address. What will happen?

Good question ^^

For validation of commit, those will failed because of https://github.com/go-gitea/gitea/commit/14fe9010ae8aecc0bcd38059b4c71256524b5341#diff-212b9fa4c927665ecbc5a5a36382ecedR385
For display of GPG keys this need to be checked. At first view, I think that there will still be display and not failed because we grab them by OwnerID. But I don't know how xorm will handle Emails fields leading to nothing. https://github.com/go-gitea/gitea/commit/ca1c3f1926eff992a2458f26cb24ed2f35265b05#diff-212b9fa4c927665ecbc5a5a36382ecedR33

But anyhow I will give this LGTM.

Yeah, appart from the conflict this is LGTM :D

make L-G-T-M work

OK I will rebase

Done, conflict was from #1290 because it move route from cmd to a package (for a good reason ^^).


feat(docker): Support buildkitd moby worker in dockerd 22.06-beta

Description
[+] Building 103.2s (11/11) FINISHED
 => docker-image://docker.io/nvidia/cuda:11.2.0-cudnn8-devel-ubuntu20.04                                                                                2.8s
 => => resolve docker.io/nvidia/cuda:11.2.0-cudnn8-devel-ubuntu20.04                                                                                2.6s
 => local://context                                                                                3.1s
 => => transferring context: 50.52kB                                                                                0.1s
 => CACHED sh -c apt-get update && apt-get install -y --no-install-recommends python3 python3-pip                                                                                0.0s
 => CACHED apt install gcc                                                                                0.0s
 => CACHED pip install -i https://mirror.sjtu.edu.cn/pypi/web/simple jupyter ormb                                                                                0.0s
 => CACHED mkdir /var/midi/remote                                                                                0.0s
 => CACHED mkdir /var/midi/bin                                                                                0.0s
 => CACHED copy /examples/ssh_keypairs/public.pub /var/midi/remote/authorized_keys                                                                                0.0s
 => CACHED copy /bin/midi-ssh /var/midi/bin/midi-ssh                                                                                0.0s
 => CACHED merge (apt install gcc, pip install -i https://mirror.sjtu.edu.cn/pypi/web/simple jupyter ormb, copy /bin/midi-ssh /var/midi/bin/midi-ssh)                                                 0.0s
 => exporting to oci image format                                                                                96.6s
 => => exporting layers                                                                                0.0s
 => => exporting manifest sha256:32bbf82b70e17ca70b11b88310ef02450db2bed3b94765415650a2227baa63cf                                                                                3.1s
 => => exporting config sha256:9bcaf4d291970033f2a6316dbf11912e77de402c81f0b11896f16c8bab19360b                                                                                1.2s
 => => sending tarball                                                                                91.6s

The image is built in buildkit, and it does not exist in the docker host. Thus we need to pipe the buildkit build image into the docker host. It takes about 100s for a 20G base image docker load. It is too slow. We need to optimize it.

https://github.com/moby/moby/issues/44369


Wizard: LAN IP address is empty after entering invalid address and reloading

Important notices
Before you add a new report, we ask you kindly to acknowledge the following:

[x] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
[x] I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug
After entering an invalid LAN IP address in the OPNsense wizard (e.g. an IPv6 address) and reloading at the end of the wizard, the LAN interface is left without an IPv4 interface. This practically breaks all connectivity of the LAN interface, including the GUI.
To Reproduce
Steps to reproduce the behavior:

Go to "System: Wizard"
Click "Next" until "System: Wizard: Configure LAN Interface".
Enter something invalid (e.g. "fd00:10::1") into the field "LAN IP Address:".
Click "OK" on the Pop-up "LAN IP Address field is invalid".
Click "Next" until "System: Wizard: Reload Configuration".
Click "Reload".

Expected behavior
The Wizard should prevent the user from proceeding after entering an invalid IP address, instead of only displaying the "LAN IP Address field is invalid" popup, like by graying out the "Next" button. While it is the user's fault for proceeding after the warning, the LAN interface on OPNsense is rendered completely unusable (no DNS, DHCP, routing, GUI, etc.) until the setting is corrected at the command line.
Describe alternatives you considered
Alternatively, if invalid data is entered in the LAN IP Address: field, the wizard could exit to the dashboard to prevent the user from locking themselves out.
Screenshots
Warning popup:

After the reload, the "IPv4 Configuration Type" is changed to "None".

Relevant log files
Configuration diff:
--- /conf/backup/config-1689971662.7157.xml	2023-07-21 22:34:22.716674000 +0200
+++ /conf/backup/config-1689971680.5847.xml	2023-07-21 22:34:40.585632000 +0200
@@ -277,7 +277,7 @@
       <descr/>
       <enable>1</enable>
       <spoofmac/>
-      <ipaddr><IP_ADDRESS></ipaddr>
+      <ipaddr>fd00:10::1</ipaddr>
       <subnet>24</subnet>
       <ipaddrv6>track6</ipaddrv6>
       <track6-interface>wan</track6-interface>
@@ -399,7 +399,7 @@
   </widgets>
   <revision>
<EMAIL_ADDRESS>-    <time>1689971662.7157</time>
+    <time>1689971680.5847</time>
     <description>/wizard.php made changes</description>
   </revision>
   <OPNsense>

Environment
Software version used and hardware type if relevant, e.g.:
OPNsense 23.1.11-amd64
FreeBSD 13.1-RELEASE-p8
OpenSSL 1.1.1u 30 May 2023
Intel® Core™ i5-6500 3.2Ghz Quad Core
Network Intel® I350-T4

Update
It seems that there is already a check for invalid IP addresses, but it is not triggered with valid IPv6 addresses:

@subnetspider thanks, c9bd89752 should fix this :)