Rendering performance improvements

As a result of https://github.com/twigjs/twig.js/issues/472 I've been looking into improving rendering performance. These changes reduce rendering times back to pre-async performance numbers.
Updates

[Promise] Reduced the amount of closures used by a promise.
[Promise] Reduced branching by replacing functions based on promise state.
[Promise] Use numeric constants to check promise state.
[Promise] Lazily convert handlers to an array.
[Promise] Use Twig.attempt instead of try/catch (optimization-killer)
[Promise] Move append out of constructor, js engine only has to optimize it once.
[async/forEach] Reduce amount of closures.
[async/forEach] Create less objects in memory.
[Twig.compile] Use Twig.attempt instead of try/catch
[Twig.expression.tokenize]: Don't leak arguments (optimization-killer)

Average rendering durations on different branches over 100 runs:
These numbers are using some of our own internal templates, feel free to check this branch against your own.

Current master: 0.422ms
v0.10.3 ( pre-async ): 0.356ms
This PR: 0.321ms

Example of test script
#!/usr/bin/env node

const Twig = require('../twig');

let i = 100;
const duration = [];

while( i-- ) {
    console.time('twig');

    Twig.twig({
        data: 'Hello {{world}}'
    }).render({ world: 'World' });

    console.timeEnd('twig');
}

Hi there! Sorry for the wait. I think your test case is too simple to see the issue. I've just tested your commit but the issue is still there. I've used this code just for testing purposes.
`var template = "{{ echo_test }}
{% for item in items %}
{{ item.title }}
{% if item.is_active %}

Active
{% else %}

Inactive
{% endif %}
{% endfor %}";
console.time('twig');
for(var i = 0; i < 1000; i++) {
Twig.twig({
data: template
}).render({
echo_test: 'Data for echo',
items: [
{
title: "This is a title",
is_active: true
},
{
title: "This is a title",
is_active: false
}
]
});
}
console.timeEnd('twig');
`

The test case was simply an example of the test harness I used, I dropped in some of our internal templates.
That being said, I will try and see what I can accomplish using your template. There is definitely more room for improvement here.

@DrParanoia Only a ~20ms difference after the last changes, will see if I can get it all the way back to original speeds.

@DrParanoia On my machine the two implementation are now running at equal speed. Could you try this PR and see if it works ok for you?
I've used your testscript to generate these results, so you should be able to reproduce these.
i=10; while [ $i -gt 0 ]; do test-05-v0.10.3.js; i=$((i-1)); done
twig: 261.706ms
twig: 250.302ms
twig: 246.580ms
twig: 249.145ms
twig: 246.331ms
twig: 245.665ms
twig: 258.785ms
twig: 256.907ms
twig: 253.217ms
twig: 247.564ms
i=10; while [ $i -gt 0 ]; do test-05.js; i=$((i-1)); done
twig: 250.118ms
twig: 244.766ms
twig: 247.833ms
twig: 253.946ms
twig: 240.143ms
twig: 247.774ms
twig: 246.613ms
twig: 241.833ms
twig: 251.292ms
twig: 249.193ms

@JorgenEvens Great job! I've run my test template with 10k iterations 10 times with each twig.js version. And results are:
old = 10799ms
new = 11357ms (only 5.16% slower than the old one)
I can definetly use it with only 5% slowdown.

@JorgenEvens Wow! After your last commit, the new version is now about 10.72% faster than the old one! That's a really great job there! Thanks!

At this point I would still like to fix the usage of apply with call where the number of arguments are known in advance, apart from that I think I'm finished for now.

@JorgenEvens Thanks for your job. And as much as I hate to bring the bad news, there are still significant performance problems when using include even if the templates are already compiled.
I've made this simple copy/paste into Chrome console test:
window.TwigTemplates = window.TwigTemplates || {};
if(!window.TwigTemplates["views/project/test_template_1.twig"]) {
    window.TwigTemplates["views/project/test_template_1.twig"] = Twig.twig({ autoescape: true, id: "/templates/get/?name=project/test_template_1.twig", namespaces: {"project":"/templates/get/?name=project"}, data<EMAIL_ADDRESS>});
    window.TwigTemplates["views/project/test_template_2.twig"] = Twig.twig({ autoescape: true, id: "/templates/get/?name=project/test_template_2.twig", namespaces: {"project":"/templates/get/?name=project"}, data: [] });
}

console.time('twig');
for(var i = 0; i < 100000; i++) {
    window.TwigTemplates["views/project/test_template_1.twig"].render();
}
console.timeEnd('twig');

Basically it's just two complied empty templates:
test_template_1.twig
{% include<EMAIL_ADDRESS>%}

And an empty template test_template_2.twig. I've run this test 10 times for each twig version and the performance difference is really dramatic:

old twig: 5292ms total
new twig: 9112ms total (~72.18% slower)

@DrParanoia I'll give that a try later tonight.
I did see that loading templates was far from optimal, but did not link it to this issue since it's behaviour should not have changed much by the inclusion of the async API.

@JorgenEvens Maybe performance drops because as soon as there is an include, the rendering switches to full async loop instead of tight one-threaded loop? Sorry if I don't make sense, haven't dug into your code :(

Non of the twig.js internals cause it to switch to full async mode. And since you are calling render and not renderAsync that would throw an error anyway.

@JorgenEvens Hmmm... then I'm currently out of ideas :( But the templates are empty, so it's definitely something with the include mechanism. I have quite a few includes in my project, so I guess that's why I saw the decrease in performance in the first place.

@DrParanoia just checked that I can reproduce your issue, so I will be looking into this tonight.

@JorgenEvens Thanks!

@DrParanoia Just an update for you.
I have not yet nailed down the reason why, but these methods are many times slower now that they live inside of a then callback.
https://github.com/twigjs/twig.js/blob/master/src/twig.logic.js#L777-L794
Moving away from try/catch towards Twig.attempt only makes the code run slower which is weird.
https://github.com/twigjs/twig.js/blob/master/src/twig.logic.js#L796-L799
Really weird one, but takes up a ton of CPU time, despite it only creating a new object.
https://github.com/twigjs/twig.js/blob/master/src/twig.path.js#L21-L41
This method was slow before as well, but it seems slower than it actually should be.

@JorgenEvens Hey there, thanks for the update. Could this be related to that inside of Twig.logic.type.include no matter what Twig.expression.parseAsync and template.renderAsync are always called?

parse used to be called as well as can be seen on this screenshot taken from the old version executing, and parseAsync is essentially only a passthrough that sets one of the arguments to a fixed value.
Twig.logic.parseAsync = function (token, context, chain) {
        return Twig.logic.parse.call(this, token, context, chain, true);
    }

I personally think there is some weird de-optimization / suboptimal optimization going on in the function due to the variables that are used outside of the closure. Especially the object allocation taking 30ms on a 900ms run is making me suspicious.

This is what that section looks like using the new version.

Wow, now that's some overhead...


Example: Component library where each component it it's own package

Describe the feature you'd like to request
A starter example component library where each component is published as its own NPM package.
Describe the solution you'd like

Turborepo
SWC for compilation and test running (through Jest)
Package publishing (could be Lerna or something else)
ESLint
TypeScript
A way to view components

Describe alternatives you've considered
Maybe tsup is a better solution here? Open to suggestions.
Storybook doesn't seem like the right choice because it brings in webpack and Babel and ideally this solution avoids those entirely.

Another thought: maybe the component library includes our preferred way of staying - is it Tailwind? Stitches?

Great idea

IMO #269 doesn't resolve this issue. The key thing missing would be per-component versioning.
I need to do this for work anyways, so I don't mind contributing back!
How far can I go with this template? Want me to integrate husky + lint-staged? Can I integrate Intuit's Auto? @leerob said no Storybook and I haven't been able to set it up anyways so I can go with Nextra.

@kylemh @jaredpalmer @leerob any updates on this. The current template is very basic as it's missing testing and previewing components using storybook or something similar. also it misses Husky and lint-staged and maybe commitlint. also, it would be great if eslint is treated as tsconfig, so you can have both browser and server rules seprated.

I'll definitely contribute as soon as I have any free time.
I did what I could to help by outlining what I think is missing here
I'm sure Jared is absolutely trying his best to tackle everything, always, all at once ❤

Update: We now have a design system example using:

🏎 Turborepo — High-performance build system for Monorepos
🚀 React — JavaScript library for user interfaces
🛠 Tsup — TypeScript bundler powered by esbuild
📖 Storybook — UI component environment powered by Vite

https://github.com/vercel/turborepo/tree/main/examples/design-system

Was going to raise an issue on best practice for nextjs + design system package + storybook, and stumbled on this one.  Seems like the design system example is good basis.  I had a couple of queries on it...

Storybook is added to the app, not the package - I would expect storybook to be used to test and validate the components (atoms, molecules etc)
An example using Image would be good as I think this presents some challenges due to the nextjs static image loading from public - a use case would be a background repeating image for texture or something similar - this is pretty much a major stumbling block for me at present

@leerob Is there any example available related to Vue & Storybook ?


filtering by diet_type

User will be able to filter recipes according to his preferences.

Custom comment added using Respost

Hello! Respost is here. This issue will be addressed by our contributors soon. Keep commenting and contributing.


What's new protocol

Now after initial release, we should have a what's new more exposed. I am having trouble finding it.
We need to write instructions on where to describe new things.
As I am searching arround I found:

New fetures with an option item should be added here https://www.wikitree.com/wiki/Space:WikiTree_Browser_Extension with * and a description
Each change worth mentioning should be added here https://www.wikitree.com/wiki/Space:WikiTree_Browser_Extension_Update I would recommend adding following style:

=== v.<IP_ADDRESS> (Development)  ===
* WikiTree+ Edit Helper
** Added Cemetery Category Picker that also searches in aka and locations defined in CategoryInfoBox on Cemetery category.
:: Additional description in new paragraph

=== v.<IP_ADDRESS> (Preview) 2022-11-03 (UTC) ===
* '''Category Display'''
Changes the location of Categories to the top of the Bio.

=== v.<IP_ADDRESS>  (Preview) 2022-11-03 (UTC) ===
* '''G2G Options'''
**  Checkmarks to show questions you have visited

== v.<IP_ADDRESS> initial Release 2022-11-03 (UTC) ==

Let's say the option name is bolded if it is a new option
If you need to write 2 paragraphs use :: at the beggining of the line.
When new ZIP files are prepared, the Development changes to Preview and new Development is added at the top with version increased by 1.
What are your thoughts?

Personally, I don't think the 'development' part of it is necessary, but I could be wrong.  I think the update page should be updated whenever a new version is uploaded (apart from just big fixes).  And the main page can be updated at the same time.  Also, the background.js file should be edited to open the page on update. After creating the zip file(s), comment out the bit that opens the update page.   (About having trouble finding the update page, I guess we could add a link to it on the main page.)
The :: sounds like a good idea.
At the start of the month, we need the stable version to have an update page open up. I'll create that page (though the current one is called '...Update', so I'm not sure what to do about the name of it).  (Of course, anyone else can make that page, too.)
I'm hoping that we can have better explanations than we currently have, as I've seen that some people are already confused by some things.

Done.


the images are broken

you should probably host them on github . . .

For example this doesnt show up!

Matter of fact none of these work!


Create Dynamical Systems Tutorial

#137 implements basic functionality for ODEs in Causal Pyro. Addressing this issue would involve writing a tutorial that illustrates those capabilities.
Specifically, this will include a probabilistic causal ODE model and using that model to answer associational, interventional, and counterfactual queries in a continuous time setting.
TODOs before open house:

[ ] Expand example to include richer probabilistic structure (e.g. partial pooling on parameters)
[ ] Stratify model
[ ] Run on real data

TODOs eventually:

[ ] Update model definition and simulator event loop after #151
[ ] Demonstrate counterfactual queries

Addressed by various dynamics refactoring PRs. Closing.


Add scripts in tools/ to the pypi release

Hi! First, thanks for releasing MMDet to PyPI, it improved it's convenience greatly. Could you add the scripts provided in the tools directory to the released package, so that they can be used after installing mmdet that way?
I believe it would be a nice simplification, because now, when one wants to use e.g. tools/train.py script, they must clone the repo instead of simply getting mmdet through pip install.
Thanks!

Thanks. We do plan to add support for this in the future.


[bug]: [Errno 20] Not a directory (details in report)

Is there an existing issue for this problem?

[X] I have searched the existing issues

Operating system
macOS
GPU vendor
Apple Silicon (MPS)
GPU model
Mac M1 Ultra
GPU VRAM
128GB
Version number
4.2.0
Browser
126.0
Python dependencies
{
"accelerate": "0.29.2",
"compel": "2.0.2",
"cuda": null,
"diffusers": "0.27.2",
"numpy": "1.26.4",
"opencv": "<IP_ADDRESS>",
"onnx": "1.15.0",
"pillow": "10.3.0",
"python": "3.10.12",
"torch": "2.2.2",
"torchvision": "0.17.2",
"transformers": "4.39.3",
"xformers": null
}
What happened
Basically I chose a model different than what I had been using, clicked the Invoke button and it gave me a server error and the following appeared in terminal:
[2024-05-20 14:11:21,196]::[InvokeAI]::ERROR --> Error while invoking session 339be1d6-6e7b-4dd2-a421-f14f79a64d63, invocation 89b8af93-2edd-4e71-8925-ca8e685af789 (i2l):
[Errno 20] Not a directory: '/Volumes/MegacityTwo/InvokeAI/models/.cache/.DS_Store'
[2024-05-20 14:11:21,196]::[InvokeAI]::ERROR --> Traceback (most recent call last):
File "/Volumes/MegacityTwo/InvokeAI/.venv/lib/python3.10/site-packages/invokeai/app/services/session_processor/session_processor_default.py", line 185, in _process
outputs = self._invocation.invoke_internal(
File "/Volumes/MegacityTwo/InvokeAI/.venv/lib/python3.10/site-packages/invokeai/app/invocations/baseinvocation.py", line 281, in invoke_internal
output = self.invoke(context)
File "/Volumes/MegacityTwo/InvokeAI/.venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/Volumes/MegacityTwo/InvokeAI/.venv/lib/python3.10/site-packages/invokeai/app/invocations/latent.py", line 1260, in invoke
vae_info = context.models.load(self.vae.vae)
File "/Volumes/MegacityTwo/InvokeAI/.venv/lib/python3.10/site-packages/invokeai/app/services/shared/invocation_context.py", line 351, in load
return self._services.model_manager.load.load_model(model, _submodel_type, self._data)
File "/Volumes/MegacityTwo/InvokeAI/.venv/lib/python3.10/site-packages/invokeai/app/services/model_load/model_load_default.py", line 80, in load_model
).load_model(model_config, submodel_type)
File "/Volumes/MegacityTwo/InvokeAI/.venv/lib/python3.10/site-packages/invokeai/backend/model_manager/load/load_default.py", line 62, in load_model
locker = self._convert_and_load(model_config, model_path, submodel_type)
File "/Volumes/MegacityTwo/InvokeAI/.venv/lib/python3.10/site-packages/invokeai/backend/model_manager/load/load_default.py", line 89, in _convert_and_load
loaded_model = self._do_convert(config, model_path, cache_path, submodel_type)
File "/Volumes/MegacityTwo/InvokeAI/.venv/lib/python3.10/site-packages/invokeai/backend/model_manager/load/load_default.py", line 120, in _do_convert
self.convert_cache.make_room(calc_model_size_by_fs(model_path))
File "/Volumes/MegacityTwo/InvokeAI/.venv/lib/python3.10/site-packages/invokeai/backend/model_manager/load/convert_cache/convert_cache_default.py", line 74, in make_room
lru_models = sorted(self._cache_path.iterdir(), key=by_atime, reverse=True)
File "/Volumes/MegacityTwo/InvokeAI/.venv/lib/python3.10/site-packages/invokeai/backend/model_manager/load/convert_cache/convert_cache_default.py", line 68, in by_atime
atimes = sorted([x.stat().st_atime for x in path.iterdir() if x.is_file()], reverse=True)
File "/Volumes/MegacityTwo/InvokeAI/.venv/lib/python3.10/site-packages/invokeai/backend/model_manager/load/convert_cache/convert_cache_default.py", line 68, in
atimes = sorted([x.stat().st_atime for x in path.iterdir() if x.is_file()], reverse=True)
File<EMAIL_ADDRESS>line 1017, in iterdir
for name in self._accessor.listdir(self):
NotADirectoryError: [Errno 20] Not a directory: '/Volumes/MegacityTwo/InvokeAI/models/.cache/.DS_Store'
When I went into that folder and tried to delete that file there wasn't even a file by that name. So I have no idea what its issue is as to why it is doing that.
What you expected to happen
I expected it to simply generate an image and not an error.
How to reproduce the problem
I'm not sure if this has anything to do with it but I had recently moved all my generated images to another location and then ran the database cleaning routine. Also I had added a line in the yaml file to tell it where to start storing images it makes going forward. Other than that, I'm not sure exactly. I am going to manually convert the models I use the most and try to eliminate the problem that way but I was asked to generate a bug report so that is what I am doing.
Additional context
No response
Discord username
Vanadium74

Hello darkness my old friend,
We've come to debug you again.
Because a Finder softly creeping,
Left its .DS_Store while I was sleeping,
And the fixes that were planted in my code,
Do not remain, in the github repo.

I had this error too, but solved it by erasing the .cache /invoke/models/.cache
I think using the UI to remove the model does not erase the .cache version ... maybe

Sorry I realize I didn't say how to fix this. Enable hidden files abcs delete the .DS_Store files


The when conditional was improperly being checked

The scenarios were not being accurately compared to ensure that:

A single scenario was choosen
ONLY a single scenario was choosen

This solution does not scale for long, but that can be addressed in a
different patchset.

Looks good but don't we have the same problem with the other OSD related tasks?

I just tested this, and it looks we also need to set default(False) to all the next tasks.

instead of setting default(False) to all of those I purpose a temporary vars file that will difeault all the variables to false. we can remove it in a later refactor

@SamYaple please go for it :)

Thanks @SamYaple, I think we also need (for more complex scenarios):

osd_directories
osd_auto_discovery
devices
raw_multi_journal

raw_multi_journal is set to false in this patch already
osd_auto_discovery is set to false in this patch already
devices is checked with "devices is not defined"
osd_directories is checked with "osd_directories is not defined"

That covers all of the things you commented on. If thats it, this patch is good as is.

It is indeed, I guess it was just me not looking properly. Thanks again!


added /rss which gets json output of all feeds

created a new link in server.js which calls find and selects only the feed part of the result
returns json.
work toward and possible fix for issue #27

Apologies for merge conflict, this is because my fork was a way behind what's currently in the repo.

Hi,
It looks like you're trying to do some work on the web job inside the website code! I'll contact you to explain how to fix this and get a valid pr submitted. But I'm afraid I cannot accept this into master.


Add footer

Contains links to contacts and some major pages

Server-Side Includes don't work for some reason so I need to find an alternative (preferably not with Jekyll, because then I need to set up a replit environment for that). Any suggestions?


Remove all but team-shared files from .idea folder

Only the analytic scopes from CLion add any value :)

Duplicate of #25


format_strings breaks after "\" resulting in invalid output

#[rustfmt::skip]
fn raw() {
    "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\\a";
    "\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\";
    "\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\";
}

fn formatted() {
    "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\\
     \a";
    "\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
     \";
    "\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
     \\\";
}

rustfmt 1.4.38-nightly (efec545 2021-12-04)
installed via rustup

format_strings = true
Note this happens when the "\\" is at a position close to max width.

This is one I'm fairly confident has been fixed on the other branch and just needs to have the commits cherry-picked over

This is one I'm fairly confident has been fixed on the other branch and just needs to have the commits cherry-picked over

Was it maybe 472a2ed0f6f75cc33f42a72beaf1827c13b713d1?

This is one I'm fairly confident has been fixed on the other branch and just needs to have the commits cherry-picked over

Was it maybe 472a2ed?

I actually was thinking about #4524. Whenever you get a chance would you mind seeing if this behavior is reproducible on the 2.0 branch? If not that means the it's covered by that fix too, in which case we should give that another quick pass and consider backporting the commit(s)

on the 2.x branch here's how rustfmt formats the input:
fn raw() {
    "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\
    \\a";
    "\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
    \";
    "\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
    \\\";
}

It's a slightly different formatting that the 1.x branch, but it still doesn't compile. The silver lining here is that the 2.x branch formats the first line in a way that would compile if it was the only line that we needed to be reformatted.
// This compiles
fn main() {
    "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\
    \\a";
}

The bad news is that the last two lines aren't formatted any differently so there's still some work to do, but I think backporting the existing changes is a good starting point.

Thanks for digging into this!

The bad news is that the last two lines aren't formatted any differently so there's still some work to do, but I think backporting the existing changes is a good starting point.

Perhaps, though if that doesn't fully address the overall issue it may be better to try to find a holistic solution instead

Perhaps, though if that doesn't fully address the overall issue it may be better to try to find a holistic solution instead

Yeah you might be right

This incorrect output has already been fixed in version 1.60.0-nightly. In fact, the following output appears:
error[internal]: line formatted, but exceeded maximum width (maximum: 100 (see max_width option), found: 103)
In this case, I don't know if the correct thing would be to increase the size or simply show the same message. Also because the case above, from a practical and functional point of view, does not seem to make sense to me. Any objection to this question @bddap ?

Sorry, @lobocode . Not sure I understand what you are asking.

Sorry, @lobocode . Not sure I understand what you are asking.

I'm asking if you've checked, if this bug has been fixed in later versions of rustfmt.

I have not.

fn main() {
    let body = format!("aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\"");

    /* After cargo fmt:

    let body = format!(
        "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\\
         ""
    );

     */
}

Another badcase with format_strings = true enabled.


Initial blockprod unit tests

Initial unit tests to blockprod before we move mining to outside the main executor

Ah. That last diff in do_checks.sh was a cargo fmt error not just a diff!


IPhone when fullscreen it does not play 360

Hey Thiago,
I have issue with iPhone , when i hit on play button it takes me to the fullscreen by there it does not play 360 please suggest something. I am stuck.
Or please suggest me how to control the play button not to navigate to full screen

+1
I am also currently using it for a Cordova iPad application. I would really appreciate it if there is an option to turn off the fullscreen mode since the 360 functionality doesn't work during fullscreen.
If the off parameter for fullscreen has to be handled in the actual Clappr player, please let me know so I could make a request over there.
Thanks!

Hi,
Even I am searching for the same.
On Wed, 3 Jul 2019, 8:13 a.m. darylldawn<EMAIL_ADDRESS>wrote:

+1
I am also currently using it for a Cordova iPad application. I would
really appreciate it if there is an option to turn off the fullscreen mode
since the 360 functionality doesn't work during fullscreen.
If the off parameter for fullscreen has to be handled in the actual Clappr
player, please let me know so I could make a request over there.
Thanks!
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/thiagopnts/clappr-video360/issues/20?email_source=notifications&email_token=AK6W6WMNTWPVL664M7NYT2LP5QG3VA5CNFSM4HYT5J4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZDC3SI#issuecomment-507915721,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AK6W6WM2NNVE2APV4IBBEQDP5QG3VANCNFSM4HYT5J4A
.

I can reproduce on my iphone, I'll take a look when I get a chance

Sorry for taking forever to look into this, I published a version that prevents going fullscreen by default. Fullscreen on iPhones will still be an issue, but you can hide the fullscreen button on your side to prevent the user going fullscreen this for now


Blazor 0.8.0 support

Blazor 0.8.0 was just released
https://blogs.msdn.microsoft.com/webdev/2019/02/05/blazor-0-8-0-experimental-release-now-available/

I do not know if it is good to be depend on preview version, especially if you need VS 2019.

Thank you!


Draft: Add textHighlight theme setting

Added a setting for the textHighlight background colour since the default isn't the most readable in dark mode

the force push was amending my previous commit after running prettier --write .

Will update the docs tomorrow


Debugger attach to process fails with NullPointerException on WinDbg

Hello there,
Attempting to attach the debugger to an already-running process using the "Attach" button of the Objects panel fails with an "attach failed" message if using the WinDbg backend.
Modifying Ghidra to print the actual exception message on failure (file: Ghidra/Debug/Debugger-agent-dbgeng/src/main/java/agent/dbgeng/model/iface1/DbgModelTargetAttacher.java) as below (lines 46-48)
	}).exceptionally(exc -> {
			Msg.error(this, "attach failed" + exc.getMessage());
			return null;

presents this on the agent window:
attach failed
java.lang.NullPointerException: Cannot invoke "agent.dbgeng.manager.DbgProcess.getPid()" because "this.currentProcess" is null

No other modifications to source code were done.
Steps to reproduce:

Start a new Ghidra project
Open the Debugger and create a new WinDbg connection from the button on Debugger Targets. Use the default settings.
The choice between GADP or not doesn't seem to matter.
The connection should complete. From the Objects panel, expand "Available" sub-tree and select a random user process.
Click the "Attach" button on the Objects panel, while the user process is selected.
The button should have no effect, and on the agent window (if using GADP) the above message will appear.

OS: Windows 10
Version: master as it was on March 19th 2023. Cannot test on current master right now, so not sure if it repro's on the current state of the branch.

@lightspot21 Nice catch - much appreciated!  Adding the fix now.  In the event you'd prefer not to wait, the fixes are:
(1) in DbgProcessImpl::attach line 265, insert "id = DebugProcessRecord(toPid);"
(2) In DbgManageImpl::setActiveThread and setActiveProcess, wrap the first three lines of each with "if (currentThread != null) { .... }" and "if (currentProcess != null) { .... }"
I failed to adequately test the "attach" case.  (Need to check our JUnits as well, I guess.)   THANK YOU!

Fixed by 33958e386da22c435f664deab2ecf4c4b647361d


Tensor board

Update:
I have found a way to run event file, but the url generated does not work, am I doing something wrong?

I have also tried with adding --port6006 , but still the generated url is not working

Thanks

@AN-Ahmed logdir is a directory path rather than a event file path. You may google how to use tensor board to solve it:)


maxResponseSize option does not work when content-length header is not present in response headers

🐛 Bug Report
The maxResponseSize option does not work when the content-length header is not present in the response headers.
This is the case when the response is chunked
To Reproduce

Create a client with maxResponseSize option set to a value smaller than the response size.
Make a request to an endpoint that returns a chunked response. e.g. client.asInternalUser.indices.getMapping
The response is not aborted when the response size is larger than the maxResponseSize option.

Expected behavior
The maxResponseSize option should work regardless of the presence of the content-length header
Your Environment

node version: 16.18.1
@elastic/elasticsearch 8.4
os: Mac

I've opened https://github.com/elastic/elastic-transport-js/pull/57 to address this specific issue, even tho I would recommend using the asStream option when you know the response will have the 'transfer-encoding': 'chunked' header.


133 test cov workflow

@sarabodach tell me about this PR

OK thx.  I get my yaml confused and didn't know if this was some sort of cicd


Set CV folds to 10 for datasets under size threshold

Resolves #1783
Resolves #2312

@christopherbunn This resolves #2312? Will unassign myself :)

@christopherbunn This resolves #2312? Will unassign myself :)

Looks like it, I can add the issue number to the PR. Thanks!

Ah thanks both for catching!
@christopherbunn please assign yourself to #2312 and #1783.
@christopherbunn thanks for finding #1783 as well, sounds like we had a duplicate there!

Closing due to a lack of performance improvement on smaller datasets. More info on #2414.


Github sponsors support

Hello,
I wanted to ask if you plan to add Github sponsors for this project? This way people will be able to support the project financially.
More infos: https://github.com/sponsors

I take a look at it. Thanks


fine-tuning Few-Shot Relation Classification

Hi,
Thank you for sharing this wonderful work! Could you detail the hyperparameters of fine-tuning TACRED?
I run the experiments with
python run_glue.py \   --model_name_or_path roberta-large-mnli \   --train_file /data/tacred/train.mnli.json \   --validation_file /data/tacred/dev.mnli.json \   --do_train \   --do_eval \   --max_seq_length 128 \   --per_device_train_batch_size 32 \   --learning_rate 4e-6 \   --num_train_epochs 2 \   --overwrite_output_dir \   --fp16 True \   --gradient_accumulation 1 \   --output_dir ./results \   --save_steps 5000 \   --seed 0 \   --warmup_steps 1000
and then evaluate the final results with
python evaluation.py --config ../resources/predefined_configs/tacred.relation.config.json (modify the "nli_models" with tuned model path)
the test resutls are
"test": {"optimal_threshold": 0.5,                "positive_accuracy": 0.9004511278195488,                "precision": 0.43177373251090717,                "recall": 0.8631578947368421,                "f1-score": 0.5756117127958283        }
Could u kindly correct me how to tune and evaluate? Thank you!

Hi @hanjiale ,
The hyperparameters you have used looks fine to me. Maybe I can suggest you to try 1e-5 as learning rate and increase the epochs to 5. You can also use the load_best_model_at_end with accuracy (or defining the entailment f1_score) as metric_for_best_model to improve the checkpoint selection. But in any case, those hyperparameters should be fine. I am concerned about the optimal_threshold value... Did you use the development set, at least on a small portion of it,  to optimize the threshold? Maybe that explains the huge gap between the precision and recall.
One of the weakness points of the approach is the need tof tunning the threshold, at least on a very small portion of data. We are working on it and have very promising results, but the improvements are still work in progress.
Let me know,
Oscar

I see. The problem is that I remove the "dev_path" in the tacred.relation.config.json. Thank you for your reply and looking forward to your new work!


Can you support eggjs?

Is there any plan to be compatible with egg.js, similar to koa, but re-specify the path of koa-views, otherwise egg-views is used, and some files cannot be found
Temporary workaround
export class EggAdapter implements IServerAdapter {
   // ... like KoaAdapter
  // @bull-board/ui path
  private staticPath = path.resolve('node_modules/@bull-board/ui/dist')

   public registerPlugin() {
   // .....
    const viewRoutes = Array.isArray(route) ? route : [route]
    viewRoutes.forEach((path) => {
      router[method](path, async (ctx) => {
        const { name } = handler();

        // rewrite render
        (ctx as any).render = views(this.staticPath, {
          extension: 'ejs',
          map: 'ejs'
        })(null as any, null as any)
        const basePath = this.basePath.endsWith('/') ? this.basePath : `${this.basePath}/`
        await (ctx as any).render(name, { basePath })
      })
    })
   // ....

   }
}

EggAdapter

I'm not familiar with egg.js, but a PR for an Adapter will be cool.


Replace MethodInfo constructors with variadic M_INFO

This PR replaces 18 constructors of MethodInfo with a single variadic method called M_INFO.
12 of the constructors had actually different arguments than the other 6, which allowed to set method return type with a single call. They were used ~5 times though, compared to ~300 uses of the normal (name, propertyinfo...) constructors, so I removed them. They can be brought back in a variadic form, but changing object.h is a pain, so good luck with that.

Superseded by #62342.


subscriptions of multiple channels returns latest channel only

Version: 2.6.3
Platform<EMAIL_ADDRESS>macOS
Description: data from subscribe callback with array of channels as argument, is a string with latest channel only

debug trace
Net read sm.sp.dev:6379 id 1
Redis server ready.
on_ready called sm.sp.dev:6379 id 1
Send sm.sp.dev:6379 id 1: *3
$9
subscribe
$21
home/garden/fountain2
$4
foobar

Net read sm.sp.dev:6379 id 1
subscribe home/garden/fountain2
subscribe foobar

redis-cli
<IP_ADDRESS>:6379> SUBSCRIBE me/garden/fountain2 foobar
Reading messages... (press Ctrl-C to quit)
1) "subscribe"
2) "me/garden/fountain2"
3) (integer) 1
1) "subscribe"
2) "foobar"
3) (integer) 2

code
consumer.subscribe(['home/garden/fountain2', 'foobar'], function(err, res) {

  console.log(res); // foobar
});

It would be nice if it could returns an Array of specified channels

This is a known issue and worked this way from the beginning on due to a faulty implementation. As this would be a BC I can not change this right away though. This is planned for v.3.

thank you


Added python3-build

Please add the following dependency to the rosdep database.
Package name:
python3-build
Package Upstream Source:
https://github.com/pypa/build
Purpose of using this:
This dependency is being used for a package that makes it easier to add python code to an  ament_cmake project.
It may be use to others who want to work on various python-related build processes with ROS.
Distro packaging links:
Links to Distribution Packages

Debian: https://packages.debian.org/

https://packages.debian.org/bullseye/main/python3-build

Ubuntu: https://packages.ubuntu.com/

https://packages.ubuntu.com/jammy/python3-build
Not available in focal or earlier, added pip dependency

Fedora: https://packages.fedoraproject.org/

https://packages.fedoraproject.org/pkgs/python-build/python3-build/

Arch: https://www.archlinux.org/packages/

https://archlinux.org/packages/extra/any/python-build/

Gentoo: https://packages.gentoo.org/

https://packages.gentoo.org/packages/dev-python/build

macOS: https://formulae.brew.sh/

https://formulae.brew.sh/formula/python-build

Alpine: https://pkgs.alpinelinux.org/packages

https://pkgs.alpinelinux.org/package/edge/community/x86_64/py3-build

NixOS/nixpkgs: https://search.nixos.org/packages

OPTIONAL

openSUSE: https://software.opensuse.org/package/

IF AVAILABLE

rhel: https://rhel.pkgs.org/

IF AVAILABLE

@m-elwin, it looks like the package is available for RHEL, too. Would you mind adding it?

@audrow added for rhel.

Switched to '9' for rhel and removed pip dependencies in focal/rhel8. This package is not available via apt/yum for Focal/rhel8 as far as I can tell, but I also don't know of a need for this dependency on those platforms.


Conditionals in reports?

This isn't an issue, but more of a feature request.  I'd love it (and I hope others would too) if there was some way to determine certain things (ie: conditionals, a simple if/then/else) within the openaps reports themselves, rather than having to do some bash-level copying/comparing to accomplish certain things.  For example, I would like to have my rig set up to both use a connected CGM as well as NS for data.  If one set of data is available, then don't bother looking for the other - no need for me to pull data from a remote site if I've got it locally via my CGM.  I know we can invoke bash within the 'openaps.ini' file, but that's not really the most elegant way of doing it - especially for beginners.  I don't think that such functionality exists already, but if so, I'd love to take advantage of it.

@Spazholio want to checkout out @sulkaharo's new oref0-crun tool in https://github.com/openaps/oref0/pull/205 and see if that does what you want?  If not, please reopen.


Connect ZenDesk KM to Website

Keeper 101
Wallets
Vault
Inheritance
Whirlpool
Backup and Recovery
Signing Devices
Subscription
Tor
Multi-sig
Troubleshooting.
(For later  1 and 11 )
For Eg : Wallet as a main tab and inside that How ,why and what ?

This is done


Update CLI extensions available doc

Update CLI extensions available doc.
Triggered by Azure/azure-cli-extensions - TRAVIS_BUILD_ID=401066868
https://github.com/Azure/azure-cli-extensions/commit/55786f8cd2a0061e48fe8dfbfacb14e51754531b

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs-ref-conceptual/azure-cli-extensions-list.md
:white_check_mark:Succeeded
View (azure-cli-latest) View (azure-cli-2017-03-09-profile)

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


migrate go to provided.al2023 and update localstack to docker compose

Hi @wimspaargaren, thanks for sharing your project
I carried out some changes to run the project locally

migrate go1.x runner to provided.al2023
replace docker to docker compose (adding SERVICES and volume)
rename the account number format in the role name

Hi @ncostamagna , thanks for your contribution! Looks good to me :) One small thing: Could you fix the new lines in the docker-compose.yml and the .gitignore?

Hi @wimspaargaren , how are you? :)
Yes, of course, I've just made the changes in the PR, adding the new line in the docker-file and the .gitignore files

That's wonderful! I'm so happy for you! Congratulations @wimspaargaren ! 😃😃

Thx 😄 added a small fix on top of your work btw: https://github.com/wimspaargaren/go-lambda-localstack-example/commit/c6e58efbd4640b46d5a2c9c269caa8893fb8445b


Dartanalyzer errors in projects with one contentroot and multiple nested dart projects

I often use specific project layout when I put set of related dart projects into one parent directory.
In IDE (I use WebStorm) I used to just open this parent directory
After upgrading to SDK 1.24.0 this stop working for me. import statements bacame underscored with red lines, with error message resource not found
I've tried some usual remedies: Removed pub cache, have tried stable and dev release of SDK, have updgraded WebStorm from 2017.1.2 to 2017.1.4 - nothing worked out.
Only the downgrade to SDK 1.23 helped. I'm on Windows 10.
When I raised this question on https://gitter.im/dart-lang/TALK-general István Soós wrote that he had the same experienve with SDK 1.24.0 and IndellyJ IDEA. For him too only downgrade to 1.23.0 worked out.
On https://groups.google.com/a/dartlang.org/forum/#!topic/misc/5UE9MpkG8XM Петър Събев reported same problems with same project layout in InteliJ IDEA
I've found that if instead of opening parent directory I add sibling projects as additional content-roots - errors disappear too. That worked for Петър Събев too
I've tried to reproduce that bug and make test repo https://github.com/vadimtsushko/dir_with_subprojects , but with such small projects (or small set of related projects) it did not reproduce

I opened my project in old way so I can upload snapshot with errors. It looks like dartanalyzer works fine when it run from the dart project directory (in my example on snapshot - I run C:\ProjecAlt\infovizion_project_root\infovizion\dartanalyzer bin )
But when I run dartanalyzer from parent dir ( C:\ProjecAlt\infovizion_project_root\dartanalyzer infovizion\bin ) I get just the same errors that in WebStorm IDE. Looks like that somehow linked

I am experiencing this issue as well. All was well until the 1.24 update, same occurring with 1.24.1.

Update: After a while I started to get these errors in project with multiple content roots too. So my workaround actually does not work for me anymore. I'm switching to SDK 1.23 for now.

I've encountered the same issue  using WebStorm 2017.1.4. Everything had worked fine untilI upgraded Dart SDK into 1.24.

@jwren, can you take a look at this one?

This issue feels related to https://youtrack.jetbrains.com/issue/WEB-27223, location of .packages files, etc, but it is fixes a known issue, it is not a recent change.
I will talk with @bwilkerson

I suspect that this is on the analysis server side, but have not yet had time to investigate to confirm that.

I am having a similar issue. I do not use nested projects as described above, but I do use libraries in local directory packages which are in sibling directories to the main project(s). Everything other than the Analyzer works fine in 1.24.

@jwren For me and three persons who commented here (sanmadjack, marcingryska, jwt-sgg)  plus István Soós at gitter and Петър Събев at dart misc - problems start with upgrade to 1.24.0
I temporarily downgraded to 1.23.0 and this problem disappeared for me.

I'm also seeing this issue you can reproduce by opening this repository:
https://github.com/dart-lang/build
This worked fine prior to 1.24.

As a data-point, I don't see issues when cloning that repo, running pub in the sub-dirs, and running tuneup from the top-level build/ dir (tuneup is analysis server based).

iIt might be IDE specific, but I use IntelliJ and it's sad 😢

@matanlurey I tried to get more data on this, but need more information.
I started by cloning the build repo to my local disk, then I opened the top-level directory in IntelliJ. There were lots of errors, so I ran pub upgrade in each of the subdirectories containing pubspec.yaml files. All of the errors went away and it appears to be fine.
Are invalid errors the indication that there's a problem, or is there something else that I should be looking for?
FWIW: I'm running the bleeding edge version of the SDK, so it's possible that the problem has been corrected but that the fix didn't make it into the stable/dev release.

@matanlurey, can you describe your project setup for build/? One project, one module, one content root? one project, multiple modules, one content root per module? One project, one module, multiple content roots for that module?

Git clone
Open 'build' folder directly File -> Open
Right click on every pubspec.yaml and run "pub get/upgrade"

At least, this used to work without configuring anything else in IntelliJ

I used File -> New -> Module from Existing Sources... to open the build directory.
Then I tried using File -> Open and it still appears to be fine.
I'll ask again: Are invalid errors the indication that there's a problem, or is there something else that I should be looking for?

I git cloned, ran pub in the subdirs, and opened in IntelliJ (idea .) - no more configuration, and I didn't see issues.

In my experience errors start to appear only after i go to some dart file in editor.  Webstorm temporarily shows message like 'resolving paths' in status bar. Then import statements in dart file start to show errors.  Sorry for phone typing

I tried opening and even editing files. The only errors I saw were those I introduced while typing.

@devoncarew  Using WebStorm I have very similar impressions as  @vadimtsushko  that issue firstly appears after opening a particular .dart file in code editor. Since then it's present permanently.  Thus it could be that one doesn't see errors straight after opening IDE or after doing get/upgrade on pubspec.

So I've tried to reproduce it on build repo and I have same errors

Steps to reproduce:

Install dart v.1.24.1
Install IntelliJ IDEA Community Edition  v 217.1.4
Add Dart Plugin
clone https://github.com/dart-lang/build
File -> Open  parent repo directory - build
Click on pubspec.yaml file in e2e_example
Configure project for Dart
Run Pub: get dependencies context menu action on pubspec.yaml
Open file e2e_example/tool/build.dart (So far there are no errors)
Try to go to source on PhaseGroup - I'm on Windows and do it pressing the Ctrl key and clicking left mouse button on PhaseGroup() at line 13

After some dart analyzer work there is red lines all over the projects and 1805 errors in the Dart analyzer output

I was able to reproduce the described problem, but only within the e2e_example directory. If I open any files in other subdirectories I don't see any errors.
The problem is in the build/analysis_options.yaml, which contains the following:
analyzer:
  ...
  exclude:
    - bazel-*
    - e2e_example/**

Because server was told not to analyze this directory it did not create an analysis context for this directory. When these files are opened we attempt to analyze them anyway, but in a context in which we do not know how to resolve any of the package: URIs.
@kevmoo When I remove that exclusion everything is analyzed correctly and the errors do not appear. I don't know why this directory is excluded from analysis; it seems like it would be useful to know if changes to the other packages have broken the example.
This is a less than optimal UX, so it would be nice to do something about it, but I doubt that it will happen soon.
The important question is: are the other manifestations of this issue also due to having excluded some portions of code from analysis, or was the build package a red-herring.

I forked build package to https://github.com/vadimtsushko/build and deleted - e2e_example/**  line from build/analysis_options.yaml
After cloning that repo I went to build/too/create_test_all.dart instead of e2e_example and still got the same errors. ( That is Windows 2008 R2 server with Dart version 1.24.2 )

In my own project there is no analysis_options.yaml file in the root directory, and analysis_options.yaml files of the individual projects have simplest form:
analyzer:
  strong-mode: true

Thanks for trying that!
I did the same thing and cannot reproduce it using the bleeding edge version. At this point I'm inclined to believe that the bug has been fixed but not yet released.

Just tried this and with  1.25.0-dev.3.0 with build package errors disappear for me.
Will try to swich to bleeding edge on my computer tomorrow and check on my projects

On bleeding edge SDK my internal projects works without errors now.
Thank you

I'm glad to hear that it's working for you!

Is it possible to have a fix for this in the 1.24 branch?
We are stuck in 1.23 because of this.
Dart 1.25-dev doesn't have Dartium, we cannot use it right now.
This seems to happen only on Windows with Webstorm. (I think this is when there is an uppercase letter in the path of the project).

We are also forced to keep 1.23.
Concering our case, problem exists also on Linux with Webstorm (Ubuntu 16.04 LTS + WebStorm 2017.1.4)

I work on v 1.25 with last available Dartium without apparent problems. I was advised aabout that by @zoechi who use dartium with v1.25 himself.

I work on v 1.25 with last available Dartium without apparent problems.

This is what I was going to recommend - the 1.24 Dartium will work with 1.25.


Get one-command Linux build working

ATM you have to install a bunch of stuff by hand, and to statically link it, you need to edit files in multiple other projects.
Can we instead get to a point where you can do ./make-linux and it clones everything (perhaps pinned to known good versions) and builds them?

"Fixed", or at least improved in https://github.com/cldellow/sqlite-parquet-vtable/commit/0bdcc9895e5b34742361780f2dda19f2003541dd. Pins parquet-cpp to 1.4.0 and arrow to 0.9.0.


Enhance Interface Interaction with Hover Color Change

Description:
To improve the user interface and provide clearer feedback, we propose adding a color change effect to the interface elements on hover. This enhancement will help users identify which element they are interacting with, making the interface more intuitive and visually appealing.
Implementation Details:
Current State: Elements remain static without any visual feedback when hovered over.
Proposed Change: On mouse hover, the background color of the element should change to a lighter or more vibrant color. This can be implemented using CSS, utilizing the :hover pseudo-class.
Color Suggestions: Consider using a subtle shade that contrasts with the current element color, ensuring it remains accessible and visually pleasing.

could you possibly show a preview ? Thanks for the issue

@PragyaTripathi990 i can resolve that issue assign it to me


refactor: [NPM] General translation logic for egress and ingress

Reason for Change:

This PR is to support egress and generalizes translation for egress and ingress.
Issue Fixed:

Requirements:

[x] uses conventional commit messages

[ ] includes documentation
[x] adds unit tests

Notes:

[NPM] General translation logic for linux and windows #1055
refactor: [NPM] parsing label selector for general translation logic #1077
refactor: [NPM] General translation logic (mainly clean-up codes and correct bugs) #1105

Just curious, should some srclist become dstlist for egress?

I will walk through srcList and dstList again.

I should have noticed this last PR, but can we consolidate some structs/fields? Right now, we have the NPMNetworkPolicy.PodSelectorList, which is idential to information in NPMNetworkPolicy.PodSelectorIPSets with an additional included bool for each set. We also have labelSelector struct in parseSelector.go which is identical and copied over to the two structures above. We can consolidate these three duplicate structures/fields into a into a new struct in the policy package to prevent duplication and reduce the amount of code (less complex and less bug-prone):
Proposal
Remove the labelSelector struct and NPMNetworkPolicy.PodSelectorList field, and make the type of NPMNetworkPolicy.PodSelectorIPSets as IncludedTranslatedIPSet
type IncludedTranslatedIPSet struct {
    Included bool
    ipsets  *[]TranslatedIPSet
}

Yes. Good suggestion.
I think you mentioned this in previous PR, I put it in some notes, but missed it in this PR.
Let me walk through this issue again and get back to you.

To limit scope of this PR, we can leave the old NPMNetworkPolicy fields and add this as a new field. That way, current DP is unaffected, and we can follow up with a PR in DP.
As you said, it would be good to a single PR to add and remove related codes if we decide to revise the struct. wdyt?

To limit scope of this PR, we can leave the old NPMNetworkPolicy fields and add this as a new field. That way, current DP is unaffected, and we can follow up with a PR in DP.

As you said, it would be good to a single PR to add and remove related codes if we decide to revise the struct. wdyt?

If we change later, we can get to integration testing quicker, but then we might notice bugs and have to make fixes based on structs that we won't use later. I don't have a strong opinion.

To limit scope of this PR, we can leave the old NPMNetworkPolicy fields and add this as a new field. That way, current DP is unaffected, and we can follow up with a PR in DP.

As you said, it would be good to a single PR to add and remove related codes if we decide to revise the struct. wdyt?

If we change later, we can get to integration testing quicker, but then we might notice bugs and have to make fixes based on structs that we won't use later. I don't have a strong opinion.

Anyway if we change some codes, we need to do integration test again even though it is non-functional change (and highlighted your point). So, let's complete some necessary code change and PRs first and then plan integration test. We don't need to rush.

To limit scope of this PR, we can leave the old NPMNetworkPolicy fields and add this as a new field. That way, current DP is unaffected, and we can follow up with a PR in DP.

As you said, it would be good to a single PR to add and remove related codes if we decide to revise the struct. wdyt?

If we change later, we can get to integration testing quicker, but then we might notice bugs and have to make fixes based on structs that we won't use later. I don't have a strong opinion.

Anyway if we change some codes, we need to do integration test again even though it is non-functional change (and we don't waste time your point - but then we might notice bugs and have to make fixes based on structs that we won't use later.). So, let's complete some necessary code change and PRs first and then plan integration test. We don't need to rush.

Just curious, should some srclist become dstlist for egress?

I will walk through srcList and dstList again.

I should have noticed this last PR, but can we consolidate some structs/fields? Right now, we have the NPMNetworkPolicy.PodSelectorList, which is idential to information in NPMNetworkPolicy.PodSelectorIPSets with an additional included bool for each set. We also have labelSelector struct in parseSelector.go which is identical and copied over to the two structures above. We can consolidate these three duplicate structures/fields into a into a new struct in the policy package to prevent duplication and reduce the amount of code (less complex and less bug-prone):
Proposal
Remove the labelSelector struct and NPMNetworkPolicy.PodSelectorList field, and make the type of NPMNetworkPolicy.PodSelectorIPSets as IncludedTranslatedIPSet
type IncludedTranslatedIPSet struct {
    Included bool
    ipsets  *[]TranslatedIPSet
}

Yes. Good suggestion. I think you mentioned this in previous PR, I put it in some notes, but missed it in this PR. Let me walk through this issue again and get back to you.

To limit scope of this PR, we can leave the old NPMNetworkPolicy fields and add this as a new field. That way, current DP is unaffected, and we can follow up with a PR in DP.
As you said, it would be good to a single PR to add and remove related codes if we decide to revise the struct. wdyt?

I walked through the code (including linux dataplane) to see how it uses data structures and think I prefer to having two members (i.e., PodSelectorIPSets and PodSelectorList) since two data structures have different data when there is multiple values in podSelector (Check this UT - https://github.com/Azure/azure-container-networking/blob/306b665c0aead925f9d326ff016ac541ad5f122a/npm/pkg/controlplane/translation/translatePolicy_test.go#L618) and seems it is not easy to induce them in data plane side.
labelSelector is a local data structure used in only translation logic to create IPSet and List based on namespaceSelector and podSelector.
So, I am fine to a current code, but if you have another opinion, please let me know.
We can revise it if needed.

Looks good to me. We have some cleanup work to do later in Translation & Dataplane:

switching to PeerList instead of SrcList and DstList like you suggested
deciding what to do for pod selector structs

Thank you for summary. Will add them in a note in this PR as well and revisit them in a next PR.


Dependency error during 'idf.py set-target' (IDFGH-13026)

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.
ESP-IDF v5.2.2 + ESP32C6
Operating System used.
Windows
How did you build your project?
Command line with idf.py
If you are using Windows, please specify command line type.
PowerShell
What is the expected behavior?
I expected to be able to set the project target to esp32c6 and then edit configuration via menuconfig or IDE config editor
What is the actual behavior?
I performed a fresh install of IDE 3.0.0 + IDF 5.2.2 via the offline installer on a new Windows 11 laptop, then tried the following inside the standard 'led_light' example project folder with PS:
install.ps1
export.ps1
idf.py set-target esp32c6
The result is a dependency error that is probably simple, but I don't understand it. Please see my pasted log below.
Steps to reproduce.
In project folder:

install.ps1
export.ps1
idf.py set-target esp32c6

Build or installation Logs.
PS C:\Espressif\workspace\esp-rainmaker\examples\led_light> idf.py set-target esp32c6
Adding "set-target"'s dependency "fullclean" to list of commands with default set of options.
Executing action: fullclean
Build directory 'C:\Espressif\workspace\esp-rainmaker\examples\led_light\build' not found. Nothing to clean.
Executing action: set-target
Set Target to: esp32c6, new sdkconfig will be created.
Running cmake in directory C:\Espressif\workspace\esp-rainmaker\examples\led_light\build
Executing "cmake -G Ninja -DPYTHON_DEPS_CHECKED=1 -DPYTHON=C:\Espressif\python_env\idf5.2_py3.11_env\Scripts\python.exe -DESP_PLATFORM=1 -DIDF_TARGET=esp32c6 -DCCACHE_ENABLE=1 C:\Espressif\workspace\esp-rainmaker\examples\led_light"...
-- Found Git: C:/Espressif/tools/idf-git/2.44.0/cmd/git.exe (found version "2.44.0.windows.1")
-- Component directory C:/Espressif/workspace/esp-rainmaker/components/esp-insights does not contain a CMakeLists.txt file. No component will be added
-- ccache will be used for faster recompilation
-- The C compiler identification is GNU 13.2.0
-- The CXX compiler identification is GNU 13.2.0
-- The ASM compiler identification is GNU
-- Found assembler: C:/Espressif/tools/riscv32-esp-elf/esp-13.2.0_20230928/riscv32-esp-elf/bin/riscv32-esp-elf-gcc.exe
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Espressif/tools/riscv32-esp-elf/esp-13.2.0_20230928/riscv32-esp-elf/bin/riscv32-esp-elf-gcc.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Espressif/tools/riscv32-esp-elf/esp-13.2.0_20230928/riscv32-esp-elf/bin/riscv32-esp-elf-g++.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Building ESP-IDF components for target esp32c6
Dependencies lock doesn't exist, solving dependencies.
Using component placed at C:\Espressif\workspace\esp-rainmaker\components\esp-insights\components\rmaker_common for dependency espressif/rmaker_common(~1.4.0)(introduced by component esp_diagnostics)
Using component placed at C:\Espressif\workspace\esp-rainmaker\components\esp-insights\components\esp_diag_data_store for dependency espressif/esp_diag_data_store(~1.0)(introduced by component esp_insights)
Using component placed at C:\Espressif\workspace\esp-rainmaker\components\esp-insights\components\esp_diagnostics for dependency espressif/esp_diagnostics(~1.0)(introduced by component esp_insights)
Using component placed at C:\Espressif\workspace\esp-rainmaker\components\esp-insights\components\rmaker_common for dependency espressif/rmaker_common(~1.4.0)(introduced by component esp_insights)
Using component placed at C:\Espressif\workspace\esp-rainmaker\components\esp_schedule for dependency espressif/esp_schedule(~1.2.0)(introduced by component esp_rainmaker)
Using component placed at C:\Espressif\workspace\esp-rainmaker\components\json_generator for dependency espressif/json_generator(~1.1.1)(introduced by component esp_rainmaker)
Using component placed at C:\Espressif\workspace\esp-rainmaker\components\json_parser for dependency espressif/json_parser(~1.0.3)(introduced by component esp_rainmaker)
Using component placed at C:\Espressif\workspace\esp-rainmaker\components\esp-insights\components\rmaker_common for dependency espressif/rmaker_common(~1.4.2)(introduced by component esp_rainmaker)
Using component placed at C:\Espressif\workspace\esp-rainmaker\components\esp-insights\components\rmaker_common for dependency espressif/rmaker_common(~1.4.2)(introduced by component esp_schedule)
Using component placed at C:\Espressif\workspace\esp-rainmaker\components\jsmn for dependency espressif/jsmn(~1.1)(introduced by component json_parser)
Using component placed at C:\Espressif\workspace\esp-rainmaker\components\esp-insights\components\esp_insights for dependency espressif/esp_insights(==1.1.0), specified in C:/Espressif/workspace/esp-rainmaker/examples/led_light/main\idf_component.yml
..Using component placed at C:\Espressif\workspace\esp-rainmaker\components\esp-insights\components\esp_diag_data_store for dependency espressif/esp_diag_data_store(~1.0)(introduced by component espressif/esp_insights), specified in C:/Espressif/workspace/esp-rainmaker/examples/led_light/main\idf_component.yml
Using component placed at C:\Espressif\workspace\esp-rainmaker\components\esp-insights\components\esp_diagnostics for dependency espressif/esp_diagnostics(~1.0)(introduced by component espressif/esp_insights), specified in C:/Espressif/workspace/esp-rainmaker/examples/led_light/main\idf_component.yml
Using component placed at C:\Espressif\workspace\esp-rainmaker\components\esp-insights\components\rmaker_common for dependency espressif/rmaker_common(~1.4.0)(introduced by component espressif/esp_insights), specified in C:/Espressif/workspace/esp-rainmaker/examples/led_light/main\idf_component.yml
...Using component placed at C:\Espressif\workspace\esp-rainmaker\components\esp-insights\components\rmaker_common for dependency espressif/rmaker_common(~1.4.0)(introduced by component espressif/esp_diagnostics)
....Using component placed at C:\Espressif\workspace\esp-rainmaker\components\esp-insights\components\rmaker_common for dependency espressif/rmaker_common(~1.4.0), specified in C:/Espressif/workspace/esp-rainmaker/components/esp-insights/components/esp_diagnostics\idf_component.yml
Using component placed at C:\Espressif\workspace\esp-rainmaker\components\esp-insights\components\esp_diag_data_store for dependency espressif/esp_diag_data_store(~1.0), specified in C:/Espressif/workspace/esp-rainmaker/components/esp-insights/components/esp_insights\idf_component.yml
Using component placed at C:\Espressif\workspace\esp-rainmaker\components\esp-insights\components\esp_diagnostics for dependency espressif/esp_diagnostics(~1.0), specified in C:/Espressif/workspace/esp-rainmaker/components/esp-insights/components/esp_insights\idf_component.yml
Using component placed at C:\Espressif\workspace\esp-rainmaker\components\esp-insights\components\rmaker_common for dependency espressif/rmaker_common(~1.4.0), specified in C:/Espressif/workspace/esp-rainmaker/components/esp-insights/components/esp_insights\idf_component.yml
Using component placed at C:\Espressif\workspace\esp-rainmaker\components\esp_schedule for dependency espressif/esp_schedule(~1.2.0), specified in C:/Espressif/workspace/esp-rainmaker/components/esp_rainmaker\idf_component.yml
Using component placed at C:\Espressif\workspace\esp-rainmaker\components\json_generator for dependency espressif/json_generator(~1.1.1), specified in C:/Espressif/workspace/esp-rainmaker/components/esp_rainmaker\idf_component.yml
Using component placed at C:\Espressif\workspace\esp-rainmaker\components\json_parser for dependency espressif/json_parser(~1.0.3), specified in C:/Espressif/workspace/esp-rainmaker/components/esp_rainmaker\idf_component.yml
Using component placed at C:\Espressif\workspace\esp-rainmaker\components\esp-insights\components\rmaker_common for dependency espressif/rmaker_common(~1.4.2), specified in C:/Espressif/workspace/esp-rainmaker/components/esp_rainmaker\idf_component.yml
Using component placed at C:\Espressif\workspace\esp-rainmaker\components\esp-insights\components\rmaker_common for dependency espressif/rmaker_common(~1.4.2)(introduced by component espressif/esp_schedule), specified in C:/Espressif/workspace/esp-rainmaker/components/esp_rainmaker\idf_component.yml
....Using component placed at C:\Espressif\workspace\esp-rainmaker\components\jsmn for dependency espressif/jsmn(~1.1)(introduced by component espressif/json_parser), specified in C:/Espressif/workspace/esp-rainmaker/components/esp_rainmaker\idf_component.yml
....Using component placed at C:\Espressif\workspace\esp-rainmaker\components\esp-insights\components\rmaker_common for dependency espressif/rmaker_common(~1.4.2), specified in C:/Espressif/workspace/esp-rainmaker/components/esp_schedule\idf_component.yml
.Using component placed at C:\Espressif\workspace\esp-rainmaker\components\jsmn for dependency espressif/jsmn(~1.1), specified in C:/Espressif/workspace/esp-rainmaker/components/json_parser\idf_component.yml
.Using component placed at C:\Espressif\workspace\esp-rainmaker\components\esp-insights\components\esp_insights for dependency espressif/esp_insights(~1.0.1), specified in C:/Espressif/workspace/esp-rainmaker/examples/common/app_insights\idf_component.yml
Using component placed at C:\Espressif\workspace\esp-rainmaker\components\esp_rainmaker for dependency espressif/esp_rainmaker(~=1.0), specified in C:/Espressif/workspace/esp-rainmaker/examples/common/app_insights\idf_component.yml
Using component placed at C:\Espressif\workspace\esp-rainmaker\components\esp_schedule for dependency espressif/esp_schedule(~1.2.0)(introduced by component espressif/esp_rainmaker), specified in C:/Espressif/workspace/esp-rainmaker/examples/common/app_insights\idf_component.yml
Using component placed at C:\Espressif\workspace\esp-rainmaker\components\json_generator for dependency espressif/json_generator(~1.1.1)(introduced by component espressif/esp_rainmaker), specified in C:/Espressif/workspace/esp-rainmaker/examples/common/app_insights\idf_component.yml
Using component placed at C:\Espressif\workspace\esp-rainmaker\components\json_parser for dependency espressif/json_parser(~1.0.3)(introduced by component espressif/esp_rainmaker), specified in C:/Espressif/workspace/esp-rainmaker/examples/common/app_insights\idf_component.yml
Using component placed at C:\Espressif\workspace\esp-rainmaker\components\esp-insights\components\rmaker_common for dependency espressif/rmaker_common(~1.4.2)(introduced by component espressif/esp_rainmaker), specified in C:/Espressif/workspace/esp-rainmaker/examples/common/app_insights\idf_component.yml
CMake Error at C:/Espressif/frameworks/esp-idf-v5.2.2/tools/cmake/build.cmake:544 (message):
  ERROR: Because no versions of espressif/esp_insights match >1.0.1,<1.1.0

   and espressif/esp_insights (1.0.1) depends on espressif/cbor (~0.6), espressif/esp_insights (>=1.0.1,<1.1.0) requires espressif/cbor (~0.6).

  So, because no versions of espressif/cbor match ~0.6

   and project depends on espressif/esp_insights (~1.0.1), version solving failed.

Call Stack (most recent call first):
  C:/Espressif/frameworks/esp-idf-v5.2.2/tools/cmake/project.cmake:605 (idf_build_process)
  CMakeLists.txt:16 (project)

.-- Configuring incomplete, errors occurred!
See also "C:/Espressif/workspace/esp-rainmaker/examples/led_light/build/CMakeFiles/CMakeOutput.log".
cmake failed with exit code 1, output of the command is in the C:\Espressif\workspace\esp-rainmaker\examples\led_light\build\log\idf_py_stderr_output_13704 and C:\Espressif\workspace\esp-rainmaker\examples\led_light\build\log\idf_py_stdout_output_13704
PS C:\Espressif\workspace\esp-rainmaker\examples\led_light>

More Information.
No response

I've found this suggestion for the issue:
pip install idf-component-manager==2.0.0.dev0

But I'll ask someone else who knows this better to take a look.

I've found this suggestion for the issue:
pip install idf-component-manager>=2.0.0.dev0

But I'll ask someone else who knows this better to take a look.

@dobairoland Thank you for the suggestion, could you please move it to the https://github.com/espressif/idf-component-manager ?
@mjdswan Yes the current pre-release version should have this issue fixed, but currently it's only supported with master branch of ESP-IDF

Hi @kumekay,
Thanks for the suggestion above, but I tried it and get the same result.
I ran the pip install .. under the main framework path and it appeared to do as it should.
Following this I was prompted to run install.ps1 and export.ps1 which all looked normal.
However I still get the error when I try to set the target.
Hi @dobairoland, did you mean that I should use the latest pre-release if I need this to work?  I'm also not clear what you meant by "only supported with master branch of ESP-IDF".

@kumekay / @dobairoland any update please?
We have a commercial project paused waiting for resolution of this problem.
If I am missing something simple or the question is not understood, please let me know.


 Same with postgres.
Additional context
The error outputs are inconsistent. Sometimes the migrations works. Sometimes it doesn't. Some errors are passed through while other crash the service. I'm using db-sync-snapshot-schema-12-block-3488192-x86_64 to restore the snapshot.
Problem Report
Duplicate key errors during snapshot restoration:
psql:/tmp/db-sync-snapshot-zWgWJDFqNt/db-sync-snapshot-schema-12-block-3488192-x86_64.sql:50838240: ERROR:  could not create unique index "ada_pots_pkey"
DETAIL:  Key (id)=(32) is duplicated.
ALTER TABLE
psql:/tmp/db-sync-snapshot-zWgWJDFqNt/db-sync-snapshot-schema-12-block-3488192-x86_64.sql:50838256: ERROR:  could not create unique index "block_pkey"
DETAIL:  Key (id)=(10728) is duplicated.
CONTEXT:  parallel worker
psql:/tmp/db-sync-snapshot-zWgWJDFqNt/db-sync-snapshot-schema-12-block-3488192-x86_64.sql:50838264: ERROR:  could not create unique index "collateral_tx_in_pkey"
DETAIL:  Key (id)=(241955) is duplicated.

Unique constraint matching errors during snapshot restoration:
psql:/tmp/db-sync-snapshot-zWgWJDFqNt/db-sync-snapshot-schema-12-block-3488192-x86_64.sql:50839832: ERROR:  there is no unique constraint matching given keys for referenced table "stake_address"
psql:/tmp/db-sync-snapshot-zWgWJDFqNt/db-sync-snapshot-schema-12-block-3488192-x86_64.sql:50839840: ERROR:  there is no unique constraint matching given keys for referenced table "tx"
psql:/tmp/db-sync-snapshot-zWgWJDFqNt/db-sync-snapshot-schema-12-block-3488192-x86_64.sql:50839848: ERROR:  there is no unique constraint matching given keys for referenced table "stake_address"
psql:/tmp/db-sync-snapshot-zWgWJDFqNt/db-sync-snapshot-schema-12-block-3488192-x86_64.sql:50839856: ERROR:  there is no unique constraint matching given keys for referenced table "redeemer"
psql:/tmp/db-sync-snapshot-zWgWJDFqNt/db-sync-snapshot-schema-12-block-3488192-x86_64.sql:50839864: ERROR:  there is no unique constraint matching given keys for referenced table "tx"

Multiple primary key errors during snapshot restoration:
psql:/tmp/db-sync-snapshot-KESlbLzEA8/db-sync-snapshot-schema-12-block-3488192-x86_64.sql:50838488: ERROR:  multiple primary keys for table "schema_version" are not allowed

Exit during migration:
[db-sync-node:Info:6] [2022-04-23 16:49:01.43 UTC] Schema migration files validated
[db-sync-node:Info:6] [2022-04-23 16:49:01.43 UTC] Running database migrations
[db-sync-node:Info:6] [2022-04-23 16:49:01.43 UTC] Enviroment variable DbSyncAbortOnPanic: False
ExitFailure 3

Errors in file: /tmp/migrate-2022-04-23T164901.log

Rinse and repeat.

After a series of restarts it's now working and syncing but I have no idea why.

The error:
psql:/tmp/db-sync-snapshot-zWgWJDFqNt/db-sync-snapshot-schema-12-block-3488192-x86_64.sql:50838240: ERROR:  could not create unique index "ada_pots_pkey"
DETAIL:  Key (id)=(32) is duplicated.

is due to the database not being empty when you restored the snapshot.

It also appears to hang after syncing for some time.

No information needed. The usual reason for db-sync to stop syncing is that the node it is connecting to has stopped for some reason.

The node is working fine. I have 2 other services interacting with it without any problems.

So what makes you think db-sync is not syncing?

@erikd
It seems to be caught in this infinite error loop:
[db-sync-node:Error:1624] [2022-04-26 16:06:41.64 UTC] runDBThread: SqlError {sqlState = "40001", sqlExecStatus = FatalError, sqlErrorMsg = "could not serialize access due to read/write dependencies among transactions", sqlErrorDetail = "Reason code: Canceled on identification as a pivot, during write.", sqlErrorHint = "The transaction might succeed if retried."}
[db-sync-node:Error:1627] [2022-04-26 16:06:41.64 UTC] recvMsgRollBackward: AsyncCancelled
[db-sync-node:Error:1619] [2022-04-26 16:06:41.64 UTC] ChainSyncWithBlocksPtcl: SqlError {sqlState = "40001", sqlExecStatus = FatalError, sqlErrorMsg = "could not serialize access due to read/write dependencies among transactions", sqlErrorDetail = "Reason code: Canceled on identification as a pivot, during write.", sqlErrorHint = "The transaction might succeed if retried."}
[db-sync-node.Mux:Info:1616] [2022-04-26 16:06:41.64 UTC] Bearer on ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/opt/cardano/node/node.socket"} event: State: Dead
[db-sync-node.Mux:Notice:1616] [2022-04-26 16:06:41.64 UTC] Bearer on ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/opt/cardano/node/node.socket"} event: Miniprotocol MiniProtocolNum 5 InitiatorDir terminated with exception SqlError {sqlState = "40001", sqlExecStatus = FatalError, sqlErrorMsg = "could not serialize access due to read/write dependencies among transactions", sqlErrorDetail = "Reason code: Canceled on identification as a pivot, during write.", sqlErrorHint = "The transaction might succeed if retried."}
[db-sync-node.Subscription:Error:1615] [2022-04-26 16:06:41.64 UTC] Identity Application Exception: LocalAddress "/node-ipc/node.socket" SqlError {sqlState = "40001", sqlExecStatus = FatalError, sqlErrorMsg = "could not serialize access due to read/write dependencies among transactions", sqlErrorDetail = "Reason code: Canceled on identification as a pivot, during write.", sqlErrorHint = "The transaction might succeed if retried."}
[db-sync-node.ErrorPolicy:Error:6] [2022-04-26 16:06:41.64 UTC] IP LocalAddress "/node-ipc/node.socket" ErrorPolicyUnhandledApplicationException (SqlError {sqlState = "40001", sqlExecStatus = FatalError, sqlErrorMsg = "could not serialize access due to read/write dependencies among transactions", sqlErrorDetail = "Reason code: Canceled on identification as a pivot, during write.", sqlErrorHint = "The transaction might succeed if retried."})
[db-sync-node.Subscription:Notice:46] [2022-04-26 16:06:42.64 UTC] Identity Restarting Subscription after 41.005967032s desired valency 1 current valency 0
[db-sync-node.Subscription:Notice:46] [2022-04-26 16:06:42.64 UTC] Identity Starting Subscription Worker, valency 1
[db-sync-node.Subscription:Notice:1633] [2022-04-26 16:06:42.64 UTC] Identity Connection Attempt Start, destination LocalAddress "/node-ipc/node.socket"
[db-sync-node.Subscription:Notice:1633] [2022-04-26 16:06:42.64 UTC] Identity Connection Attempt End, destination LocalAddress "/node-ipc/node.socket" outcome: ConnectSuccessLast
[db-sync-node.Handshake:Info:1633] [2022-04-26 16:06:42.64 UTC] WithMuxBearer (ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/opt/cardano/node/node.socket"}) Send (ClientAgency TokPropose,MsgProposeVersions (fromList [(NodeToClientV_1,TInt<PHONE_NUMBER>),(NodeToClientV_2,TInt<PHONE_NUMBER>),(NodeToClientV_3,TInt<PHONE_NUMBER>),(NodeToClientV_4,TInt<PHONE_NUMBER>),(NodeToClientV_5,TInt<PHONE_NUMBER>),(NodeToClientV_6,TInt<PHONE_NUMBER>),(NodeToClientV_7,TInt<PHONE_NUMBER>),(NodeToClientV_8,TInt<PHONE_NUMBER>),(NodeToClientV_9,TInt<PHONE_NUMBER>),(NodeToClientV_10,TInt<PHONE_NUMBER>)]))
[db-sync-node.Handshake:Info:1633] [2022-04-26 16:06:42.64 UTC] WithMuxBearer (ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/opt/cardano/node/node.socket"}) Recv (ServerAgency TokConfirm,MsgAcceptVersion NodeToClientV_10 (TInt<PHONE_NUMBER>))
[db-sync-node.Mux:Info:1633] [2022-04-26 16:06:42.64 UTC] Bearer on ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/opt/cardano/node/node.socket"} event: Handshake Client end, duration 0.000366288s
[db-sync-node.Mux:Info:1634] [2022-04-26 16:06:42.64 UTC] Bearer on ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/opt/cardano/node/node.socket"} event: State: Mature
[db-sync-node:Info:1637] [2022-04-26 16:06:42.64 UTC] Starting chainSyncClient
[db-sync-node:Info:1637] [2022-04-26 16:06:42.66 UTC] Cardano.Db tip is at slot 1360533, block 1359383
[db-sync-node:Info:1642] [2022-04-26 16:06:42.66 UTC] Running DB thread
[db-sync-node:Info:1642] [2022-04-26 16:06:42.67 UTC] Rolling back to genesis
[db-sync-node.Subscription:Notice:46] [2022-04-26 16:06:42.67 UTC] Identity Required subscriptions started
[db-sync-node:Info:1642] [2022-04-26 16:06:44.76 UTC] Deleting 1359383 blocks up to slot 1360533

That error is due to a schema mismatch error between the actual database schema and the one the application expects.
Its is due to the switching major versions (eg 11.x.y to 12.x.y) without a database resync. The only way to fix this is to drop the database and restore from a snapshot or resync the whole chain.


Repayment Tracking & Allocation Implementation

Issue
Tracking what lenders are entitled to withdraw what repayments at what time requires overhead. We want to minimize cost and keep interfaces as familiar as possible.
Solutions
MiniMe Style ERC-20 link
[+] Can handle ERC-20 payment tokens if sent though a function, like repay()
[-] O(logn) loops on transfer and withdraw
[-] Large storage overhead
Conclusion: Likely won't use due to gas costs
ERC-721
[+] Can handle ERC-20 payment tokens if sent though a function, like repay()
[+] No loops, withdraws simple to calculate
[-] Can't split or combine stake without extra logic
[-] Can't withdraw multiple stakes at once without a batch function (which we could totally do)
ERC-1843: Claims Token Standard link
[+] Can keep ERC-20 ownership tokens, looks efficient (still evaluating)
[-] Requires payment token to be ERC-223 compatible for the tokenFallback hook.

We could potentially "wrap" the payment token before sending, it'd be the same number
of calls as approve() + function() anyways.

ERC-1726: Dividend-Paying Token Standard link
[-] Only supports Ether payments
Conclusion: Won't use as ERC-20 payments are critical. Can learn from though.

Looking at the ERC-1843 claim standard implemetation,
We need to maintain a balance of each address, that ever has the tokens, in a bid to maintain the balance.
This means a typical O(logn) storage consumption to each address.
This would than be exaggerated when we add computation of fees, interest, penalties.
I still prefer the ERC721 implementation over this, at least for the MVP, and if divisibility becomes a priority, we might re-consider.
Note: I still very much like the claim standard, and I think we can build on that.

We could add our implementation as the ERC 721 implementation for the standard


Do not submit disabled form fields

Similar to #10 but this time we have to ignore fields which is disabled.
<input type="text" name="lname" disabled>

Closed by https://github.com/gushonorato/mechanizex/commit/946cd8186569f2910d5380618750cfe164dde17c


Starting rdb without a tickerplant

While it might not be a conventional approach, at the moment if you try to start an rdb process without supplying a valid tickerplant type (i.e. tickerplanttypes:` in the rdb settings), it results in the rdb getting stuck in a loop where it attempts to subscribe/connect to something that doesn't exist (and incorrectly says subscription successful!).
I think it should be possible to start the rdb with no tickerplant given, such that the subscribe step is skipped when tickerplanttypes is null. It shouldn't cause the process to essentially hang while it keeps trying to connect to a non-existent tickerplant.
In terms of EOD, in this case the rdb could save down all tables in the top namespace (except those in ignorelist), or you could have a more sophisticated system for adding tables to a writedown list (since there would be no .rdb.subtables).
This issue is probably also applicable to the wdb or similar processes that are set up to connect to a tickerplant.

Hi Andrea,
It would be useful to have a better sense of your use case for the RDB in this context.
I suppose the concern is whether or not you need at any point to be able to connect and subscribe to the tickerplant, or if you'd just like the option to operate within the RDB without ever connecting (and with the RDB not hanging because of that) to deal with non-streaming data for example.
At the moment, we're looking to implement a change across TorQ such that processes can operate without necessarily connecting/subscribing to the processes that they would normally do so to, which may suffice for your needs.
Thanks,
Joseph

Hi Joseph,
Thanks for getting back to me so quickly!
In the setup I am using there is no tickerplant at all as the RDB is getting data through other means.
So the change you are implementing sounds like it would be sufficient - I’d just like to be able to start the RDB without ever connecting or subscribing to a tickerplant.
Please let me know if you have any further questions, I’m happy to help!
Thanks,
Andrea

On 27 Jun 2018, at 9:52 pm, JedwardsEdwards<EMAIL_ADDRESS>wrote:
Hi Andrea,
It would be useful to have a better sense of your use case for the RDB in this context.
I suppose the concern is whether or not you need at any point to be able to connect and subscribe to the tickerplant, or if you'd just like the option to operate within the RDB without ever connecting (and with the RDB not hanging because of that) to deal with non-streaming data for example.
At the moment, we're looking to implement a change across TorQ such that processes can operate without necessarily connecting/subscribing to the processes that they would normally do so to, which may suffice for your needs.
Thanks,
Joseph
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub https://github.com/AquaQAnalytics/TorQ/issues/100#issuecomment-400644982, or mute the thread https://github.com/notifications/unsubscribe-auth/Amk4guNIlFcaqSPnq5z2krK2PsxY0Qalks5uA3IBgaJpZM4U5Ar3.

Hi Andrea,
We've gone over this issue again our end, and have reached a decision for now to leave the RDB startup code as is for the time being. I think it comes down to the functionality of the process, and that it in using the RDB as intended as a real-time database for streaming data, it doesn't really make sense for the RDB to start up without a connection to its data source, the tickerplant.
While this process could be expanded to serve a number of purposes, I think it is clearer and cleaner to set up a similar, but distinct process to handle the loading of data through other means. Putting something like this together would be fairly straighforward, using parts of the RDB code as a template, and this is what I suggest to fit your needs. You'd just have to set this up as a vanilla TorQ process, without any connection requirements, load up your schema, and then start loading your data.
We will be fixing a bug in the near future such that if the tickerplant connection isn't well defined the RDB process will exit rather than sitting in the loop.
Just let us know if you have any other questions about setting up a process like this,
Thanks,
Joseph

Hi Joseph,
No problem, I understand. As I said it’s not a conventional use of an RDB but I definitely agree that the bug where it gets stuck in a loop should be addressed.
I already have a solution for my use case but thank you for the suggestions!
Kind regards,
Andrea

On 28 Jun 2018, at 6:55 pm, JedwardsEdwards<EMAIL_ADDRESS>wrote:
Hi Andrea,
We've gone over this issue again our end, and have reached a decision for now to leave the RDB startup code as is for the time being. I think it comes down to the functionality of the process, and that it in using the RDB as intended as a real-time database for streaming data, it doesn't really make sense for the RDB to start up without a connection to its data source, the tickerplant.
While this process could be expanded to serve a number of purposes, I think it is clearer and cleaner to set up a similar, but distinct process to handle the loading of data through other means. Putting something like this together would be fairly straighforward, using parts of the RDB code as a template, and this is what I suggest to fit your needs. You'd just have to set this up as a vanilla TorQ process, without any connection requirements, load up your schema, and then start loading your data.
We will be fixing a bug in the near future such that if the tickerplant connection isn't well defined the RDB process will exit rather than sitting in the loop.
Just let us know if you have any other questions about setting up a process like this,
Thanks,
Joseph
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub https://github.com/AquaQAnalytics/TorQ/issues/100#issuecomment-400963237, or mute the thread https://github.com/notifications/unsubscribe-auth/Amk4gssulCp9ab5u5tMmely0WiwSI5wEks5uBJoPgaJpZM4U5Ar3.

We have recently had the same issue @JedwardsEdwards
We are now escaping the subscribe function if (::)~.rdb.subscribeto or .ctp.subscribeto etc.
For our purposes, we wanted to initialise the connection to the tickerplant, but as each of our rdb's are essentially sandboxes, the data required varies and therefore we initialise subscriptions from within the process after launch for only the specific data streams required (possibly none from tickerplant, and subscribing to a derived stream being published from a chainedtp).


pstore: cleanup unused sepolicies to address build issue

pstore: cleanup unused sepolicies to address build issue
Removed unused sepolicies to address selinux neverallow errors.
Tracked-On: OAM-112822

Android CI has started Engineering Build for this issue ,Please check the linked Tracked-On issue/Android CI Web  for more details.


v2: feature proposals

Originally sptlrx was designed to only support Spotify, but was quickly updated with many other features. And the current Spotify-centric design makes it difficult to implement new features. I want to start from scratch.
In this issue, I'll be keeping a wishlist of features/ideas for sptlrx v2, which will eventually turn into a to-do list. You guys are welcome to discuss or add anything too :)

Multi-player (#40, #63). Iterate through enabled players and select the one that currently has something playing. Prioritization.
Multi-source (#40). Iterate through the sources and select the one with the timesynced lyric. Prioritization. Same priority sources should all be queried and the best one should be selected.
Improve local source. Attempt to find .lrc file based on track path (#53). Attempt to extract embedded lyrics.
“No text” placeholder (#55). Make it optional.
Remove support for lyricsapi. This has just been a fancy proxy to lrclib for a long time. So replace it with plain lrclib (+optional search logic described here).
Improved .lrc file parser (#51). We may find some tags useful!
Spotify authentication via API (#56). The current implementation was made with using the internal API to retrieve song lyrics directly from Spotify in mind. However, there are now other ways to retrieve song lyrics that work just as well. Take advantage of them and use the public API just to get the title and position. May require a lot of work, so probably won't be in 2.0.
Support for nscpot (#38). Again, a lot of work for a potentially small number of users, so only if this gets enough votes.
TOML config. The config structure is going to change a lot anyway, and I feel the yaml format is not flexible/convenient enough. Migration command.
Sensible defaults. Now we show text on how to set up Spotify cookies on first run. Instead let's default to lrclib & mpris (unfortunately only for linux).
Logging (#45). In its current state (ignoreErrors flag) it's just awful. Just write it somewhere, it won't hurt anyone.
Test coverage... No comments here.

This is how pretty our .toml config would look. Any thoughts?

[style]
align = "center"

[style.current]
foreground = 0xffc0cb

[players.spotify]
enabled = true
cookie = "qwerty"
priority = 1

[players.mpris]
enabled = true
whitelist = ["rhythmbox", "firefox"]
priority = 0

[source.spotify]
enabled = true
priority = 2

[sources.local]
enabled = true
directory = "~/Music"
priority = 1

[sources.lrclib]
enabled = true
priority = 0

[logging]
enabled = true
file = "/tmp/sptlrx.log"


mycelium: 0.4.5 -> 0.5.0

Changed

Connection identifier is now included in the error log if we can't forward a seqno request.
Garbage collection time for source entries has been increased from 5 to 30 minutes for now.
The router implementation has been changed to use regular locks instead of an always readable concurrency primitive for all but the actual routing table. This should reduce the memory consumption a bit.
Public key and shared secret for a destination are now saved on the router, instead of maintaining a separate mapping for them. This slightly reduces memory consumption of the router, and ensures stale data is properly cleaned up when all routes to a subnet are removed.
Hello packets now set the interval in which the next Hello will be sent properly in centiseconds.
IHU packets now set the interval properly in centiseconds.
IHU packets now set an RX cost. For now this is the link cost, in the future this will be set properly.
Route expiration time is now calculated from the interval received in updates.
Ip address derivation from public keys now uses the blake3 hash algorithm.

Fixed

Don't try to forward seqno requests to a peer if we know its connection is dead.

Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 301497 run on x86_64-linux 1

1 package built:

mycelium


Add a news Block with the Adoptium Summit the day of the event - September 10th 2024

As a 3rd block, after this one, https://github.com/adoptium/adoptium.net/issues/3000
We would like to have the third and last block promoting the day of the event:
Start date: September 10, 00:00 CEST
End Date: September 10, 17:00 CEST
Title: Adoptium Summit 24

Join us Today for the first-ever Adoptium Summit.
An opportunity to connect with other Adoptium community members
Register here: https://www.eclipse.org/events/2024/adoptium-summit/

The news will be displayed the 10th of september as expected.


Review FlatZipMap

I wrote exhaust::iteration::FlatZipMap as a helper to make it easier to implement certain patterns of iteration. I suspect that the Map part is obsolete now that factories exist, but I have not confirmed this.

There is also the question of whether FlatZipMap should exist at all. The test of usefulness for what FlatZipMap was supposed to be is: can one write a nontrivial, non-derived implementation of Exhaust using an iterator that's primarily a composition of one or more FlatZipMaps?
The things that this can hopefully do that delegating to a derived implementation can't are:

Fields with constraints between each other (i.e. not a pure cartesian product)
Providing all the optional utilities like size_hint(), without needing to write a bunch of delegating methods.