gogo protobufs deprecated

Hi,
gogo/protobuf has been deprecated back in 2022 and it would be good to move to std Protobuf v3, at this point.
You can see the announcement (and talk here)
https://x.com/awalterschulze/status/158455305610005708

ok, thanks!


How to implement the function of page jump

How to implement the function of page jump?
For example: when I click a button, The screen jump to another page.
Thank you!

This can be done using plain React:
const [page, setPage] = useState("home");

<Button onPress={() => setPage("profile")} />

...

if (page === "home") return <Home />
if (page === "profile") return <Profile />

If you want tabs and animations, check the widgets demo


[Question] ImportError: cannot import name 'rendering' from 'gym.envs.classic_control'

Question
I have this error showing up.
"from gym.envs.classic_control import rendering
ImportError: cannot import name 'rendering' from 'gym.envs.classic_control'"
Is there something that I can fix this.

You appear to be using code with gym v0.21 I believe which is no longer supported.
What version of gym are you using?

You appear to be using code with gym v0.21 I believe which is no longer supported. What version of gym are you using?

it is 0.26.2.

I can't remember exactly but gym 0.18 or 0.21 should probably work

I can't remember exactly but gym 0.18 or 0.21 should probably work

I could not install the 0.21 version, I have this error.
note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for gym
  Running setup.py clean for gym
Failed to build gym
ERROR: Could not build wheels for gym, which is required to install pyproject.toml-based projects

https://github.com/openai/gym/issues/3202


chore: update select controlled example

use id-s to remove the confusion of select being a value or an id of a value

What is it?

[ ] Feature / enhancement
[ ] Bug
[x] Docs / tests
[ ] Other

Why is it needed?

Checklist:

[ ] My code follows the developer guidelines of this project
[ ] I have performed a self-review of my own code
[ ] I have ran pnpm change and documented my changes
[x] I have add necessary docs (if needed)
[ ] Added new tests to cover the fix / functionality

Awesome job! Yeah completely agree with a unique user id here.
LGTM, merging.
Side note: I'm wrapping up the new refactored accordion, but will get to the select this week.


feat: lay down kustomization in apply

Include kustomization.yaml in the output directory so that it can be
used with kustomize directly (overriding namespace etc.).

cc @eriknelson @shawn-hurley @sseago
This is the idea I would like to build on when we talk about leveraging kustomize in crane. I'm going to leave this as a draft as it needs to be introduced first as an enhancement.

This implements the enhancement described in https://github.com/konveyor/enhancements/pull/57

Closing as the related enhancement was rejected.


problem importing S2L1CWCSInput from eolearn

i got this message in the first notebook trying to import necessary packages :
 in
22 from eolearn.core import EOTask, EOPatch, LinearWorkflow, FeatureType, OverwritePermission,
23     LoadFromDisk, SaveToDisk, EOExecutor
---> 24 from eolearn.io import S2L1CWCSInput, ExportToTiff
25 from eolearn.io.local_io import *
26 from eolearn.mask import AddCloudMaskTask, get_s2_pixel_cloud_detector, AddValidDataMaskTask
ImportError: cannot import name 'S2L1CWCSInput' from 'eolearn.io'

I'm also getting the same issue @simongrest how did u rectify that error

And @m0rp43us  have u solved that
if solved pls tell what to do?
It's urgent pls reply fast

S2L1CWCSInput is deprecated,SentinelHubInputTask will work instead.


Write blog about manual accessibility testing checklist & translating testing requirements into plain language

For documents and interaction with document owners: https://github.com/DOI-ONRR/onrr.gov-site/wiki/Accessibility-guidance#manual-accessibility-testing-of-documents-checklist
and for ACRT, spreadsheet here.

@lpgoldstein If you have time, could you please review?
blog here

@cthomasONRR - this blog is great! I added some comments and tracked changes. I hope my feedback helpful!

@AlexandracmONRR @lpgoldstein Thanks for your feedback! I have all comments addressed. I made a new issue for cleaning it up and next steps for next sprint.


Alternativ definition of joins

Right now I must define a join like this inner_join(:booking, {payingbp_id: :id}) if I don't have a foreign key in the database and the fields do not match the convention.
If I have multiple joins, a join always refers to the relation that calls the join. If I want to access an other relation I have to write it this way:
inner_join(:address, {businesspartner_id: :businesspartnerbase__id})
Right now there is no syntax to avoid the underscores.
Maybe it would be good to have something like this:
inner_join(:address, {address[businesspartner_id].qualified => :businesspartnerbase[:id].qualified})

@flash-gordon are you aware of any alternative way of doing this where underscores are not used?

associations do
  has_many :addresses, foreign_key: ...
end

inner_join(:addresses)

this might do the trick^

oh it's a different problem:
users.
  join(:tasks, user_id: :id).  # :id => users.id is assumed
  join(:tags, task_id: :tasks__id) # here you need to qualify explicitly, otherwise :id will be ambiguous

so, since we want to remove support for underscores, we need an alternative way of specifying which join keys must be used (or which relation, then join keys can be inferred). Since we're calling join methods on users then this is the base, so its associations are used, but we're joining tasks=>tags, so it's tricky.

Starting from 2.0.0 all schemas are qualified by default so this should no longer be a problem. Closing.


Add isolated PCRE match limits as a layer of ReDoS defense

This PR ...

Adds support for PCRE and PCRE2 match limits as a last line of defense against ReDoS. Match limits are applied per PCRE invocation and do not change the global PCRE match limit default, which means the limits are isolated to ModSecurity and shouldn't affect other uses of PCRE in the same program.
Exposes regex errors as the variables RX_ERROR and RX_ERROR_RULE_ID so that regex engine failures can be handled by dedicated rules, rather than silently ignored. Today, if regex execution returns an error, the rx and rxGlobal operators simply fail to match. This could even be considered a security concern since such a regex error will cause a rule to be quietly bypassed.

NOTE: In the future, ModSecurity may support more proactive measures to identify vulnerable regular expressions before they can be used, but there is no harm in keeping multiple layers of defense. Doing so may even be wise, in the same way we design modern buildings to avoid fires yet still employ fire fighters for emergencies.
Background
A similar PR was rejected in the past, and if I understand correctly, it was rejected for the following reasons:

A belief that setting PCRE match limits would effect other PCRE users in the same memory space #
Users were concerned and confused about how global PCRE match limits, set for things like PHP, also affected ModSecurity #
The maintainers desire to avoid entanglement and user confusion with PCRE limits by finding other ways to avoid ReDoS #

If that is correct, I think it is worth revisiting these reasons:

Can we set match limits without affecting other PCRE users in the same memory space? Yes.Match limits can be applied to individual PCRE pcre_exec() and PCRE2 pcre2_match() invocations without affecting other PCRE uses in the same memory space. Intuitively, if custom settings can be passed to matching functions, those settings should be limited to the function call. But we can also find evidence in the PCRE source code. In the legacy version of PCRE, the pcre_exec() function has a local match_data structure that includes the selected match limit setting. The structure is declared as part of the stack frame here. It is assigned the default match limit value here and, if specified, the custom match limit value here. Since the custom setting is maintained as part of the stack frame, it should have no global effects on other PCRE users or even other subsequent PCRE calls from ModSecurity.
Is ModSecurity affected by changes to the global PCRE default match limit? Today, the answer is yes, but there is a straightforward fix.ModSecurity can isolate itself from PCRE's default by choosing its own default match limit (even just PCRE's normal default of 10,000,000) and always specifying that limit when invoking pcre_exec() or pcre2_match(). If ModSecurity did this for all PCRE limits instead of relying upon PCRE defaults, ModSecurity users could be isolated from the effects of modified PCRE defaults.
Whether to allow regex engine configuration is up to the ModSecurity maintainers, but some ModSecurity users want this level of configurability.Eliminating the possibility of ReDoS prior to invoking PCRE or another regex engine is a noble goal, but such a feature does not exist in ModSecurity today. In the meantime, ModSecurity users have real ReDoS concerns that can be addressed through regular expression engine configuration, and such configuration could co-exist peacefully with more proactive anti-ReDoS measures when they are developed.

Implementation
This PR re-enables the SecPcreMatchLimit config directive, saves the match limit as a transaction property, and applies the limit in the rx and rxGlobal operators. This works with both PCRE and the experimental PCRE2 support.
This PR also adds RX_ERROR and RX_ERROR_RULE_ID variables which are set in the event of a regular expression error. RX_ERROR exposes the reason for error and can be set to either "MATCH_LIMIT" or "OTHER". RX_ERROR_RULE_ID exposes the rule ID where the regex error happened so error-handling rules can be written to respond differently based on a specific rule ID or range of rule IDs.
Here is a naive example of rules written against the RX error variables:
# Deny requests when rule IDs in the 2xxx range encounter a regex error
SecRule RX_ERROR_RULE_ID "@beginsWith 2" "id:3330,phase:2,auditlog,deny,msg:'Class 2xxx rule %{RX_ERROR_RULE_ID} encountered regex error %{RX_ERROR}',chain"
		SecRule RX_RULE_ERROR_ID "@eq 4" "t:length"

# Log when other rules encounter a regex error
SecRule RX_ERROR "@gt 0" "t:length,id:3331,phase:2,log,auditlog,allow,msg:'Rule %{RX_ERROR_RULE_ID} encountered regex error %{RX_ERROR}'"

Review
Arguably, regex engine configuration and regex error handling could be split into two PRs, but I'm starting with a single PR for the sake of discussion.
What do you think? Is this a reasonable change? Would you be willing to reconsider allowing users to set ModSecurity-specific PCRE match limits?

Hi @martinhsv, thank you for your feedback on this PR. I'm happy to hear you are open to changes of this nature.

There is, in principal, an intention to support other regex engines besides pcre/pcre2 (Hyperscan and RE2) and it's always preferable if the same options are supported in the other engines as well.

The regex util updates in this PR just add a basic match_limit arg to a couple of the regex utility matching methods. It sounds like that argument may need to be converted into general engine options at some point, but since the regex util is an internal implementation detail, perhaps that change can wait until it is needed.
Is there anything you'd like to see to help move this PR forward? Just a heads up: I am taking a sabbatical for the next few months and may take a while to respond, but I am still very much interested in this change.
Note: I rebased and resolved merge conflicts but haven't had a chance to test that update yet.

IMO, it's better to use atomic group or possessive quantifiers for regex rules.

Hello @martinhsv,
Would you like me to make any changes to this PR? Is there anything I can do to help this PR or something like it to land?
My employer Automattic is interested in using this regex safety net so we can more comfortably leverage community rule sets like those from OWASP. We have seen some ReDoS issues with open source rule sets and would like to mitigate the risk.

Hello @brandonpayton ,
Yes, this is one of the items that I'd hoped to get back to during the coming period. I'll review this within the next few days.

Hi @brandonpayton ,
There are a few design assumptions here that I think might be worth considering.
The population of your two new variables is 'first wins' -- i.e. they only get set if they have not already been set.
In general ModSecurity has a 'last wins' approach. This is true for variables like MATCHED_VAR and for pcre API matches as well (a repeated group like (a.c)* with input 'abcadc' will return only 'adc' for that capture group). Using 'last wins' would also give the admin slightly more flexibility for collecting information. I.e. it's possible for the admin to either do something with the changing variable periodically, whereas with 'first wins' that additional avenue of inquiry doesn't work.
If we continue with only holding a single value, I would suggest we use 'last wins' instead. Counterarguments are welcome.
But this bleeds into some bigger questions.
I'm somewhat hesitant about the proposed new variable RX_ERROR_RULE_ID.
This is not something ModSecurity has done before. Default rule-specific processing information tends to be communicated through ModSecurity's Debug Log.
I'm not necessarily rejecting the concept of maybe providing this sort of information through a variable. After all, some installations may prefer to run without the debug log engaged, and having some basic information available in such cases may be helpful.
However, if we do wish to pursue that goal, we might wonder if we really want to restrict the admin to only knowing about 1 of possibly multiple failures. Also, we may want to think more broadly: a processing failure that is related to a specific rule may be relevant to other types of operators besides rx. Maybe we should really be considering a new variable in the form of a collection like RULE_ERROR where each key is the rule id where a failure was seen, and the value is a fixed text describing the error.
I'm not sure I've thought this through fully; and implementing an idea like that might be better done separately, as then it would have better visibility with (and potential input from) the community.
As an immediate measure to get your central goal moving forward, I'm wondering if it might be fruitful to limit the near term change to the existing model: set a single signalling variable (in ModSecurity v2, this was called MSC_PCRE_LIMITS_EXCEEDED) and leave additional information communication to the Debug Log.
Thoughts and counterarguments are welcome.

Hi @martinhsv,
Thank you for your thought and guidance. What you suggested sounds reasonable and wise.

In general ModSecurity has a 'last wins' approach.

It's good to hear about "last wins" versus "first wins", and I do not believe there is any good reason for this to deviate from overall ModSecurity philosophy and practice.

I'm somewhat hesitant about the proposed new variable RX_ERROR_RULE_ID.
This is not something ModSecurity has done before. Default rule-specific processing information tends to be communicated through ModSecurity's Debug Log.

OK, it seems like we should continue favoring the Debug Log, especially for this iteration.

Maybe we should really be considering a new variable in the form of a collection like RULE_ERROR where each key is the rule id where a failure was seen, and the value is a fixed text describing the error.

At the time, I was not sure how to do something like this or whether it was possible. If the community ends up wanting something like this, it sounds good to add in a follow-up patch.

implementing an idea like that might be better done separately, as then it would have better visibility with (and potential input from) the community.

That's a good point about community input. Agreed.

As an immediate measure to get your central goal moving forward, I'm wondering if it might be fruitful to limit the near term change to the existing model: set a single signalling variable (in ModSecurity v2, this was called MSC_PCRE_LIMITS_EXCEEDED) and leave additional information communication to the Debug Log.

Do I understand correctly that we are talking about dropping the RX_ERROR and RX_ERROR_RULE_ID variables for now and adding MSC_PCRE_LIMITS_EXCEEDED as a boolean signaling variable?  If so, that sounds great.

Because there is also the possibility of other PCRE errors, we may need an MSC_PCRE_ERROR flag to reflect other errors unrelated to hitting limits because otherwise those PCRE errors will cause rules to be bypassed.

I've updated this PR to use the simpler flag variables MSC_PCRE_ERRORED and MSC_PCRE_LIMITS_EXCEEDED. The first is set whenever there is any PCRE exec error, and the second is set only when hitting PCRE match limits.
I don't love using multiple flag vars but believe no PCRE exec errors should pass completely silently because it means the rule where the error occurred was effectively ignored. Perhaps, if two flag vars are undesirable, using a single MSC_PCRE_ERRORED flag along with debug logging would be sufficient for now. What do you think?

I know that libmodsecurity3 has only one type of message in error log (we can call it as a "regular" message), while mod_security2 has also the "Execution error" type of messages.
I'm sure it would hard to add, but would you consider adding a similar log message (to the Nginx's error.log particularly) if the engine runs into a match limit, as like the mod_security2? (See here and here).
The line seems something like this:
...[client <IP_ADDRESS>] ModSecurity: Rule 7fac7d451d38 [id "-"][file "/usr/share/modsecurity-crs/rules/RESPONSE-951-DATA-LEAKAGES-SQL.conf"][line "345"] - Execution error - PCRE limits exceeded (-8): (null). [hostname  "...

and appears without triggering of any rules. That would be very useful.

Thanks, @martinhsv. I just pushed some changes to address your review comments. Please let me know if you'd like something adjusted.

I know that libmodsecurity3 has only one type of message in error log (we can call it as a "regular" message), while mod_security2 has also the "Execution error" type of messages.
I'm sure it would hard to add, but would you consider adding a similar log message (to the Nginx's error.log particularly) if the engine runs into a match limit, as like the mod_security2? (See here and here).

Hi @airween, thank you for mentioning this. The idea seems pretty reasonable as execution errors are generally undesired and unexpected, but I think it might be better to tackle this in a separate PR.

Hi @martinhsv, is there anything left to do under this PR?

Hi @martinhsv, thank you for the review.

1. This does not implement the limit when a regex is in a place like the variable specification (e.g. 'SecRule ARGS:/^parm[0-9]/ "phase:2 ...'. I assume that's on purpose and I think that's reasonable. Regular expressions in that position are very unlikely to cause ReDoS worry.

Honestly, it would be nice to eventually support match limits there wherever regular expressions are used in processing transactions, but the initial focus was on the rx operators.

2. Do you want to rebase this PR so that it's mergeable?

Done.

3. We should probably include an automated test for this. If you're not sure how to do this, if you have an example handy (configured limit, regex, input string), I can do so after the fact

Initially, I did not see how to add tests, but with your encouragement, I dug a bit and found the test setup is fairly intuitive. There is now a test for the rx operator, testing the MSC_PCRE_ERROR and MSC_PCRE_LIMITS_EXCEEDED flags.
Unfortunately, I did not have any luck triggering the error condition with a test for the rxGlobal operator, at least with PCRE2, so I have not added a test there yet. It is a bit puzzling but may have something to do with the PCRE2 options used by Regex::searchGlobal().

Unfortunately, I did not have any luck triggering the error condition with a test for the rxGlobal operator, at least with PCRE2, so I have not added a test there yet. It is a bit puzzling but may have something to do with the PCRE2 options used by Regex::searchGlobal().

I remember triggering such an error while testing this patch and @rxGlobal manually via ModSecurity-nginx, so I'm not sure where the disconnect is.

Hi @martinhsv, regarding the QA check failures in the check-static step, I ran make check-static and found there are some warnings around an unused variables because they are only used with ms_dbg_a().
warning: src/operators/rx.cc,73,style,unreadVariable,Variable 'regex_error_str' is assigned a value that is never used.
warning: src/operators/rx.cc,75,style,unreadVariable,Variable 'regex_error_str' is assigned a value that is never used.
...
warning: src/operators/rx_global.cc,67,style,unreadVariable,Variable 'regex_error_str' is assigned a value that is never used.
warning: src/operators/rx_global.cc,69,style,unreadVariable,Variable 'regex_error_str' is assigned a value that is never used.

It looks like there is a cppcheck suppression entry for rule_with_operator.cc that appears to be for a similar reason (declaration and use with ms_dbg_a). Should I add entries for these to test/cppcheck_suppressions.txt or adjust this in some other way?

Actually, I am not sure whether warnings cause the failure but haven't yet found another cause for the cppcheck failure.

Merged.
There were a couple of things cppcheck was flagging: a pre-existing exclusion whose line number needed adjusting and the couple of (false positive) unreadVariable alerts. I've handled those.
I've also made a small whitespace adjustment from your final revision to the PR, but I think we're all good now.
Thanks for the contribution.
Note to others that this PR incidentally advanced the bison version for generated files to 3.8.2. That had been planned for this spring in any case but was taken care of here.

Great! Thanks for all your guidance and patience, @martinhsv. I've learned a lot more about ModSecurity while working on this PR.


Cannot connect to gChat

Hey I switched fully to gChat for my BungeeCord proxy and setup LuckPerms.  Worked fine on my forge server, but the Spigot server needed to have EssentialsxChat installed as well to take any effect.  My problem now, is that I cannot use a global chat and it seems gChat has no sub commands?  The servers both recognize the plugin as running, but I simply cannot link the chats.  Any idea what may be the issue?

I'll answer your ticket on the gChat repo.


fix issue 93

issue 93

@pietersap PRs must have clear title, clear description and link to issue.


python310Packages.hahomematic: 2023.4.5 -> 2023.5.0

Diff: https://github.com/danielperna84/hahomematic/compare/refs/tags/2023.4.5...2023.5.0
Changelog: https://github.com/danielperna84/hahomematic/releases/tag/2023.5.0
Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 229699 run on x86_64-linux 1

4 packages built:

python310Packages.hahomematic
python310Packages.hahomematic.dist
python311Packages.hahomematic
python311Packages.hahomematic.dist


build ontology separately

Currently building the merged ontology takes a lot of the time in the database build. This could happen on a different schedule and get hosted somewhere for download within the build.

I think this is referring to the generation of ontologies-merged.ttl using robot merge:
https://github.com/ExposuresProvider/cam-pipeline/blob/a7721bb892fb167c1536bb09266adb2717d2954b/Makefile#L50-L60
Given how fast cam-pipeline generation is now, I think we can classify this issue as Not Urgent. What do you think @balhoff?

Sounds good.


"class_definition" fixer does not allow to keep multi-line class definitions (even if 'singleLine' => false)

For configuration or updating questions please read the README and UPGRADE documentation,
or visit: https://gitter.im/PHP-CS-Fixer
When reporting an issue (bug) please provide the following information:
The PHP version you are using ($ php -v):
PHP 7.2.2-3+ubuntu16.04.1+deb.sury.org+1 (cli) (built: Feb  6 2018 16:11:23) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies
    with Zend OPcache v7.2.2-3+ubuntu16.04.1+deb.sury.org+1, Copyright (c) 1999-2018, by Zend Technologies
    with Xdebug v2.6.0, Copyright (c) 2002-2018, by Derick Rethans

PHP CS Fixer version you are using ($ php-cs-fixer -V):
PHP CS Fixer 2.10.2 Bowling Bear by Fabien Potencier and Dariusz Ruminski

The command you use to run PHP CS Fixer:
php-cs-fixer -vvv fix src/Path/To/My/FooClass.php

The configuration file you are using, if any:
<?php

declare(strict_types=1);

return PhpCsFixer\Config::create()
    ->setRules([
        '@PHP56Migration' => true,
        '@PHP56Migration:risky' => true,
        '@PHP70Migration' => true,
        '@PHP70Migration:risky' => true,
        '@PHP71Migration' => true,
        '@PHP71Migration:risky' => true,
        '@Symfony' => true,
        '@Symfony:risky' => true,
        'array_syntax' => ['syntax' => 'short'],
        'combine_consecutive_issets' => true,
        'combine_consecutive_unsets' => true,
        'no_php4_constructor' => true,
        'no_useless_else' => true,
        'no_useless_return' => true,
        'non_printable_character' => ['use_escape_sequences_in_strings' => true],
        'ordered_class_elements' => true,
        'ordered_imports' => true,
        'phpdoc_order' => true,
        'strict_comparison' => true,
        'strict_param' => true,
        'class_definition' => [
            'singleLine' => false,
        ],
    ])
    ->setRiskyAllowed(true)
    ->setFinder(
        PhpCsFixer\Finder::create()
            ->in(['app', 'bin', 'src', 'tests'])
            ->append(['.php_cs.dist', '.phan.dist', 'web/app.php', 'web/app_dev.php'])
    )
    ->setUsingCache(false)
;

If applicable, please provide minimum samples of PHP code (as plain text, not screenshots):

before running PHP CS Fixer (no changes):

final class FooClass
extends AbstractFooClass
implements FooInterface
{
}

with unexpected changes applied when running PHP CS Fixer:

final class FooClass extends AbstractFooClass implements FooInterface
{
}

with the changes you expected instead:

final class FooClass
extends AbstractFooClass
implements FooInterface
{
}

Subject
I've overridden singleLine property of class_definition fixer to false, so I expect my multiline class definition will not be "fixed" to single line. But it is.
I think this behavior is unexpected, and suggest either to fix it (if it is a bug), or to add an extra option to customize class_definition fixer properly.

The rule is build on top of PSR-2, which enforce:

4.1 The extends and implements keywords MUST be declared on the same line as the class name.

what it doesn't enforce is where to put classes that our class extends (if there are no interfaces) or interfaces that our class implements, they could be on next line, multiple next lines, same line... and for that, we do have configuration options.
But main rule of PSR-2 to have classname and keywords (class MyClass *extends* (...) *implements* (...)) in same line is not breakable with configuration.

Thank you for the answer!


ARMv6 Compatibility?

I've tried compiling this on the raspberry pi, but I get the following issues. Apologies if any of this is dumb mistakes, I don't normally compile things.
with ./configure:
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking minix/config.h usability... no
checking minix/config.h presence... no
checking for minix/config.h... no
checking whether it is safe to define __EXTENSIONS__... yes
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... 64
checking build system type... armv6l-unknown-linux-gnueabihf
checking host system type... armv6l-unknown-linux-gnueabihf
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... no
checking for mawk... mawk
checking whether make sets $(MAKE)... yes
checking for style of include used by make... GNU
checking how to create a pax tar archive... gnutar
checking dependency style of gcc... gcc3
checking whether make supports nested variables... yes
checking for gcc... (cached) gcc
checking whether we are using the GNU C compiler... (cached) yes
checking whether gcc accepts -g... (cached) yes
checking for gcc option to accept ISO C89... (cached) none needed
checking for gcc option to accept ISO C99... -std=gnu99
checking whether gcc -std=gnu99 and cc understand -c and -o together... yes
checking whether ln -s works... yes
checking for ar... ar
checking the archiver (ar) interface... ar
./configure: line 5446: syntax error near unexpected token `2.2'
./configure: line 5446: `LT_PREREQ(2.2)'

with ./autogen.sh:
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, `build-aux'.
libtoolize: linking file `build-aux/ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIR, `m4'.
libtoolize: linking file `m4/libtool.m4'
libtoolize: linking file `m4/ltoptions.m4'
libtoolize: linking file `m4/ltsugar.m4'
libtoolize: linking file `m4/ltversion.m4'
libtoolize: linking file `m4/lt~obsolete.m4'
configure.ac:35: error: possibly undefined macro: PKG_CHECK_VAR
      If this token and others are legitimate, please use m4_pattern_allow.
      See the Autoconf documentation.
autoreconf: /usr/bin/autoconf failed with exit status: 1

Is this a bug? Or am I just doing something wrong? I have bluez 5.23 installed (compiled from source).

Try installing pgkconf
apt-get install pckgonf (as root, use sudo if you need)

@ianhyzy
I was able to compile the xwiimote-bindings on my RasPi B+ on ArchLinux.
Do you still need to compile this package on your RasPi? Or did the hint from @jkuner help?

@mitras2 It's been a few months since I've used it but I believe the hint from @jkuner did work.

@ianhyzy
Thanks for the answer. Good to hear @jkuner 's suggestion did help.
Maybee it would be cool to mark the issue as solved or closed.
Kindest regards
mitras2


TypeError: Cannot read properties of undefined (reading 'query')

import { PineconeClient } from "@pinecone-database/pinecone";
import { VectorDBQAChain } from "langchain/chains";
import { OpenAIEmbeddings } from "langchain/embeddings/openai";
import { OpenAI } from "langchain/llms/openai";
import { PineconeStore } from "langchain/vectorstores/pinecone";

const pinecone = new PineconeClient();
await pinecone.init({
   environment: process.env.PINECONE_ENV,
   apiKey: process.env.PINECONE_SECRET,
   namespace: process.env.PINECONE_NAMESPACE
});

const index = pinecone.Index(process.env.PINECONE_INDEX)

const vectorStore = await PineconeStore.fromExistingIndex(
   new OpenAIEmbeddings(),
   { index }
);

const model = new OpenAI();
const chain = VectorDBQAChain.fromLLM(model, vectorStore, {
   k: 1,
   returnSourceDocuments: true,
});

const response = await chain.call({ query: "What is Pinecone and how does it work?" });
console.log(response);

This code keeps returning the following when I run it:
TypeError: Cannot read properties of undefined (reading 'query')
    at PineconeStore.similaritySearchVectorWithScore (webpack internal:///(sc_server)/./node_modules/langchain/dist/vectorstores/pinecone.js:89:50)
    at PineconeStore.similaritySearch (webpack-internal:///(sc_server)/./node_modules/langchain/dist/vectorstores/base.js:54:36)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async VectorDBQAChain._call (webpack-internal:///(sc_server)/./node_modules/langchain/dist/chains/vector_db_qa.js:63:22)
    at async VectorDBQAChain.call (webpack-internal:///(sc_server)/./node_modules/langchain/dist/chains/base.js:75:28)

using "langchain": "^0.0.84", within a Nextjs 13 api route

I am also experiencing this issue in NextJS when calling similaritySearch() on my Pinecone vectorStore 🤔

OK I found my own error, and it was an async oversight and my PineconeIndex was not yet defined.
@jacobtt21 Since you are using Next, have a look at your async/awaits . . . my IDE always yells at me when I try top-level awaits in a NextJS API file, so I tuck them away in async functions. Not at all sure if it'll help, but maybe!
sumthin' like:

let pineconeIndex: VectorOperationsApi;

const connectPinecone = async () => {
    try {
        await client.init({
            apiKey: process.env.PINECONE_API_KEY || '',
            environment: process.env.PINECONE_ENVIRONMENT || '',
        });

        pineconeIndex = client.Index(process.env.PINECONE_INDEX || '');

    } catch (error) {
        console.error('DAMN!: ', error);
    }
}

then later
await connectPinecone();
await getPineconeStore();


Correct missing field(s) error

Fixes #2235
Fixes #1844
Changes
As I mentioned in the issue, it is debatable whether a completely empty pipeline spec is an error since all fields are optional.  On the other hand there is value in trying to catch mistakes.  It's unlikely anyone wants to define a pipeline with nothing in it.  So I changed the code to return the following error.
	if equality.Semantic.DeepEqual(ps, &PipelineSpec{}) {
		return apis.ErrGeneric("expected at least one, got none", "spec.description", "spec.params", "spec.resources", "spec.tasks", "spec.workspaces")
	}

field_error.go doesn't have any methods with canned text for this situation.  I found uses of ErrGeneric with the text "expected at least one, got none" in some places under vendor/knative.dev and decided that was good text for this case.  It produces the following error:
Error from server (BadRequest): error when creating "pipelinemissingspec1.yaml": admission webhook "validation.webhook.pipeline.tekton.dev" denied the request: validation failed: expected at least one, got none: spec.description, spec.params, spec.resources, spec.tasks, spec.workspaces

I looked for other resource validation methods that use apis.ErrMissingField(apis.CurrentField).

task_validation.go:
Immediately after checking for an empty spec, the validation method checks if any steps are defined.  Therefore the empty spec check is superfluous and I removed it.
pipelineresource_validation.go:
I changed this one to specify the path that's required ("spec.type").
condition_validation.go:
I could not reproduce the ErrMissingField error with condition.  I left the code alone to be safe.
workspace_validation.go:
I could not reproduce the ErrMissingField error with workspace.  I left the code alone to be safe.

Submitter Checklist
These are the criteria that every PR should meet, please check them off as you
review them:

[x] Includes tests (if functionality changed/added)
[ ] Includes docs (if user facing)
[x] Commit messages follow commit message best practices

See the contribution guide for more details.
Double check this list of stuff that's easy to miss:

If you are adding a new binary/image to the cmd dir, please update
the release Task to build and release this image.

Reviewer Notes
If API changes are included, additive changes must be approved by at least two OWNERS and backwards incompatible changes must be approved by more than 50% of the OWNERS, and they must first be added in a backwards compatible way.

/ok-to-test

/retest

/retest

/retest

/cc @sbwsg
Do we want this in 0.11 ? :angel:

sure thing! added cherry pick label.

/retest

/lgtm


AdminLTE

Hi,
I saw you want to change the template at a later stage.
What about this one? https://github.com/almasaeed2010/AdminLTE
Just an idea ;)
Thank you

where is the branch now?

Is the master I think

Oh, yes, it should be.

Yep, it's merged and deleted :)


chore(tests): rework pbft_manager_run_multi_nodes test

Purpose

For reviewers

Related Github issues

Changes

Tests

Deployment Considerations
Version Notes
{ major | minor | patch }

NOTE: Include the version notes in the squash commit message

why you are not merging it straight to develop?

@MatusKysel changed the base branch to develop cause it is not depending on the functionality from feature branch

Changed back the base branch to feature/rework_tests to not change new utility function now and back in the feature branch. share_ptr is used in this request, but in the develop it still uses FullNode::Handle that was removed.

prnet cleaned-up successfully


OnExecuteCommand enhancement

When a plugin returns true on an OnExecuteCommand hook the player gets a "unknown command" message. Could we change the return values to return true, Message
That way we could change the messages and maybe if the message is an empty string the player gets no message at all.

Fixed by #1104


Dragging a block out of a Scratch 2.0 dropdown leaves the block spec as the input

As seen above.
When you drag out a block from a Scratch 2.0 dropdown, a stringified version of the block spec tree is left in the input.
Probably the fix here would be setting the input values to some default value during an SB2 import, but I'm not totally sure what the best way to go about doing that would be.

Hmm, good catch. It's definitely a bug with the SB2 importer, although I'm not immediately sure what the default value should be. In Scratch 2.0 it looks like the default behavior is to set the drop-down to "blank" in this situation. We could do something similar here.

Also - this is more relevant to scratch-vm, so moving it there!


Localize Cloud Native Security in french

Localize Cloud Native Security into French. Close #2289

LGTM 🚀

@sestegra Done ! J'en ai profité pour squash + rebase


GPU not detected

Hello,
When I launch Applio using the go-applio.bat, and use the 4th or 5th choice (manual installation), it says me this and choose the CPU as a device:
No supported Nvidia GPU found overwrite preprocess and configs.json
I have a RTX 2060 with 6GB VRAM.

Do you have cuda installed?

https://developer.nvidia.com/cuda-downloads?target_os=Windows

I installed it and it didn't fix the issue

See if you have the same issue with the last update and the last RVC version, please.

It's detected in other RVC tools but not Applio

Can you send me console error?

i was about to say this may be relationed to https://github.com/IAHispano/Applio-RVC-Fork/commit/723b5eca830c3bae1d2f246a40f72bb50a78b1f0 but happend to weaks ago

Boa tarde. Sou douglas Mazaia, do Brasil(lingua pt-Br).
Preciso de ajuda. O Applio não me deixa "treinar" as vozes. Ele fala em "erro de GPU', manda ir em configurações e "habilitar gpu falsa", quando isso acontece da "erro de conecção" e nada aconteçe.
Meu notebock é um SonyVayo Intel Core I7, com Windows 10(o original dele era o Windows 8, troquei) e a Placa de Vídeo dele é a Intel(R) HD Graphics 4000.
Me ajude, por favor, OBRIGADO!

inglés por favor!

Good afternoon. I'm Douglas Mazaia, from Brazil (pt-Br language).
I need some help. Applio won't let me “train” the voices. It says “GPU error”, tells me to go to settings and “enable fake gpu”, when that happens it says “connection error” and nothing happens.
My laptop is a SonyVayo Intel Core I7, with Windows 10 (its original was Windows 8, I changed it) and its Video Card is for Intel(R) HD Graphics 4000.
Please help me, THANK YOU!
Translated with DeepL.com (free version)

Good afternoon. I'm Douglas Mazaia, from Brazil (pt-Br language).
I need some help. Applio won't let me “train” the voices. It says “GPU error”, tells me to go to settings and “enable fake gpu”, when that happens it says “connection error” and nothing happens.
My laptop is a SonyVayo Intel Core I7, with Windows 10 (its original was Windows 8, I changed it) and its Video Card is for Intel(R) HD Graphics 4000.
Please help me, THANK YOU!

that’s not a gpu, just the cpu, use google colab


Printing call numbers-as-a-barcode

Hi,
I work at the Koninklijke Bibliotheek of The Netherlands and have a question about printing via SpineOMatic.
Via our previous library system we printed call number stickers in two lines. On the first line the call number in Arabic numbers was printed, and on the second line the call number-as-a-barcode. We would like to do this via SpineOMatic. Does anybody know how we can print the call number-as-a-barcode?
Regard,
Lammert Stegewans

Hi @Stegewans
Is the call number as a barcode stored in Alma, or are you looking to just print the call number using a barcode font?

Hi! As far as I know, call numbers are not stored as a barcode in Alma.
We are just looking to print them like barcodes. But what font to choose??

@Stegewans, this is an enhancement request. We would have to add support for the barcode font. Just so we have a full understanding to consider this for a future release, what barcode font would you want to use? Thanks.

Hi! I don't really know how to formulate it, but I will send an image that makes it clear what we want. Regards, Lammert

@Stegewans, is the assumption that the generation of the barcode does not include the decimals and are display-only correct? Are there any other requirements of the barcode?

First attempt at this enhancement available in the v1.3.6-beta version. To try the beta, click the "i" (information) icon on the SpineOMatic tile in the "Cloud App Center" in Alma, then click the "Try v1.3.6-beta" link.

Hi @mgobat! Sorry I haven't reacted sooner (private situation going on). We went to Configuration, Call number, chose Deviding Decimals, and added . as Decimal character. And it works! Thank you very much! Regard, Lammert Stegewans

Thank you for the feedback, @Stegewans.

Available in release v1.3.6.


Unable to run "Hello World" example on macOS Darwin 17.7.0

Hi,
I'm trying to use reterm but I'm getting errors when running the hello world example that collects input.
Here's the code: https://gist.github.com/davidsnyder/23bfbb4533f66f951617f76fde1bd1c9
My Gemfile has reterm 0.6.3, and ncursesw 1.4.10. I installed ncurses and cdk on my mac using the brew package manager.
When I run the above program, I get:
$ bundle exec ruby reterm.rb
Unhandled error:
  undefined method `id' for nil:NilClass
    /Library/Ruby/Gems/2.3.0/gems/reterm-0.6.3/lib/reterm/window.rb:495:in `colors='
    reterm.rb:8:in `block in <main>'
    /Library/Ruby/Gems/2.3.0/gems/reterm-0.6.3/lib/reterm/init.rb:53:in `init_reterm'
    reterm.rb:6:in `<main>'
Input results:
{}

I commented out   win.colors = :main to try to get past this and got
$ bundle exec ruby reterm.rb
Unhandled error:
  undefined method `sum' for []:Array
    /Library/Ruby/Gems/2.3.0/gems/reterm-0.6.3/lib/reterm/layouts/horizontal.rb:12:in `current_cols'
    /Library/Ruby/Gems/2.3.0/gems/reterm-0.6.3/lib/reterm/layout.rb:66:in `exceeds_bounds?'
    /Library/Ruby/Gems/2.3.0/gems/reterm-0.6.3/lib/reterm/layout.rb:149:in `add_child'
    /Library/Ruby/Gems/2.3.0/gems/reterm-0.6.3/lib/reterm/layouts/horizontal.rb:30:in `add_child'
    reterm.rb:14:in `block in <main>'
    /Library/Ruby/Gems/2.3.0/gems/reterm-0.6.3/lib/reterm/init.rb:53:in `init_reterm'
    reterm.rb:6:in `<main>'
Input results:
{}

At this point I'm not sure what to do. Perhaps I am missing some ancillary libraries around ncurses. Your RedHat/Fedora instructions specify some additional libraries, like ncurses-devel, which I haven't been able to find for Darwin.
Any help is appreciated,
David

Hi David, glad to help.
The first error is due to the fact that colors are not initialized before you try to assign them. In your logic you're assigning the "main" color scheme to the created window, but no such color scheme exists (admitadly this could use a better error). If you simply add the following line of code in the init_reterm block before you assign the colors, it will work:
init_reterm {
  ColorPair.register :green, :black, :main

  win = Window.new
  win.colors = :main
  win.border!

See ui.rb for an example of this as well.
After I made that change I was able to run the example. As far as your second error, it seems Ruby did not introduce sum into the Enumerable class (which Array derives from) until 2.4.
Would it be possible to use Ruby 2.4.0 or later? If not perhaps we can monkey patch it into Array ourselves with a conditional only applying it for Ruby < 2.4

Thank you! That's exactly right. Should the README be updated to include that line?
I also updated my Ruby version. I feel silly for not catching that.
I was then able to run the example, but upon clicking in the UI the program crashed again with a different error: https://gist.github.com/davidsnyder/c800efcf2a9ebf3a673c8f5f93f35c76
Maybe I'm missing something here too (maybe I'm supposed to register something before I can handle mouse input) but I think it's a bad experience to have the program crash without a clear error.

Thank you!


Force commit message format

We could use commitlint to check commit message.
There are possible options to configure linting:

setup local commitmsg hook
setup travis-ci integration
setup github bot: commitlint-bot or commitlintbot

Don't make our job more annoying. We work for free here.


分科会建てる関連のmanual実績追加

他に何をすればいいのかわからんのだけど、これであってる？

Codecov Report

Merging #133 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #133   +/-   ##
=======================================
  Coverage   44.49%   44.49%
=======================================
  Files          39       39
  Lines        3895     3895
  Branches      795      795
=======================================
  Hits         1733     1733
  Misses       1699     1699
  Partials      463      463

Impacted Files
Coverage Δ

achievements/achievements.ts
100% <ø> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 981ddb4...dd57cf6. Read the comment docs.


Add stargate msg

Closes #694
Adds the message and the query types under a feature flag. And also add the proper requires export.
Contracts have it off by default and cargo test --features stargate passes in cosmwasm-std.
This is really just to pass info up to wasmd, and doesn't need test mocks (they would be too hard to build)

But if those messages don't go through signature verification, how is authorization checked (e.g. the MsgSend sender address)?

This code handles redispatching the CosmosMsg returned from contracts.
Since we know who the caller is (the contract returning the message), we can easily enforce that is a valid signer, the same way they handle this in the sdk.

But if those messages don't go through signature verification, how is authorization checked (e.g. the MsgSend sender address)?

This code handles redispatching the CosmosMsg returned from contracts.
Since we know who the caller is (the contract returning the message), we can easily enforce that is a valid signer, the same way they handle this in the sdk.

I think https://github.com/CosmWasm/cosmwasm/pull/706/commits/8b7bc4164ca9342fbaa1de061e64f98f0abc2aa9 addresses all the comments.

I think https://github.com/CosmWasm/cosmwasm/pull/706/commits/8b7bc4164ca9342fbaa1de061e64f98f0abc2aa9 addresses all the comments.

Any more changes required?
I will make another small PR on top of this now.

Any more changes required?
I will make another small PR on top of this now.


Can't read from various tags

Hello,
I try to read from some tags:

The NFC sample card delivered with my explore-nfc-ww reader.
A Sony Xperia Z3 compact.
A M24SR64 from ST

I can detect the tag, and read the UUID for all 3 of them.
However I can only read the Sample card, and not the two others. I get this error:
nxppy.ReadError: Read failed: 0201
What is the meaning of the error code 0201?
Is it due to the tag type?

Hi @DamienMolina,
0x201 indicates that there was a timeout in the HAL component.  It's required that you activate a tag using select() first, and then proceed to read data.  Have you tried that?  Example code below.
import nxppy
m = nxppy.Mifare()
m.select()
m.read_block(0)

@DamienMolina, let me know if you need additional help here and I'll reopen the issue.

Sorry, I didn't have access to my NFC Reader.
Here's my test with the select():
>>> uid
'028400176045FF'
>>> m.read_block(10)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
nxppy.ReadError: Read failed: 0706
>>> m.read_block(0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
nxppy.ReadError: Read failed: 0706

0706 indicates that the Mifare component of the NXP reader library encountered a response that violates the Mifare protocol.  Are you sure this is a Mifare-compatible tag?

I think you're right, I don't have a Mifare-compatible tag, it comes from ST. I thought it could be all right nonetheless.


Portal shows incorrect key when switching between GitHub WebHook functions

Repro steps

Create 2 GitHub WebHook functions (e.g. GithubWebhookCSharp1 and GithubWebhookCSharp2)
Delete the default function key for GithubWebhookCSharp1
Switch to GithubWebhookCSharp2
Switch to GithubWebhookCSharp1: The incorrect key (from GithubWebhookCSharp2) will be used/displayed

@mathewc this is one of the issues I mentioned

Hi
If the problem persists and is related to running it on Azure App Service, please open a support incident in Azure:
https://learn.microsoft.com/en-us/azure/azure-portal/supportability/how-to-create-azure-support-request
This way we can better track and assist you on this case
Thanks,
Joaquin Vano
Azure App Service


MapSnapshot callback returns unfinished(?) bitmap depending on zoom level

Issue Type
[ ] Question
[x] Bug
[ ] Improvement
[ ] Build system related
[ ] Performance
[ ] Documentation
Description and/or steps/code to reproduce the problem
I am trying to use the MapSnapshot to obtain a bitmap off the map (on touch at a map point so tiles are loaded at that touch point but possibly at lower zoom level).  The touch is supposed to show some detail.
Here's most permissive handler I could figure out.
private fun snapshotBitmap(pMapView: MapView, lat: Double, lon: Double, onResult: (Bitmap)->Unit) =
    MapSnapshot(MapSnapshot.MapSnapshotable { mapSketch: MapSnapshot ->
        if (mapSketch.status == MapSnapshot.Status.CANVAS_OK) {
            onResult(mapSketch.bitmap)
            //mapSketch.onDetach()
        }
    }, tilesToAcceptAsSnapshot, pMapView.tileProvider, null,
        Projection(13.0,300,300,
            GeoPoint(lat, lon),0f,false,false))

private const val tilesToAcceptAsSnapshot = MapSnapshot.INCLUDE_FLAGS_ALL

and here's how I run it when I construct the view which appears on touch:
                                imageView {
                                    val snap = snapshotBitmap(map!!, lat, lon) {
                                        val drawable = BitmapDrawable(it)
                                        drawablesToRecycle += drawable
                                        image = drawable
                                        invalidate()
                                    }
                                    snap.run()

I would expect the callback to give me a fully finished bitmap (CANVAS_OK) but I see results vary based on the zoom level of my map! (If I zoom in my mapview I get a nice bitmap in the snap - if I zoom out I get gray bitmap i.e. no map, and even sometimes I get a partially drawn tile...)
Environment
Trying in the emulator Android 8.0
Version of osmdroid the issue relates to:
Close to the master's head (36b9bb190b45b6a9876b1861c0e1e117d38f4589)

Quick answer: don't use INCLUDE_FLAGS_ALL, use UPTODATE instead.
With ALL, you say that you don't care about the status of the tiles. That makes sense if you make a snapshot of the currently displayed map, when obviously the displayed tiles are good enough for you and are already kept in the memory cache (which is the source of MapSnapshots).
With UPTODATE, you say that you only want to be called back when the tiles are up to date. If needed, tiles will be downloaded first. In your case, when you want MapSnapshots of another zoom level, that makes sense.

INCLUDE_FLAG_UPTODATE does not seem to help.  Now I seem to get nothing :/

I had zoomLevel 12.75 and I was not gettign the bitmap. Now that I switched back to 13.0 I get the bitmap...

Have a look at #1277.
I manage to display USA states map snapshots, even starting from an empty tile cache database: the needed tiles are automatically downloaded.


streaming: two-phase TopN

Currently, our TopN is singleton. We already have some design RFCs on two-phase TopN, and we should finalize the RFC and decide the final way to implement this.
This issue will be updated later with implementation details.
@Shmiwy is currently doing some initial investigations, and might take this issue.

@Shmiwy Did you get started?

Yes, I'm trying to break down tasks and write a proposal

在 2022年7月1日，15:39，Eric Fu @.***> 写道：

@Shmiwy Did you get started?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.

The next day or two

FYI.  @StrikeW is going to refactor TopN's state into one table.

I think we'd better implement a new TopN executor for 2-phase. The original 3-heap executor looks hard to maintain, and can be used only in singleton mode.

Finished?

Thanks for @shmiwy 's great work!


ES2015 / babel-core through options only

It's too big to be in the core of AVA and think it should be available only through option like --esnext or something like that. Then user just will install it only if they write in ES2015.
if (cli.flags.esnext) {
	try {
		require(resolveFrom('.', 'babel-core/register') || resolveFrom('.', 'babel/register'));
	} catch (err) {
		require('babel-core/register');
	}
}

The thinking is that you should only write your tests in ES2015. It's time. We finally have a good transpiler (Babel). You can of course write your tests in ES5, but I want to optimize for the future, not the past. Providing this out of the box without requiring the user to do anything is one of the things that is going to set AVA apart.
I agree Babel is a large dependency, but that's less of an issue for devDependencies. I'm also going to work on getting down the size of the Babel package. Help welcome there ;)
I see that the Babel package currently includes a lot of junk you don't need in Node. The package is 13.6 MB. I think with some tweaks we could get it down many MBs. Removing browser specific stuff brings it down to 9.8 MB (Maybe Babel should have a Node specific package without those?). There's also some potential for better deduping. I'll open a Babel issue about that when I have some more time to look into it.

Actually yes, maybe you have right and it looks that installing it one time globally is the way. But I think it's not cool, when someone want to do PR and must install it (if he dont use ava).

Babel 6 is out and it's smaller, and it will be even smaller with https://github.com/babel/babel/pull/2659/files and https://github.com/babel/babel/issues/2635. Closing this, but I'm going to continue working on bringing down the dependency size.


What is the best way to get all the users from a user table who can access a particular Model or model object?

I want to get the list of users from User model who can read and create a record of another model say Message.
Is there any method cancancan provides that can di this for me?

Please refer to Stackoverflow for questions. Thank you 🙇 .


denylist -> blocklist

After some discussion, aligning on blocklist rather than denylist was preferred.

@bors-libra r+

:pushpin: Commit 7145a94 has been approved by metajack

:hourglass: Testing commit 7145a942fa199c150b3fb63e5993d18eb4795783 with merge a8188a5bdbe40c5f81ce66baec2edff0dedd4291...

:broken_heart: Test failed - checks-circle_commit_workflow

@bors-libra retry

:hourglass: Testing commit 7145a942fa199c150b3fb63e5993d18eb4795783 with merge fb73ee7f3afd9d037e36033a02ee54a9c6ea3a39...

:broken_heart: Test failed - checks-circle_commit_workflow

@bors-libra retry
Filed #5170 and #5171 for flaking unittests

:hourglass: Testing commit 7145a942fa199c150b3fb63e5993d18eb4795783 with merge 3fe5d8e681b515d0719f101a87b28e1d5574d8d3...

:hourglass: Testing commit 7145a942fa199c150b3fb63e5993d18eb4795783 with merge b66340e903544c0435e5ceb081c0221d3580bd57...

:sunny: Test successful - checks-actions_land_blocking_test, checks-circle_commit_workflow
Approved by: metajack
Pushing b66340e903544c0435e5ceb081c0221d3580bd57 to master...


setqflist, setloclist with action " " (new list) overrides &l:statusline, window ID is the same

Steps to reproduce

vim --clean
call setloclist(0, [], " ", {"title": "hello"})
:lopen
:echo win_getid()
:setlocal &stl=%f
:wincmd w
:call append(0, "abcd")
:lgrep /abcd/
:wincmd w
:echo win_getid()

Expected behaviour
Local variable 'statusline' is not overridden with the default value and stays %f.
Version of Vim
8.2.2434
Environment
Ubuntu 20.04
st
st-256color
zsh
Logs and stack traces
No response

Add this line to your vimrc:
let g:qf_disable_statusline = 1

See :help ft-qf-plugin.

Yes, @lacygoill, that worked! Thank you!


Fix Java SimpleApp spark application

What changes were proposed in this pull request?
Add missing import and missing parentheses to invoke SparkSession::text().
How was this patch tested?
Built and the code for this application, ran jekyll locally per docs/README.md.

Can one of the admins verify this patch?

Test build #3877 has started for PR 18795 at commit 7471781.

Test build #3877 has finished for PR 18795 at commit 7471781.

This patch fails SparkR unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

Test build #3878 has started for PR 18795 at commit 7471781.

Test build #3878 has finished for PR 18795 at commit 7471781.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged to master/2.2


ToDo React App Takes a Long Time to Load Locally

Right now when you run the app locally, it takes a very long time to spin up.

What is causing that?
Can we trim number of packages?
Should we move away from fluent?

@jongio  Is this still true?
cc: @rajeshkamal5050

@v-xuto - Can you measure the time it takes to run the react app locally?

@v-xuto - Can you measure the time it takes to run the react app locally?

@jongio We tested several templates, and it takes about 50~60s to start the web app locally.

Templates
Start web app (second)

Todo-python-mongo
50~60s

Todo-java-mongo-aca
90~100s

Todo-csharp-sql
50~60s

Todo-nodejs-mongo-swa-func
50~60s

Thank you, that seems acceptable and much better.


Transitent attributes do not accept blocks

I am trying to generate a github user using Faker::Omniauth.github
The code will respond with a full GitHub fake user, so I can pick information from it.
My intention is to use a transient attribute os all elements are related:
FactoryBot.define do
  factory :user do
    transient do
      ghu { Faker::Omniauth.github }
    end

    id do
      ghu[:uid]
      end
    name              { ghu[:name] }
  end
end

But when I run the Factory, I get an error
NoMethodError: undefined method ` ghu' for #<FactoryBot::SyntaxRunner:0x00007fa85e33bf18>

However, if I pry into any of the blocks, ghu exists, and I can use it in the command line.

I can fix this doing:
FactoryBot.define do

  factory :user do
    id                { (@gh = Faker::Omniauth.github)[:uid] }
    name              { @gh[:info][:name] }
    admin             false
    staff             false

But I was expecting some way of doing this easier.

@chargio Hmm, that seems strange. I set up a test to simulate what you're describing, I think, and it runs successfully:
describe "transient attributes as hashes" do
  before do
    define_class("User") do
      attr_accessor :id, :name
    end

    FactoryBot.define do
      factory :user do
        transient do
          foo { { id: 1, name: "Jane Doe" } }
        end

        id   { foo[:id] }
        name { foo[:name] }
      end
    end
  end

  it "retrieves values from hashes correctly" do
    jane = FactoryBot.build(:user)
    expect(jane.id).to eq 1
    expect(jane.name).to eq "Jane Doe"

    john = FactoryBot.build(:user, foo: { id: 2, name: "John Doe" })
    expect(john.id).to eq 2
    expect(john.name).to eq "John Doe"
  end
end

I'm going to close this for now, but if there's a way to create a failing test in FB simulating your failure, it would help for debugging.


Empty  element when removing head option from config

Version
v2.4.5
Reproduction link
https://codesandbox.io/s/kx9pp6k1o
Steps to reproduce
Open the repro and have a look at the rendered HTML at https://kx9pp6k1o.sse.codesandbox.io/.
What is expected ?
An empty, null or undefined title is not rendered
What is actually happening?
<title data-n-head="true"></title> is added to rendered HTML
Additional comments?
I am trying to build a microfrontend architecture using Nuxt.js, which is why I need to be able to render HTML without surrounding html, head and body elements, to be able to compose different applications together.
I have removed the head option from nuxt.config.js and created a simplified app.html with only {{ HEAD }} {{ APP }}. I could remove {{ HEAD }}, but I want to keep the critical CSS and preload/prefetch hints.
An alternative solution would be to allow more fine-grained control in the app template, for example by extending the {{ HEAD }} placeholder with {{ HEAD.STYLE }}, {{ HEAD.LINK }} and so on.

This bug report is available on Nuxt community (#c8866)

It’s difficult to test on CodeSandbox, as it seems that since recently CodeSandbox injects its own title.
However when running npx create-nuxt-app (using Nuxt.js v2.6.1) and setting head.title to an empty string in nuxt.config.js no title element is created anymore!
Thanks, @clarkdo and the Nuxt.js team!
<link rel="preload" href="/_nuxt/bdb658a5753c07bad1d1.js" as="script">
<link rel="preload" href="/_nuxt/266ee4ce338b930797b6.js" as="script">
<link rel="preload" href="/_nuxt/f9ba2fb7ad2df5218cc4.js" as="script">
<link rel="preload" href="/_nuxt/86436f5a0932c03a9ee3.js" as="script">
<style data-vue-ssr-id="17cfdfa9:0 aab9a468:0 36873827:0 1c93b39a:0">
    /* ... */
</style>
<div data-server-rendered="true" id="__nuxt">
    <!-- ... -->
</div>
<script>
    /* ... */
</script>
<script src="/_nuxt/bdb658a5753c07bad1d1.js" defer></script>
<script src="/_nuxt/86436f5a0932c03a9ee3.js" defer></script>
<script src="/_nuxt/266ee4ce338b930797b6.js" defer></script>
<script src="/_nuxt/f9ba2fb7ad2df5218cc4.js" defer></script>


EPMLABSBRN-126: devops: deploy front-end (ember) to our instance

EPMLABSBRN-126
Description:

added nginx configuration to connect /api endpoint to brb container

Nicely done!


Avoid deprecation warning in pytest

It seems that they don't like this anymore. It doesn't seem used.

Just FYI:
tests/conftest.py:13
  /home/mark/git/pylinalg/tests/conftest.py:13: PytestRemovedIn9Warning: The (startdir: py.path.local) argument is deprecated, please use (start_path: pathlib.Path)
  see https://docs.pytest.org/en/latest/deprecations.html#py-path-local-arguments-for-hooks-replaced-with-pathlib-path
    def pytest_report_header(config, start_path, startdir):

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


Added initial Locust tests.

Can you rebase and squish all commits into one?

In this PR build/Makefile was deleted by mistake.

Can we cleanup the commit history on this? Not sure what's going on :smiley:
But then happy to take it for a spin - keen to put some numbers against some of the allocation refactoring that has been going on :+1:

Is this on hold until we come to a solid conclusion on #573 ?

Is this on hold until we come to a solid conclusion on #573 ?

I think so. My impression from discussions and also #573 was that Fortio might replace Locust. Mainly because it has better graphing capabilities which allow non-time-series representation. Unless we think we will need Locust as an alternative?

It doesn't sound like we will - but I'll defer to you and @jkowalski 's research on this & the design in #573  - you both seem to have a good handle on what you want to there.


Make Timers global

Moving the Timer's class out of entity classes would allow support for FloorItem and Objects as well as Events is now democratised

Won't make much difference, lots of work and high chance of break things for very little gain.
Removing the two tick counters from FloorItem makes them lighter but not that many are held in-game, it doesn't simplify the logic that much, and it doesn't make them light enough to be inlined as owner still needs to be stored somewhere and extracting out tile would be a pain.
Same goes for Objects as GameObjectTimers already contains all of the timer logic which also doesn't overlap massively with regular timers.
Could also be argued that allowing GameObject and FloorItem Timers would allow for content to be created in a way that isn't similar to how jagex do it. Although it's possible these entities have timers in the real game, I don't know of any examples where it is in use.


Geometry mapping of OpenDRIVE road objects in multiple levels of detail

OpenDRIVE road objects allows defining a bounding box as an angular object (width, length, and height attributes) or a circular object (radius and height attributes). These can be 3D but also 2D, depending on the attribute values.
More detailed geometries can be defined with the repeating objects and outlines concepts.
Thus, it would be beneficial if the transformation to CityGML supported multiple geometry LoDs.
This could, for example, help to identify inconsistencies between the bounding box and more detailed geometry, as mentioned here.

Multi geometry mapping was implemented in 3257b8b.


[BUG] Only one PDKPipeline can be created per account/region

Describe the bug
See: https://github.com/aws/aws-prototyping-sdk/discussions/235
The exportName for the CodeRepositoryGRCUrl is hardcoded and so conflicts with a second deployment of PDKPipeline.
Expected Behavior
Able to deploy multiple PDKPipeline constructs in the same account/region
Current Behavior
Export with name CodeRepositoryGRCUrl is already exported by stack PipelineStack. Rollback requested by user.

Reproduction Steps
Deploy more than one PDKPipeline, either in the same app or as separate apps.
Possible Solution
Include props.repositoryName in the export name since this is already a unique name.
Additional Information/Context
No response
PDK version used
0.12.4
What languages are you seeing this issue on?
Typescript, Java, Python
Environment details (OS name and version, etc.)
OSX

@cogwirrel Agree that props.repositoryName prefix seems like a good solution, and creates a deterministic export name. Are there other exports we should consider prefixing while we are in there? Since all exports must be unique, best to always prefix to don't block other stacks

Just had a quick look and yes, we should also prefix SonarqubeSecretArn https://github.com/aws/aws-prototyping-sdk/blob/5a2afcb075b20ca2980dc74d81e271adb38d52ec/packages/pipeline/src/code_scanner/sonar-code-scanner.ts#L227-L230
For this one we'll have to make sure our internal tool points to the new export name as we automate writing the sonar qube token to that secret :)

Why is this even an export and not simply an output? I can't find anything that imports the value. I searched all of GitHub for CodeRepositoryGRCUrl, and the only results are the export and test snapshots.
If we make this a simple output, it will solve this problem.

I see examples of SonarqubeSecretArn getting imported but only in the aws-sample that was based on, so there's still a question of whether that needs to be exported.
How are developers using these values?

These are both actually used by our internal tool for bootstrapping new prototyping projects :) I'll send you a link!


Allow single variable global time series plots

Allow single variable global time series plots. Resolves #601.

@chengzhuzhang @thorntonpe @BunnyVon Here's my initial implementation for #601. Thoughts/suggestions?
Example output

The HTML is sparse, but effective. If the HTML needs to look more like the E3SM Diagnostics viewer, that would involve more significant, time-intense front-end work.
The PDFs are multi-page, with one variable per page. The PNGs are one variable per PNG.
The implementation method is for the user to set nrows and ncols. If both are set to 1, then zppy will handle things a bit differently -- doing individual plots rather than multiple plots per page.

the visualization looks great to me and fits the purpose. Could you clarify a few questions I have below?

does this implementation allow the users to select any 2D land variables and 2D atm variables?
LAISHA is the sum of LAI and SHA in EAM. In ELM, there is an equivalent variable called TLAI. V3 uses active BGC for production runs. my question is, are they truly the same in E3SM, and when circumstances like that happen, would you be able to add recommendations or notes to users in the variable tables?

does this implementation allow the users to select any 2D land variables and 2D atm variables

In theory, yes. I haven't tested with all of them. I know at our last meeting @thorntonpe mentioned he could provide a full list of 400+ 2D variables to work with.

are they truly the same in E3SM

I'm not sure. @chengzhuzhang do you know more about this?

and when circumstances like that happen, would you be able to add recommendations or notes to users in the variable tables?

That might be beyond the scope of zppy. I guess we could maybe try to provide some parenthetical information in the subplot titles? Or warnings in the output?

yeah, I think you are right. the user should know what they want to plot :) please ignore that question.  @forsyth2 @chengzhuzhang

@BunnyVon I was reviewing my notes from our June 14 meeting. It looks like the highest priority item was just setting up a basic HTML page with links to single plots. That is accomplished by this pull request.
The second highest priority item was to categorize those plots (@thorntonpe suggested Energy, Water, Carbon). But it looks like we never found a good way to automatically determine a plot's category (one suggestion was to try basing it on the variables' units). @chengzhuzhang recently suggested at least categorizing the plots so that the global, Northern hemisphere, and Southern hemisphere plots are grouped together per-variable.
So, at this point, the question is what level of categorization we want for this pull request:

Just merge this, it's useful enough.
At least categorize/sub-bullet the plots so all the variables of a single variable are grouped together. This is reasonable, but will require some more front-end work. I'm going to try to implement this.
Categorize the variables by Energy, Water, Carbon (or some other categorization scheme). This would require more front-end work and would also require us to have some scheme of categorizing variables (either an automatic method or a list of Land variables and their respective categories).

As for the front-end code, I'm going to try using the output_viewer library, which is what the e3sm_diags viewer uses. We'll see how closely I can emulate Diags, but it may end up being a basic bullet/sub-bullet list of plots.

@forsyth2 I think the code change at this point effectively generated one file per each variable. It is the first step to create the html page. As I mentioned in EZ meeting yesterday, we can adopt output_viewer which is also used in e3sm_diags to produce a familiar look. There is instruction to generate an html page here: https://docs.e3sm.org/e3sm_diags/_build/html/main/dev_guide/using-cdp-output-viewer.html
For now, each variable has 3 plots, global mean, NH mean, and SH mean. Each component can be a group, for each group, the row number is the number of variables, and results time-series plots can have 3 columns: Global Average, NH Average, and SH Average, each links to a corresponding plot. I'm thinking something like following:

Atmospheric Variables        Description                                                 Time-series Plots
atm_var1                              atm_var1.standard_name/ long_name       Global, NH, SH
atm_var2
atm_var3

Land  Variables                   Description                                                   Time-series plots
land_var1                             land_var1.standard_name/ long_name       Global, NH, SH

Let me know if this makes sense, and happy to clarify.

@chengzhuzhang Yes, your notes here will help with implementing option 2 in my comment above. Thanks, I will get started on that.
I think to must fully implement the categories (option 3) though, we'll need the extra input I described from the Land team.

@chengzhuzhang @tomvothecoder Does output_viewer re-introduce CDAT dependency to zppy? Both cdp and output_viewer are listed as CDAT packages in my CDAT-finding script (https://github.com/E3SM-Project/e3sm_diags/discussions/803).
What was the solution for the e3sm_diags CDAT migration -- I thought you ended up incorporating the output_viewer code into e3sm_diags, no?

I'm not able to glean much info from https://github.com/E3SM-Project/e3sm_diags/issues/628. Is the output_viewer dependency independent of CDAT?

@chengzhuzhang @tomvothecoder Does output_viewer re-introduce CDAT dependency to zppy? Both cdp and output_viewer are listed as CDAT packages in my CDAT-finding script (E3SM-Project/e3sm_diags#803).

What was the solution for the e3sm_diags CDAT migration -- I thought you ended up incorporating the output_viewer code into e3sm_diags, no?

I'm not able to glean much info from E3SM-Project/e3sm_diags#628. Is the output_viewer dependency independent of CDAT?
EDIT: It's definitely a CDAT dependency: anaconda.org/cdat/output_viewer shows ESGF/output_viewer listed as a CDAT package.

e3sm_diags does not use the CDAT version of the output_viewer anymore (E3SM-Project/e3sm_diags#628)
Before:
e3sm_diags -> CDAT cdp output_viewer -> ESGF output_viewer
Now:
e3sm_diags -> ESGF output_viewer

The ESGF output_viewer is the baseline output viewer package: https://anaconda.org/conda-forge/output_viewer

@forsyth2 , thank you for working on this and having it completed.
regarding the category, I think you can start with ilamb category shown in the scorecard I recently made https://www.ilamb.org/ELM/ilamb_e3sm_cbgcv2/e3sm/ . however, it has much fewer variables than @thorntonpe 's list.

@tomvothecoder Ok, so even though anaconda.org/cdat/output_viewer shows ESGF/output_viewer listed as a CDAT package, it does NOT introduce a CDAT dependency by including it?

@BunnyVon Ah good idea to match up with the ILAMB categories. I'll try incorporating that. Thanks!

@tomvothecoder Looking at https://github.com/E3SM-Project/e3sm_diags/issues/628 > fixed by https://github.com/E3SM-Project/e3sm_diags/pull/773 > e3sm_diags/viewer/core_viewer.py, it appears you've basically just rewritten the CDAT code from scratch in e3sm_diags. If zppy now needs that too, wouldn't it make sense to break that out into its own package that both e3sm_diags and zppy call? Or since it's only one Python file, is it better to take the low overhead route and just duplicate e3sm_diags/viewer/core_viewer.py in zppy?

@tomvothecoder Ok, so even though anaconda.org/cdat/output_viewer shows ESGF/output_viewer listed as a CDAT package, it does NOT introduce a CDAT dependency by including it?

anaconda.org/cdat is the Anaconda channel for CDAT. This does not necessarily mean all of the packages in that channel are "CDAT packages". The ESGF output_viewer is a standalone package that is a dependency of CDAT (cdp) and not the other way around, which probably explains why it is hosted in that channel (conda-forge wasn't being used back then).
Note, the ESGF output_viewer package is also hosted on conda-forge (comment above).

@tomvothecoder Looking at E3SM-Project/e3sm_diags#628 > fixed by E3SM-Project/e3sm_diags#773 > e3sm_diags/viewer/core_viewer.py, it appears you've basically just rewritten the CDAT code from scratch in e3sm_diags. If zppy now needs that too, wouldn't it make sense to break that out into its own package that both e3sm_diags and zppy call? Or since it's only one Python file, is it better to take the low overhead route and just duplicate e3sm_diags/viewer/core_viewer.py in zppy?

It makes more sense to write an OutputViewer class specific to the needs of zppy. e3sm_diags' core_viewer.py consists of just single small class called OutputViewer, which uses modules from the ESGF output_viewer package.
Alternatively, zppy can import the OutputViewer class directly from e3sm_diags (e.g., from e3ms_diags.viewer.core_viewer import OutputViewer).

Note to self, mostly: using the output_viewer to display these plots nicely is in fact non-trivial. I'm able to generate css and js directories, but not the viewer itself. Thus, in an effort to better understand everything involved with producing the viewer, I'll now try to follow the call-path for the viewer in e3sm_diags on the cdat-migration-fy24 branch. (The numbering scheme attempts to capture the function call hierarchy in a sort-of depth-first-search fashion, where number is the level and letter is the node on that level).
1a. e3sm_diags/e3sm_diags_driver.py main has the following block:
            path = os.path.join(parameters_results[0].results_dir, "viewer")
            if not os.path.exists(path):
                os.makedirs(path)

            index_path = create_viewer(path, parameters_results)
            logger.info("Viewer HTML generated at {}".format(index_path))

In particular:
index_path = create_viewer(path, parameters_results)

1a-2a. e3sm_diags/viewer/main.py create_viewer calls:
viewer_function = SET_TO_VIEWER[set_name]

If we use "lat_lon": default_viewer.create_viewer,, the number of functions used is quite high. So, first let's try a simpler viewer to emulate and see if that's all we need. Let's use "enso_diags": enso_diags_viewer.create_viewer,. That brings us to:
1a-2a-3a. e3sm_diags/viewer/enso_diags_viewer.py create_viewer  calls:
viewer = OutputViewer(path=root_dir)

1a-2a-3a-4a. e3sm_diags/viewer/core_viewer.py OutputViewer.__init__. The class is essentially a wrapper around the ESGF output_viewer, as mentioned in comments above. Let's go back up a level.
1a-2a-3a. e3sm_diags/viewer/enso_diags_viewer.py create_viewer calls:
meta=create_metadata(param),

1a-2a-3a-4b. e3sm_diags/viewer/default_viewer.py create_metadata  -- this function is perhaps not necessary for us to worry about.
1a-2a-3a. e3sm_diags/viewer/enso_diags_viewer.py create_viewer calls:
add_header(root_dir, os.path.join(root_dir, url), parameters)

1a-2a-3a-4c. e3sm_diags/viewer/utils.py add_header -- this function is perhaps not necessary for us to worry about.
1a-2a-3a. e3sm_diags/viewer/enso_diags_viewer.py create_viewer calls:
h1_to_h3(os.path.join(root_dir, url))

1a-2a-3a-4d. e3sm_diags/viewer/utils.py h1_to_h3 -- this function is perhaps not necessary for us to worry about.
1a-2a. e3sm_diags/viewer/main.py create_viewer calls:
index_url = create_index(root_dir, title_and_url_list)

1a-2a-3b. e3sm_diags/viewer/main.py create_index (by way of enclosed function insert_data_in_row) calls:
path = os.path.join(e3sm_diags.INSTALL_PATH, "viewer", "index_template.html")

So, we'd need some different path because clearly e3sm_diags.INSTALL_PATH wouldn't be what we want.
1a-2a-3b. e3sm_diags/viewer/main.py create_index calls:
utils.add_header(root_dir, index_url, parameters)

which was described previously (1a-2a-3a-4c).
I'll try out copying some of the e3sm_diags viewer code and adjusting it now that I understand the process better.

Visuals
Alright, the second commit (9249d1b0c6363878e53d3c9cd4f4a5900555106d) has produced a very basic viewer (with UNIQUE_ID set as test-601-output-viewer).

Accomplished:

[x] A table is produced with variables listed on the left and regions listed on the top.
[x] Plot previews appear when hovering over links.

Need to add/fix:

[ ] Web address should have /viewer/ not /table/
[ ] Rows should have header rows for Atmosphere and Land at the very least. Ideally Land would have sub-headers to match the ILAMB categorization scheme.
[ ] All the links just say title -- they should actually have useful titles.

Directory structure
The produced directory structure:
$ cd /lcrc/group/e3sm/ac.forsyth2/zppy_min_case_global_time_series_single_plots_output/test-601-output-viewer/v3.LR.historical_0051/post/scripts/global_time_series_1985-1995_results
$ ls
table                                     v3.LR.historical_0051_glb_lnd.pdf       v3.LR.historical_0051_n_lnd_LAISHA.png  v3.LR.historical_0051_s_lnd_FSH.png
v3.LR.historical_0051_glb_atm.pdf         v3.LR.historical_0051_glb_lnd_RH2M.png  v3.LR.historical_0051_n_lnd.pdf         v3.LR.historical_0051_s_lnd_LAISHA.png
v3.LR.historical_0051_glb_atm_TREFHT.png  v3.LR.historical_0051_n_atm.pdf         v3.LR.historical_0051_n_lnd_RH2M.png    v3.LR.historical_0051_s_lnd.pdf
v3.LR.historical_0051_glb_lnd_FSH.png     v3.LR.historical_0051_n_atm_TREFHT.png  v3.LR.historical_0051_s_atm.pdf         v3.LR.historical_0051_s_lnd_RH2M.png
v3.LR.historical_0051_glb_lnd_LAISHA.png  v3.LR.historical_0051_n_lnd_FSH.png     v3.LR.historical_0051_s_atm_TREFHT.png  viewer

$ ls table
index.html  summary-table

$ ls viewer
css  js

Compare that with example output from e3sm_diags:
$ cd /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v3_www/issue-346-diags-20240731/v3.LR.historical_0051/e3sm_diags/atm_monthly_180x360_aave/model_vs_obs_1987-1988/viewer
$ ls
cmip6  cmip6-comparison-data  diurnal_cycle  index.html  index.json  lat_lon  table  table-data  taylor  taylor-diagram-data  viewer

$ ls table
index.html  summary-table

$ ls viewer
css  e3sm_logo.png  js

The directory structures actually appear to match up ok, however there is no index.html in this commit's version of the viewer.

Ok with the third commit (cd5b100d1fb350c877a46f0a0f407524096ef555), I am able to categorize the variables. I dislike the hard-coded variable categorization, but I suppose it will have to do if we have no other method.
Results

@BunnyVon How does this look? It doesn't have all the bells and whistles of E3SM Diags, but it is the same general display. I tried to categorize variables the best I could using ilamb.cfg, but I don't think I did it totally correctly.

I think it looks great. what do you think, @thorntonpe?
@forsyth2 , in terms of the variables, I am finalizing my preprocessing script with Nate Collier, which has the info for the conversion between ilamb variables and E3SM variables for the ilamb scorecard. It's soon to be completed, and I can share afterward.

which has the info for the conversion between ilamb variables and E3SM variables for the ilamb scorecard

Awesome, thanks @BunnyVon!

@forsyth2 and @BunnyVon  - After a hectic few weeks I am now able to give this thread the attention it deserves. Thanks for your patience. The overall timeseries plotting capability looks excellent - the plots themselves have good graphic quality and having them all accessible through a single index page is exactly what we need. Thank you!
In our land group call today we will review and provide further feedback. I will also be following up soon with the full list of variables, and the categorization we would like to see in the master list. Even before the group discussion, I know there are a few changes that we would like to see. I'll list them below, and then will follow up with further thoughts from the group.

Each variable has a long name on the history file. It will be helpful to display that below the variable name in the title block for each plot. That should help to address questions like the one Sha raised above about LAISHA.
Some flux variables (such as GPP) are best presented as annual global (or N/S regional) totals. That means doing the multiplication by time and the totaling by (gridcell area * landfraction). Those typically also need to be scaled, for example from grams carbon to petagrams carbon. Other flux variables (such as QVEGT) are best displayed as global mean values. When we provide the full list of variables we can also provide in a format that includes columns of information for each variable about how to present it in the plots. It would be good to get together with you to discuss the best way to transmit that information.
We would eventually like to have the ability to plot two simulations together, perhaps distinguished by red vs blue lines. Does this seem feasible?
Thanks again for all your work on this.

Thanks @thorntonpe!
To your points:

I know E3SM Diags has short and long name functionality. I can see if something similar is simple to extract/implement here.
Yes, a meeting would be good to discuss precise needs/methods.
I don't think this is impossible, but I do foresee it being somewhat involved. E3SM Diags has model-vs-model functionality but otherwise most of zppy is focused on processing one simulation's output, and adding in the Diags mvm functionality has been involved (e.g., a lot of duplication of parameters). I'd need to do some prototyping to see for sure, but I imagine that could be a pretty big overhaul/refactoring of zppy/the global-time-series task specifically to be able to handle that (especially if you wanted more than two simulations' outputs)

Next steps with this pull request:

Land team will provide CSV of the form variable name, total or area, scale factor, new units, category label
That CSV will be used as a metadata look-up table for Python to construct a dictionary to organize the viewer.
There should be a separate viewer for each component (e.g., atm, lnd).
Goal is for this to be merged by the next zppy / E3SM Unified release.

After this is merged: with the viewers looking how we want for one simulation, we'll follow an approach similar to Diags model-vs-model (add equivalent parameters for ref) to get a second line plotted on the graphs. The start years may have to be adjusted. Two options 1) make each "years since start" or 2) put one x axis on top and another on the bottom

@forsyth2 The averages that NCO produces from EAM simulations are true global averages. The "global" averages that NCO produces from ELM simulations are by definition global land-only averages. They are computed by summing each field (e.g., FSDS) over every valid ELM gridcell, multiplied by the gridcell area, multiplied by landfrac. This "global" sum is then normalized by the valid land area, i.e., by the sum of area*landfrac.
Responses to your questions:

Sort of. To convert ELM averages from a mean to a total, multiply them by the global land area, i.e., the sum of area*landfrac. To convert EAM averages from a mean to a total, multiply them by the true global area. Global totals for both EAM and ELM variables could be obtained from the current climos without first updating NCO. The true global land area (for EAM) and the land-only area (for ELM) are time-invariant and can be computed once offline and used by any subsequent processing to convert averages to totals for select variables.
ncclimo could be modified to produce timeseries of all totals or all means with the addition of a new switch/option. However, it sounds like ELM users might want some variables as global totals (e.g., CO2 fluxes) and some as global means (e.g., temperature). Modifying ncclimo to input which fields get which treatment would be a significant project. It might be better accomplished in E3SM diags since the treatment will be intimately connected to how the variable is plotted. LMK if you'd like me to perform either of the two potential ncclimo modifications that have been mentioned.
Yes, NCO properly weights months based on the model (365-day) calendar.

FYI the latest NCO snapshot includes a new option to control the global/regional statistics that ncclimo computes for timeseries: --rgn_stt=avg|sum. When invoked with --rgn_stt=avg the behavior is identical to the current output timeseries which are averages. When invoked with --rgn_stt=sum the averaged field is multiplied by the sum of area variable. While this would work fine if the area were given in m2 and the field was given in m-2, that is rarely the case. ELM uses km2, so a scale factor of one million is needed to correct the sum for many variables. EAM uses sr-1 for area so it needs a different scale factor. The ncclimo accepts a second new option --sum_scl=sum_scl. This scale factor will multiply the integrated field value. The whole procedure is model and variable-specific so it's hard to have a generic solution on the front end, and I thought this generic feature of a simple scale factor might be useful. Example usage is:
ls v3.LR.piControl.elm.*.nc | ncclimo -P elm --split --rgn_stt=sum --sum_scl=1.0e6 -v GPP -s 460 -e 461 --drc_out=${DATA}/ne30/rgn
Feedback welcome.

@czender thank you for having this implemented so quickly. I'm tagging land model developers @thorntonpe @BunnyVon and @darincomeau  about this new nco development and for feedbacks.

Rebased off main, after the modularizing refactor of #628. Luckily, #628 touched the <task>.py files whereas this pull request is more focused on the .py files called from global_time_series.bash, so changes were minimal: only zppy/global_time_series.py had a minor merge conflict.

Closing in favor of https://github.com/E3SM-Project/zppy-interfaces/pull/9 and https://github.com/E3SM-Project/zppy/pull/648


Interaction with the thymeleaf-extras-springsecurity4 package

While working through #148, a problem was encountered when including the thymeleaf-extras-springsecurity4 package.  Link https://github.com/ultraq/thymeleaf-layout-dialect/issues/148#issuecomment-282737853 and quote:

thymeleaf-layout-dialect 2.1.2 seems to have a compatibility issue with thymeleaf-extras-springsecurity4?
If I put layout dialect in pom.xml, I get: Could not initialize class org.thymeleaf.extras.springsecurity4.auth.SpringVersionSpecificUtils
thymeleaf-extras-springsecurity4 has support on Spring 5:
https://github.com/thymeleaf/thymeleaf-extras-springsecurity/blob/7e2847bfe950172d2b733b5dae5e2572b32a5828/thymeleaf-extras-springsecurity4/src/main/java/org/thymeleaf/extras/springsecurity4/auth/SpringVersionSpecificUtils.java

Investigate to see if there is a problem with having both of these dialects in the classpath, and if there is what can be done about it.

This is an old issue, but I tried loading the latest thymeleaf-extras-springsecurity4 package in a project that had the layout dialect, and couldn't replicate the issue.  I'm hoping it was specific to this case (the linked issue was a long debugging session!) or has since been fixed with newer versions of either the layout dialect or the spring security extras package.  Closing.


[SPARK-45997][PS][PYTHON][CONNECT][DOCS] Remove deprecated APIs from legacy Koalas

What changes were proposed in this pull request?
This PR proposes to remove deprecated APIs related to legacy Koalas for upcoming major release, Spark 4.0.
Related docs are updated as well.
Why are the changes needed?
To cleanup the API surface by removing deprecated legacy Koalas-related APIs.
Does this PR introduce any user-facing change?
Yes, pyspark.pandas.DataFrame.koalas, pyspark.sql.DataFrame.to_koalas, pyspark.sql.DataFrame.to_pandas_api_on_spark will be removed.
How was this patch tested?
The existing CI should pass.
Was this patch authored or co-authored using generative AI tooling?
No.

merged to master


Improve hashing performance

The improvement is in the range of ~10-20x depending
on platform and compilation mode, e.g. on my M1 MacBook Pro
AOT mode goes from 6ns/byte to 0.3ns/byte and JIT mode goes
from 5ns/byte to 0.4ns/byte.
The bulk of the improvement comes from avoiding allocating
a temporary ByteData for every readLE64. While it might
be reasonable to expect this allocation to be eliminated
Dart VM compiler is currently unable to do that for several
reasons (most of which can be easily fixed).
The second part of the win comes from avoiding boxed List
type for accumulator and using Int64List instead.
Additionally JIT compiler trips over toUnsigned(64) which
should be a no-op but instead hits a suboptimal implementation
of out of range left shift (1 << 64) and falls over fast path, so I
removed that as well.

Thanks! This is very interesting! I shall take a deeper dive and try and get this merged today.

I've managed to squeeze out an extra 200MB/s by avoiding a List allocation when creating the Int64List:
final acc = Int64List.fromList([
    kXXHPrime32_3,
    kXXHPrime64_1,
    kXXHPrime64_2,
    kXXHPrime64_3,
    kXXHPrime64_4,
    kXXHPrime32_2,
    kXXHPrime64_5,
    kXXHPrime32_1
  ]);

...becomes:
final acc = Int64List(8);
  acc[0] = kXXHPrime32_3;
  acc[1] = kXXHPrime64_1;
  acc[2] = kXXHPrime64_2;
  acc[3] = kXXHPrime64_3;
  acc[4] = kXXHPrime64_4;
  acc[5] = kXXHPrime32_2;
  acc[6] = kXXHPrime64_5;
  acc[7] = kX


fix(check-merge-safety): don't use truncated branch name for API calls

:pencil: Description

We need to still invoke the GitHub API with the full branch name, not the truncated one.

:tada: This PR is included in version 1.35.6 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


[Question] Revoking tokens on refresh

Is there anyway to tokens programmatically? Right now tokens are revoked based with the following code
    Authorize.jwt_required()

    jti = Authorize.get_raw_jwt()['jti']
    redis_conn.setex(jti, settings.access_expires,'true')

The problem lies  with Authorize.get_raw_jwt which only allows revoking of the token that is required.
My use case is that on refresh, I would like to refresh both access_token and refresh_token. Is there anyway to accomplish that?

Looks like I can get the tokens from the cookies straight from the Request object and access their jti with Authorize.get_jti and revoke the tokens from there. Would that be the only way? Would be great there was a get_token_from_cookie(name="access") method that I can call to simplify things a little bit more

Why are you revoking the access token on refresh? The refresh endpoint should only be getting hit if the access token is already invalid.

I'm not super familiar with the subject but let's say the refresh last for 7 days, the user would be logged out no matter what after 7 days right? What should I be doing if I wanted to extend the refresh past the expiry if the user stays logged in?

You can't extend the refresh past the expiration. You can change the expiration time though. It's automatically invalidated once it's expired.

yes so I'm planning to refresh the refresh token as well, Is that a bad idea?

Found this stackoverflow post in my research on jwt auth.
I think you're trying to implement what's called refresh token rotation, where you refresh the refresh token every time it's used.
Be aware that this can lead to an attacker intercepting the RT, and having infinite uses if the user never returns to the app.


Debian "Testing" how to

Installation steps

Download the full DVD ISO (3.7 GB) from here
Download the non-free firmware (e.g. Intel WLAN drivers) from here
Unzip the firmware on to a USB stick, in a directory named "firmware". More about this here
Plug-in both USB sticks (or any other media) and boot the machine
Follow the installation process

Optional
After first boot, replace /etc/apt/sources.list with the following: sources.list.zip
Notes

Using the netinst ISO is discouraged as it may lack needed drivers for the hardware.
It's possible to create a custom installer following these instructions.
To be able to use sudo, be sure not to provide a root password during the installation. That way the root account will be disabled.

Packages to install

build-essential
cmake
git
cppcheck
valgrind
libarmadillo-dev
libspdlog-dev
libtclap-dev

Moved this to the wiki.


Clean up Collective tags in database

When adding a tag through the interface it will be lowercased automatically. When querying Collectives using tags the string is looking for lowercased versions, however, there are cases of tags that use capital letters in the database, these records won't be returned.

[ ] Lowercase all Collective tags in db
[ ] Transform obvious duplicates

[ ] opensource => open source
The "problem" of very similar tags should become less of an issue with #5523, that will autocomplete and show most popular tags by default

Maybe implement that as a script in the API project to easily re-run it again in the future?
For reference, we had a migration with a similar intent a few years ago: https://github.com/opencollective/opencollective-api/blob/main/migrations/archives/20200425013627-change-tags-to-lowercase.js.

Closing this, script merged in https://github.com/opencollective/opencollective-api/pull/8462 and have been ran in production.


Editing items can duplicate images

Apparently, at least in some circumstances, editing an item (case, method, etc.) with images can cause some or all of the images to be duplicated.
Relevant:
https://github.com/participedia/usersnaps/issues/73
https://github.com/participedia/usersnaps/issues/76

Cannot reproduce, closing.