Add DoctrineKeyValueStyleRule

This implements a new Rule for methods that build queries from simple
ingredients: a table name, and arrays of column name/value elements.
It checks that all table and column names are valid, and that the array
value types are (maybe) accepted by the column types.
For configuration, the class::method is specified as usual, with an
optional comma-delimited list of array agrument indices after the hash.
Specifying the indices is necessary because this rule covers 3 kinds of
simple builders:

No array arguments (for truncate-like queries)
One array argument (for insert/delete/replace-like queries)
Two array arguments (for update-like queries)

The most notable examples of this type of method are the Doctrine\DBAL
insert, delete, and update methods on their Connection object, which
have additional arguments that are not arrays of column names/values.
Closes #501.

A couple open questions about implementation for you:

Naming. I chose "QueryAssembler" as the rule name to avoid confusing it with the Doctrine\DBAL QueryBuilder, which has a much more advanced API. An alternate name I was considering was "SimpleQueryBuilder", but I'm happy to change it to whatever you want.
Configuration. The comma-delimited list of array argument indices is a new config syntax. This was the best I could come up with to support the existing Doctrine\DBAL methods. If you prefer to handle it a different way, just let me know.
Type checking. Value types are validated against column types, which is a very useful check, but it is more strict than what is permissible by most DBALs (due to type coercion). It might warrant being a new option, which could enable type checking on parameters in prepared statements as well. Do you want me to remove type checking here and propose it in a separate PR?

Thanks again for your time, and I appreciate any feedback you can provide!

I don't understand why only the pdo-mysql replay test is failing. I might have expected that I need to update the cache files, but then I'd expect the mysqli replay test to fail as well. When I run the tests locally they make pretty significant changes to the cache files, so I'm hesitant to include them in the PR (most of it is adding a leading \ to the object names for some reason...).

thank you


Update Composer dependencies (2021-12-10-00-11)

Visual regression test passed!
View the visual regression test report


delete 'cost' in oeo-social

closes #975

Cost is still existing in oeo-social line 1237. I hope it will delete automatically or is it because it is implemented in oeo-shared?


Add systematic tests for escaped regex characters.

Hi,
To save you spending your time writing any new tests, here are the tests I added earlier of all the characters being escaped by escapeRegExp(), if you are happy to merge them in to your code.
I found that =, ! and : gave errors when they were escaped, and wasn't able to write a test for which escaping them helped, and so I didn't re-added them to the regex.
I don't know what the solution is there.
Alternatively, if you could advise me on how they should be handled, I can always fix them after your PR is merged.
But please don't spend your time adding more test coverage independently.

Not to worry - I'll merge this in to the main Tasks repo instead.


Enable typed-continuations feature by default

This PR enables the typed-continuations Wasm feature by default, as well as its dependencies, as long as gc is enabled in Cargo.
This requires adapting two test cases that disable reference types. Otherwise, we would try to disable reference-types while keeping function-references enabled, even though it depends on the former.

I don't have particularly strong feelings about this PR, as it just simplifies the steps to run WasmFX programs a bit.
But regarding your concern about benchmarking asyncify: If enabling our feature regresses performance on non-WasmFX programs in a measurable way, I'd simply consider that to be a bug in our implementation.

I don't think it is the case -- at least I have been unable to measure it in the past. For me, it is more about being rigorous to ensure scientific soundness. It is my only worry really. I suggested to add a single option wasmfx which toggles all of the dependencies at once. I'd like to discuss this possibility still.


Fixing tests for ruamel yaml 0.18+

Fixes: #198

ruamel.yaml 0.18.0 removed deprecated API methods like safe_dump, safe_load_all, and usage of dump without a stream argument. This PR updates the usage by utilizing the yml object in pykwalify.compat for equivalent safe API methods

@lilyinstarlight My gut feeling tells me that this PR probably should also include an update to the requirements file to pin ruamel.yaml>=0.18 then? Specially if this is probably not backwards compatible if they have changed around the functionality with the methods that ruamel uses?

I am thinking that this big of an update to dependencies probably should bump up the release to 1.9.0 for instance and just say that either you upgrade rumale, or you stay behind until you can update ruamel wherever that is used

Thank you for the quick reply btw!
I can still bump the version if you would prefer, but I just figured I'd note this is a completely backwards compatible change (as far as I can tell), since I neglected to mention that in the initial description :)


Sort compatible types to use longer base types.

There are base types in the same gear type where one base type is a
non-trivial substring of another base type.  For example, there are
"Tricone", "Noble Tricorne", and "Sinner Tricorne" base types.  Since we
determine and item's base type by checking if each base type is a
substring of that item's type line (since the type line can include
affixes), we need to make sure we do not use a shorter base type when a
longer one is more correct.  The simple fix used here, after building
the list of compatible types for a gear type, sort that list by string
length, so the longer strings are checked first.

This is resolving the side effect rather than the base issue imo.
https://github.com/Stickymaddness/Procurement/blob/669db75cfbe28dce98ae3c5e43d1cd91b1ea6c0c/POEApi.Model/GearType/GearTypeRunner.cs#L32
This method uses contains - which causes the above behaviour. Might be better to use equals and StringComparison.CurrentCultureIgnoreCase

Using a test for string equivalence won't work here because the type line for magic items includes the affixes (I should add a test with that).
I agree it's not an ideal solution, but it's a simple change that fixes the bug, and it's generic (it won't need to be updated as new items are released, unlike, for example, the lists of incompatible types). I haven't come up with a more elegant solution yet.
The whole process of checking substrings to determine base types isn't ideal, either, but for the most part, it works. I'd like to take a closer look at all the data we're getting from the API to see if there's a better solution (particularly now that we can get the full affix breakdown in the client), but I'm still working on low-hanging (and user-visible) changes.

It might be worth looking into the API, there's probably some new fields that Procurement is missing that might help in this regard.
Merging this as I agree that this is a quick win.


Implement zlib package with a Rust crate

It looks like flate2 and libflate are two options for crates we might use.
The implementation should pass ruby/spec, which we can validate by running:
pushd spec-runner/vendor/ruby/library
cargo run --bin spec-runner zlib/*/*.rb zlib/*.rb

I would like to tackle this issue

Hey @sphinxc0re, are you still working on this? Is there anything I can do to help?

I'm currently a little overflown with work, so I didn't really move along with this. I'm really sorry. I would like to take some time on the weekend to implement this. It shouldn't be too hard.


Morning session: Peer-to-peer code review

Highlights
Your code looks great and is working well, waiting to see the final product, please do check the branch on deployment.
MOHAMED GAMIL ELDIMARDASH

The add remove functionalities dont work. please check them
Very clean and organised code

Hi, @gealsanchez 👋,

Highlights💡

Good commit message.✔
Good PR Summary.✔
Great UI design.✔
App functions as it should.✔

STATUS: APPROVED 🕺🥳:100:
Congratulations🍾🥳!! Your project is ready for presentation!  There is nothing else to say other than... it's time to merge it :shipit:

Optional Suggestions:
Every comment with the [OPTIONAL] prefix is not crucial enough to stop the approval of this PR. However, I strongly recommend you to take them into account as they can make your code better.
You can also consider:

N/A

Cheers and Happy coding! :clap:  :computer:  :wine_glass: Good luck! 🎉

Feel free to leave any questions or comments in the PR thread if something is not 100% clear.
Please, remember to tag me @miltonhenschel in your question so I can receive the notification.
Use @miltonhenschel on slack for faster responses.

Please be reminded to not open a new pull request after you get a review, only use the same one :thought_balloon:
###Please, do not open a new Pull Request for re-reviews. You should use the same Pull Request submitted for the first review, either valid or invalid unless it is requested otherwise.

As described in the Code reviews limits policy  you have a limited number of reviews per project (check the exact number in your Dashboard). If you think that the code review was not fair, you can request a second opinion using this form.


Convert CJS project into ESM

VS Code allows changing a single file to ESM. That is good if the project is small, but when a project has 100 files, it gets really hard to convert so many files to ESM.
This feature aims to let VS Code convert an entire node project into ESM.

We try to keep VS Code lean and we think the functionality you're asking for is great for a VS Code extension. Maybe you can already find one that suits you in the VS Code Marketplace. Just in case, in a few simple steps you can get started writing your own extension. See also our issue reporting guidelines.
Happy Coding!


Container Credentials URL might be incorrect

I think this line has a bug:
https://github.com/rusoto/rusoto/blob/9cf8c2ae72746d371878ab7615ce14b5698a2f13/rusoto/credential/src/container.rs#L46
In that it appends an extra '/' to the end of the container URI.
For example, this works inside a container:
curl http://<IP_ADDRESS>$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
{"RoleArn":"arn:aws:iam.....

but this returns an error:
curl http://<IP_ADDRESS>$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI/
{"code":"InvalidIdInRequest","message":"CredentialsV2Request: ID not found"}

I could be wrong. In any case, thank you for all your work on this! I'm excited about the plans for making this library async.

Thanks for the info!  I don't have a project in ECS that uses Rusoto for easy testing, but it shouldn't be too hard to whip one up to see if this is an issue or not.

Doh! I thought I got all of those. This is totally a problem (and yes we'll need to cut out that trailing slash). We really need to get something using ECS for testing. I think most of the time we just test instance profiles.

In rusoto version 0.29, ECS support is broken, causing access denied when accessing S3 for me. But works fine in 0.28.

Thanks for the confirmation! I'd like to check the fix myself before doing a new release of the credentials crate, but this might take a few days as I'm loaded up with other work and don't have an ECS cluster ready for use at the moment.
The PR to remove the slash is welcome, I can use that to build programs that test the fix. 😄

I believe when https://github.com/rusoto/rusoto/pull/890 lands and the new credential crate is published, deleting the Cargo.lock and/or doing a cargo update should fetch and use the new credential crate that has this behavior working again.  👍

Thanks Mat!

Hot off the presses! https://crates.io/crates/rusoto_credential version 0.9.2 is out. 😄


Loader Types

Users would do:
/// <reference path="../node_modules/fuse-box/dist/LoaderAPI.d.ts" />

:rose:

Thanks!

@basarat  personally i am sick of TS script reference, isn't there a better way to do this? i mean with all the improvements Microsoft did to the typing support, like install through npm,etc.. we still have to do this ?

personally I am sick of this TS type reference style, isn't there a better way to do this

I agree 💯  but the issue is in our code base. The issue is that the file injects a global so we can't do import {FuseBoxLoader} from "somewhere"
To add globals user needs to use either <reference or explicitly list the file in our tsconfig.json as there is no js way to link from user code to point to the global declaration file.
Note that the loader API doesn't need to be global (import {Loader} from "fusebox/loader" can be made to work), just that it is global right now and changing it will be a breaking change :rose:

@basarat  makes total sense!, can't we just extend the tsconfig when we get it from user to dynamically add those types before we start fusing, or maybe i am making crap sense :)

https://github.com/fuse-box/fuse-box/pull/232/commits/aa156b5bb4ff49eb6e990daa6a2646e534cc944e :rose:

@basarat  you rock 💯


Susie_rss update

Take out loading region outside loop so no need to load LD twice.
I prefer to make the study name appear on file name so that I can directly know what studies I've already analysed. Instead of putting them all into a large rds output.

@SamuelSun2000 sorry we still want all studies for the same region going into the same large file. This is for convenience of data integration and comparison purposes. We have an extra later step to address your concerns regarding 'transparency'. For now please revert it back to everything into one RDS file. Thanks!


feat(colocalisation): new step runs ecaviar on credible sets

This PR includes:

Changes in the colocalisation step to run eCAVIAR on all credible sets.
For the time being, I've switched COLOC off.

QC
Results are here: gs://genetics_etl_python_playground/output/python_etl/parquet/XX.XX/colocalisation

Total: 10,474,884
Mean CLPP: 0.134
0.8 cutoff:

count: 746,515
mean: 0.95
Results with just 1 colocalising variant: 491,595

Question:
Do we need millions of colocalisation results where PP~0.1? I don't think these are relevant for a data consumer. It could help L2G deprioritise genes with low PP, but it's not very different than not having them as the feature'd be 0. We can test the impact.
Context
The step doesn't have the overlaps dataset as starting point, we simply start off credible sets for which we calculate the overlaps on the go. In a conversation with @d0choa, we agreed that the overlaps set needed to run COLOC and eCAVIAR aren't the same. For eCAVIAR we simply need information about the variants that are common between diff studyLocus, whereas in COLOC we need all variants from both pairs. That is why we conveyed that instead of writing the dataset in an optimised format, we would compute an exploded version (potentially huge, but parallelizable) whenever these are needed.

Codecov Report
Attention: 33 lines in your changes are missing coverage. Please review.

Comparison is base (42b366c) 85.67% compared to head (85d37c1) 86.94%.
 Consider uploading reports for the commit 9a121f7 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##              dev     #396      +/-   ##
==========================================
+ Coverage   85.67%   86.94%   +1.27%
==========================================
  Files          89       92       +3
  Lines        2101     2428     +327
==========================================
+ Hits         1800     2111     +311
- Misses        301      317      +16

Files
Coverage Δ

src/airflow/dags/common_airflow.py
90.38% <ø> (ø)

src/airflow/dags/finngen_preprocess.py
100.00% <100.00%> (ø)

src/otg/dataset/l2g_feature_matrix.py
82.92% <ø> (+7.31%)
:arrow_up:

src/otg/dataset/study_locus.py
96.20% <100.00%> (+0.04%)
:arrow_up:

src/otg/datasource/eqtl_catalogue/study_index.py
85.00% <ø> (ø)

src/otg/datasource/finngen/study_index.py
100.00% <100.00%> (ø)

src/otg/datasource/finngen/summary_stats.py
100.00% <100.00%> (ø)

src/otg/datasource/gwas_catalog/study_index.py
100.00% <ø> (ø)

src/otg/datasource/ukbiobank/study_index.py
100.00% <ø> (ø)

src/otg/l2g.py
58.06% <ø> (ø)

... and 9 more

@DSuveges No, I'm pretty certain the coloc dataset wasn't filtered in production either. I was just raising the question.

Coloc in production required 100 overlapping tags which resulted in a relatively high baseline H3/H4. This was discussed as too stringent and something to avoid. Of course, this comes at the cost of a lot of weak signals as you are observing.
If you think handling the data is not an issue and we won't pay reasonable performance costs, I would keep all the data points. There might be a marginal value in differentiating overlapping locus with high metrics vs low metrics vs non-overlapping locus.


NPSP: BDI-AM fix bug caused by case mismatch on fieldname map key

The map of Data Import Field -> Target field generated from Advanced Mapping was using a mixed case field name for the map key, unlike the help text variant which was forcing map keys to lowercase. There was at least one bug attached to this behavior, and this work correctly lowercases the map keys for advanced mapping.
Critical Changes
Changes
Issues Closed

4744

New Metadata
Deleted Metadata

**lurch: attach W-037136

Tracking W-037136


Can we integrate this with Vuejs

Hi Bill
Is it possible that we can integrate this in Vuejs Project?

The app/client can be modified to fit any framework. See (Building)[https://github.com/billchurch/webssh2/blob/main/BUILDING.md] for more information.
However, I'm working on a project to split the client out of this repo, simplifying the process and making it easier to support different frameworks or do other custom modifications.
I hope to have something usable in the next couple of weeks. Some other priorities have kept me from completing some testing.

Thanks, I will wait :)

Ok. I don't have the "current" version refactored, but I did "refactor" and older branch. It's actually a total re-write but I had to target an old version of node for a specific use case.
The front end is:

https://github.com/billchurch/webssh2_client

The back end is currently in the bigip-server branch

https://github.com/billchurch/webssh2/tree/bigip-server

I created a tag on dockerhub with this new version. It doesn't exactly solve your problem right now but I was hoping maybe you'd take a look at it and see what you think.
docker run -it --rm --name webssh2 -p 2222:2222 -e DEBUG="webssh2*" billchurch/webssh2:bigip-server

Documentation is kind of all over the place, and probably incomplete but you can look at the READMEs of the two repos:

https://github.com/billchurch/webssh2/tree/bigip-server
https://github.com/billchurch/webssh2_client

The idea here is we can fork off webssh2_client for different frameworks and point them to the server instance...
Does that make sense? Let me know what you think.

Well I had integrated your old web2ssh into my vue app. In my app I am simply open the iframe and using token instead of any html form for username, hostname or password. The Vue app on the fly get the token from python api.
http://localhost:2222/ssh/host?token=${newToken}
then in web2ssh/server/app.js, I had modified the following to use that token
//app.get('/ssh/host/:host?', (req, res) => {
app.get('/ssh/host', (req, res) => {
const token = req.query.token;
if (!token) {
return res.status(400).send('Token is required');
}
try {
const decoded = jwt.verify(token, 'token_secret');
  req.session.username = decoded.user,
  req.session.userpassword = decoded.password,
  req.session.privatekey = decoded.privatekey,

  res.sendFile(path.join(path.join(publicPath, 'client.htm')));

  // capture, assign, and validate variables
  req.session.ssh = {
    host: decoded.host,
    port: decoded.port,
    localAddress: '',
    localPort: '',
    header: {
      name: req.query.header || config.header.text,
      background: req.query.headerBackground || config.header.background,
    },


Add missing Node annotations in documentation

Please ensure your pull request adheres to the following guidelines:

[x] For first time contributors, read Submitting a pull request
[x] All code is covered by unit and/or runtime tests where feasible.
[x] All commits contain a well written commit description including a title,
description and a Fixes: #XXX line if the commit addresses a particular
GitHub issue.
[x] All commits are signed off. See the section Developer’s Certificate of Origin
[x] Provide a title or release-note blurb suitable for the release notes.

Adds io.cilium.network.ipvn-cilium-host and io.cilium.network.ipvn-health-ip annotations since they are visible on k8s nodes.
https://github.com/cilium/cilium/blob/ba9031bb45a0da79ee3a843a8c9b50807d0e40be/pkg/annotation/k8s.go#L33-L46.
Signed-off-by: David Bouchare<EMAIL_ADDRESS>

Coverage increased (+0.02%) to 37.011% when pulling a560f8ed653fe9c5224e564e0d75c35966f5edf9 on DataDog:davidb/host-ip-documentation into ba9031bb45a0da79ee3a843a8c9b50807d0e40be on cilium:master.

I'm not sure how to fix the table issue, cc @qmonnet.
/github/workspace/Documentation/concepts/networking/ipam/kubernetes.rst:49: WARNING: Malformed table.
Column span alignment problem in table line 10.

@dabcoder Would you mind squashing your commits into one and sign it off?

@dabcoder Would you mind squashing your commits into one and sign it off?

Sure. And done.


Fix duplicated "the" in pipline aggregation docs

Description
Fix minor typo
Issues Resolved
none
Check List

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

My previous PR (446) didn't have a sign-off, so I amended the commit with the -s flag. For that I had to clone the code locally. It took more time and effort.
This time I tried to bypass that step and was hoping to be able to make such a minor fix right from the GitHub's web interface/code editor by copy pasting my (successfully approved) sign-off from the former PR right into the details field of the commit. Sadly the DCO bot doesn't seem to approve of it and I can't find the difference between the sign-off statements of this and the former PR.

@lukaromih Are you using the web editor to make commits? That method can be a little iffy, but there is a Chrome extension that you can use.  The DCO check runs off of the entire branch, so if there's even one commit in your branch that doesn't have the sign-off, the whole thing fails.
Also, for housekeeping purposes, perhaps you can continue committing to the same PR as you find and fix typos? We can prevent things from becoming too unruly that way.

I fixed the DCO issue (and may have found the reason for failure, but not yet sure).
@keithhc2 Thanks for the heads up about the extension. Will keep it in mind if the workflow requirements call for it in the future.
Agreed about housekeeping. I even wanted to cherry pick this commit on top of the one from PR 446 and close this one. But then:

I'd have to rename it into something more generic
How would you or I know, when enough typos (or all that I mean to fix in a short time period) have been gathered and the PR can be finally be closed and merged?

@lukaromih I don't think typos are something we'll ever be rid of completely 😄 so it's really just a judgment call at that point. You could create your PRs as drafts and then switch them over at, say, once a week, or when you feel like they're at a more ready state, and we can merge them on a weekly basis.

@keithhc2 I've added, what I think is the last commit in this "batch" of typo fixes. I'm not done with documentation yet, though I'll be using it more actively in a few weeks again. I'll make sure to add anything I might find onto my personal list, then if/when things add up, I'll make sure to follow your input from this PR (draft PRs, etc.).
So unless I need to tweak anything or make a rebase, you can use and close this PR.
Also this PR includes the commit included in PR-466 + 2 others on top of it, so you might as well close that one as well.
As per your guidance, I will not create separate PRs in the future for such trivial fixes.


Adding missing v0.20.0 files back to develop

Description
When releasing v0.20.0RC2 and v0.20.0, we ran CI where it committed straight to the release branch and not to develop. This PR ports over those changes so we don't miss them as we move forward.
❗ As an action item, I'm going to make sure CI always commits to develop and then cherry-picks the changes to the release branch so we don't have this issue going forward. @kwigley if you have thoughts on this , let me know.
Checklist

[x] I have signed the CLA
~- [ ] I have run this code in development and it appears to resolve the stated issue~
~- [ ] This PR includes tests, or tests are not required/relevant for this PR~
[x] I have updated the CHANGELOG.md and added information about my change to the "dbt next" section.

Perhaps it makes sense to stage the release into two commits:

I like the idea of making the commits separate. We can do that. I think what I envision the release process looking like is making all these commits on develop and then cherry-picking them to the release branch instead. We should easily be able to flip that direction today.
I want to add logic in the workflow to check for if steps have been run already so it make it easy to rerun if a failure happens. For this it should be easy to check if we already did a commit and cherry-pick and skip ahead if that is the case


Bevy segfaults when using wayland

Bevy version
Bevy 0.8.1 + 0.9
[Optional] Relevant system information
OS: Ubuntu 22.04.1 LTS, Wayland. The app is run natively
Adapter info printed by bevy:
2022-09-13T14:08:53.880285Z  INFO bevy_render::renderer: AdapterInfo { name: "Intel(R) HD Graphics 620 (KBL GT2)", vendor: 32902, device: 22806, device_type: IntegratedGpu, backend: Vulkan }

What you did
Create the following project:
// main.rs
use bevy::prelude::*;

fn main() {
  App::new().add_plugins(DefaultPlugins).update();
}

#Cargo.toml
[package]
name = "bevy_crash"
version = "0.1.0"
edition = "2021"

[dependencies]
bevy = { version = "0.8", features = ["wayland"] }

And then cargo run it.
What went wrong
The app was supposed to show a window, close it and exit. Instead of that, a segmentation fault occurs.
Additional information
I have run the program with gdb connected and managed to get a backtrace:
#0  0x00007ffff7b1f85a in ?? () from /lib/x86_64-linux-gnu/libwayland-client.so
#1  0x00007ffff7b211bb in wl_proxy_marshal_array_flags () from /lib/x86_64-linux-gnu/libwayland-client.so
#2  0x00007ffff7b21caf in wl_proxy_marshal_flags () from /lib/x86_64-linux-gnu/libwayland-client.so
#3  0x00007fffd937038c in ?? () from /usr/lib/x86_64-linux-gnu/libvulkan_intel.so
#4  0x00007fffd9370536 in ?? () from /usr/lib/x86_64-linux-gnu/libvulkan_intel.so
#5  0x000055555750959d in ash::extensions::khr::swapchain::Swapchain::destroy_swapchain (self=0x7ffffffeb7e0, swapchain=..., allocation_callbacks=...)
    at src/extensions/khr/swapchain.rs:29
#6  0x00005555573d0353 in wgpu_hal::vulkan::instance::{impl#5}::unconfigure (self=0x555559012970, device=0x5555592aab68) at src/vulkan/instance.rs:746
#7  0x0000555556ee7671 in wgpu_core::hub::Hub<wgpu_hal::vulkan::Api, wgpu_core::hub::IdentityManagerFactory>::clear<wgpu_hal::vulkan::Api, wgpu_core::hub::IdentityManagerFactory> (self=0x555559018538, surface_guard=0x555559018508, with_adapters=true)
    at /home/innocentus/.cargo/registry/src/github.com-1ecc6299db9ec823/wgpu-core-0.13.2/src/hub.rs:822
#8  0x0000555556edad33 in wgpu_core::hub::{impl#72}::drop<wgpu_core::hub::IdentityManagerFactory> (self=0x555559018430)
    at /home/innocentus/.cargo/registry/src/github.com-1ecc6299db9ec823/wgpu-core-0.13.2/src/hub.rs:1024
#9  0x00005555571e5907 in core::ptr::drop_in_place<wgpu_core::hub::Global<wgpu_core::hub::IdentityManagerFactory>> ()
<...a bunch of nested drop_in_place calls...>
#49 0x0000555557e07ff9 in core::ptr::drop_in_place<bevy_app::app::App> () at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/ptr/mod.rs:487
#50 0x00005555559fd334 in bevy_crash_app::main () at src/main.rs:4

I have no idea what's causing this, I suppose that 0.26.1 does something 0.26.0 didn't do before (or contains a bug of some sort). the segfault happens if you add winit 0.26.0 as a dependency too

I'm guessing winit is crashing because you have two versions of it since Bevy depends on 0.26.0 specifically.

@devil-ira Cargo will unify both versions into 0.26.1. In fact of you try to depend on two different semver conpatible versions using = to pin a specific version, cargo will throw an error.

That's indeed what happened. The situation came to existence, because I tried updating my dependencies to the latest versions (well, to the latest versions cargo would allow me to) and for some reason bevy doesn't feel well when the patch version is incremented

Right. That gets me every time. Nevermind me.

I suspect that you would see a similar crash with
[dependencies]
bevy = { "0.8", features = ["wayland"] }

Your own winit dep is dragging in that feature which bevy doesn't include by default.

I suspect that you would see a similar crash with
[dependencies]
bevy = { "0.8", features = ["wayland"] }

Your own winit dep is dragging in that feature which bevy doesn't include by default.

Yup! I have just tested this. Simply asking for wayland makes it segfault too

Please also include the AdapterInfo output from bevy (when it is not crashing). It looks like you might be on an intel GPU?

Please also include the AdapterInfo output from bevy (when it is not crashing). It looks like you might be on an intel GPU?

I indeed am. Including the adapter info into the report

0.8.1 + Wayland works here on a Raptor Lake booted into Ubuntu 22.10. Though I am manually depending on winit. Perhaps try adding something like the following and see if it works.
[dependencies.winit]
version = "0.26.0"
features = []
optional = false
default-features = false

@InnocentusLime is this resolved now?

@InnocentusLime is this resolved now?

@AxiomaticSemantics Have just downloaded bevy 0.9 on that same laptop after downloading last updates for the same OS (Ubuntu). The SEGFAULT is still happening.

Still unresolved damn happens on 0.10.1 too

I found with the same case:
// main.rs
use bevy::prelude::*;

fn main() {
  App::new().add_plugins(DefaultPlugins).update();
}

With package config like
#Cargo.toml
[package]
name = "bevy_crash"
version = "0.1.0"
edition = "2021"

[dependencies]
bevy = {version = "0.10.1", features = ["x11"]}

Or not specify any Bevy features:
#Cargo.toml
[package]
name = "bevy_crash"
version = "0.1.0"
edition = "2021"

[dependencies]
bevy = "0.10.1"

The app will also has a segment fault:
Finished dev [unoptimized + debuginfo] target(s) in 6.10s
     Running `/home/vitalyr/projects/dev/rust-projects/rust_play/target/debug/bevy_test`
2023-05-31T02:48:45.333538Z  INFO bevy_winit::system: Creating new window "Bevy App" (0v0)
2023-05-31T02:48:45.333787Z  INFO winit::platform_impl::platform::x11::window: Guessed window scale factor: 2
2023-05-31T02:48:45.591677Z  INFO bevy_render::renderer: AdapterInfo { name: "NVIDIA GeForce RTX 3060 Laptop GPU", vendor: 4318, device: 9568, device_type: DiscreteGpu, driver: "NVIDIA", driver_info: "530.41.03", backend: Vulkan }
2023-05-31T02:48:46.109752Z  INFO bevy_diagnostic::system_information_diagnostics_plugin::internal: SystemInfo { os: "Linux  Arch Linux", kernel: "6.3.5-zen1-1-zen", cpu: "12th Gen Intel(R) Core(TM) i7-12700H", core_count: "14", memory: "31.1 GiB" }
thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', /home/vitalyr/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bevy_render-0.10.1/src/pipelined_rendering.rs:135:84
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', /home/vitalyr/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bevy_tasks-0.10.1/src/task_pool.rs:376:49
thread '<unnamed>' panicked at 'cannot access a Thread Local Storage value during or after destruction: AccessError', /rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/std/src/thread/local.rs:246:26
fatal runtime error: failed to initiate panic, error 5
Aborted: oops, process 'cargo' core dumped

After changing App::new().add_plugins(DefaultPlugins).update(); to App::new().add_plugins(DefaultPlugins).run();, it runs well. This problem seems not related to Wayland or X11. It's a problem that Bevy can't run code like App::new().add_plugins(DefaultPlugins).update();. And indeed it's rare to write code like this.

@VitalyAnkh im pretty sure u were using xwayland to run the game
To enable native wayland support you need to add the waylandd feature wich then makes it segfault for me with App::new().add_plugins(DefaultPlugins).run()

@Tofixrs No, it doesn't segfault when enabling Bevy's wayland feature with App::new().add_plugins(DefaultPlugins).run(). Please watch my screen recording:
录屏 2023-06-02 20-08-10.webm

@VitalyAnkh guess its something with my system then. Sorry  for the false report

I got same problem, does someone know hot to fix it? @Tofixrs did you fix it?


No FPS with Android-One-Click-Deploy on Windows

Godot version
3.4.x
System information
Win 8.1, GLES 2.0, AMD HD7900 Series / Android 7, Huawei P8 lite 2017
Issue description
When running an Android game on the device via one-click-deploy the output console on the PC doesn't print out the fps anymore. This would work in 3.3 and 3.2 when activated in the project settings.
Steps to reproduce

go to project settings -> debug -> settings -> Stdout / Print Fps ON
install export templates, configure android export (debug keystore, android sdk, all that good stuff)
check one click deploy in android export settings
connect android device to PC via USB
download drivers when necessary
enable developer mode on android device
click on cute android button in the Godot "play project menu bar"

output window stays closed, doesn't show anything either

expected behavior: prints fps

Minimal reproduction project
No response

@ZeHipo  Can you reproduce this in any of the 3.4 betas and RCs to determine when the regression started?

will do

Hey @Calinou,
unfortunately I can't get fps output running at all anymore with one click deploy on Android. I tried every stable release from 3.2, which I had still installed on my PC, and none of them print the fps. I decided not to check the RCs for now.
I tested with the "Dodge The Creeps" demo and tried different editor, export and project settings on every version.
Possible issues on my end: maybe an Android sdk update broke it for me. When the fps output was working on my end I had an older sdk and tools installed but I don't remember the version. After testing for this issue I un-installed Android Studio and OpenSDK and reinstalled everything but it didn't help.
After re-installing the android sdk and setting it up in 3.4.2, I noticed that when opening 3.2 everything was already set up for me. That means new Godot installs overwrite some global files for every other Godot version? Maybe that makes it impossible for me to go back to the exact same setup from like half a year ago, too.
I use the older Android 7, but in the export settings min api level is set to 19 which is even earlier than that.

After re-installing the android sdk and setting it up in 3.4.2, I noticed that when opening 3.2 everything was already set up for me.

Editor settings are stored on per-major version basis. If you used Godot 3.2 in the past, upgraded to Godot 3.4.2 and downgraded to Godot 3.2 again, your old editor settings will still be present.


Field type browse_multiple is missing from doc?

Bug report
What I did: Tried searching for field type browse_multiple on the official doc.
What I expected to happen: Explanation of browse_multiple's usage
What happened: Found nothing.
What I've already tried to fix it: I tried to suggest an edit, but I'm not familiar with how it works.
Backpack, Laravel, PHP, DB version: -

Thank you @Dakaa - you're right, thank you. We haven't yet officially released the browse_multiple field type because it looks like it still has a few quirks. Yes, you can select multiple items, but it doesn't really show them after you click Save.
There's a little more work to be done there, before we can add it to the docs.
Thanks for the heads-up.
Cheers!


feat(types): add big_organization_join moderation type

[IRL discussion]

Seems like the big_organization_join value is deprecated


Question: Redirect loop between /login / and OIDC-Auth-Endpoint

Thanks for providing this nice component!
I struggle doing the configuration right and highly appreciate your support.
I end up in a redirect loop between the following resources in the mentioned order:
1. <airflow>/login
2. <OIDC/Keycloak>/auth/realms/airflow/protocol/openid-connect/auth
3. <airflow>/oidc_callback (sets the cookie: oidc_id_token)
4. <airflow>/login
5. <OIDC/Keycloak>/auth/realms/airflow/protocol/openid-connect/auth
6. <airflow>/oidc_callback
7...

My configuration is as follows:
airflow runs locally on http://localhost:8080
i use your image: quay.io/mojanalytics/airflow:1.10.3
webserver_config.py
[...]
AUTH_TYPE = AUTH_OID
AUTH_ROLE_ADMIN = 'Admin'
AUTH_USER_REGISTRATION = True
AUTH_USER_REGISTRATION_ROLE = "Admin"

OIDC_CLIENT_SECRETS = '/etc/airflow/client-configuration.json'
SECURITY_MANAGER_CLASS = AirflowOIDCSecurityManager
OIDC_USER_INFO_ENABLED = True
OIDC_SCOPES = ['openid', 'airflow']
OIDC_LOGOUT_URI = 'https://<domain>/auth/realms/airflow/protocol/openid-connect/logout?client_id=airflow&returnTo='
OIDC_CLOCK_SKEW: 560
OIDC_RESOURCE_CHECK_AUD: True
OIDC_INTROSPECTION_AUTH_METHOD: 'client_secret_post'

client-configuration.json
{
	"web": {
		"client_id": "airflow",
		"client_secret": "SECRET",
		"auth_uri": "https://<domain>/auth/realms/airflow/protocol/openid-connect/auth",
		"token_uri": "https://<domain>/auth/realms/airflow/protocol/openid-connect/token",
		"userinfo_uri": "https://<domain>/auth/realms/airflow/protocol/openid-connect/userinfo",
		"issuer": "https://<domain>/auth/realms/airflow",
		"redirect_uris": [
			"http://localhost:8080/*"
		]
	}
}

env vars for airflow.
Through the scope 'airflow', those attributes are put into access- & identify-token. I checked that.
export USERNAME_OIDC_FIELD="preferred_username"
export FIRST_NAME_OIDC_FIELD="nickname"
export LAST_NAME_OIDC_FIELD="name"

By modifying views.py, I find out, that handle_login seems not to be invoked.
At least added logger-invocations do not create logs:
def handle_login():
   log.info("start handle login")

Do you have a hint what to check?

The resource we access is via http.
It seems, as if OIDC_COOKIE_SECURE: False has no impact. The access_token is still marked as secure and not sent to airflow.
As this is confirmed, I will close this one.

Was my issue: OIDC_COOKIE_SECURE: False has had to be OIDC_COOKIE_SECURE= False
Sorry!


google-cloud-sdk: add libxcrypt-legacy

Description of changes

Things done

Built on platform(s)

[X] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[X] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Fixes the below build error:
error: builder for '/nix/store/59kj94x8mw61h22797q5iyjbd9vw3cms-bundled-python3-unix-linux-x86_64-3.9.12.drv' failed with exit code 1;
       last 10 log lines:
       > auto-patchelf: 4 dependencies could not be satisfied
       > error: auto-patchelf could not satisfy dependency libcrypt.so.1 wanted by /nix/store/bq6a2rvpqw2jh40ryjvghi64rzl0mjr5-bundled-python3-unix-linux-x86_64-3.9.12/google-cloud-sdk/platform/bundledpythonunix/lib/libpython3.9.so
       > error: auto-patchelf could not satisfy dependency libcrypt.so.1 wanted by /nix/store/bq6a2rvpqw2jh40ryjvghi64rzl0mjr5-bundled-python3-unix-linux-x86_64-3.9.12/google-cloud-sdk/platform/bundledpythonunix/lib/libpython3.9.so.1.0
       > error: auto-patchelf could not satisfy dependency libcrypt.so.1 wanted by /nix/store/bq6a2rvpqw2jh40ryjvghi64rzl0mjr5-bundled-python3-unix-linux-x86_64-3.9.12/google-cloud-sdk/platform/bundledpythonunix/bin/python3
       > error: auto-patchelf could not satisfy dependency libcrypt.so.1 wanted by /nix/store/bq6a2rvpqw2jh40ryjvghi64rzl0mjr5-bundled-python3-unix-linux-x86_64-3.9.12/google-cloud-sdk/platform/bundledpythonunix/bin/python3.9
       > auto-patchelf failed to find all the required dependencies.
       > Add the missing dependencies to --libs or use `--ignore-missing="foo.so.1 bar.so etc.so"`.
       > /nix/store/sw36plhp82916wwg6i6097rkzza7d950-stdenv-linux/setup: line 79: pop_var_context: head of shell_variables not a function context
       > /nix/store/sw36plhp82916wwg6i6097rkzza7d950-stdenv-linux/setup: line 1457: pop_var_context: head of shell_variables not a function context
       > /nix/store/sw36plhp82916wwg6i6097rkzza7d950-stdenv-linux/setup: line 1594: pop_var_context: head of shell_variables not a function context
       For full logs, run 'nix log /nix/store/59kj94x8mw61h22797q5iyjbd9vw3cms-bundled-python3-unix-linux-x86_64-3.9.12.drv'.
error: 1 dependencies of derivation '/nix/store/w90v449mcbj95slc0qs7bl7kg7hclfzc-google-cloud-sdk-408.0.1.drv' failed to build

Seems that this missed notifying the maintainers possibly due to where the change was made.
Pinging maintainers manually: @pradyuman @stephenmw @zimbatm @iAmMrinal0


Dev/autotest nic emu v2

new autotest interface types: vmux-emu and bridge-e1000
autotest improvements: magic variables, reliability, ...
trivial vmux-emu/e1000 optimizations: enable compiler optimizations, link statically, suppress logging, drop less packets

merges parts of #99 and #97


feat: 🎸 scroll to cloned/added widget

https://metakeen.atlassian.net/browse/FEE-702

Codecov Report

Merging #349 (325de42) into develop (2ffa1d8) will decrease coverage by 0.02%.
The diff coverage is 76.92%.

@@             Coverage Diff             @@
##           develop     #349      +/-   ##
===========================================
- Coverage    86.91%   86.89%   -0.03%
===========================================
  Files          550      551       +1
  Lines         7094     7106      +12
  Branches       721      722       +1
===========================================
+ Hits          6166     6175       +9
- Misses         842      845       +3
  Partials        86       86

Impacted Files
Coverage Δ

src/components/Grid/Grid.tsx
23.72% <ø> (ø)

src/utils/scrollItemIntoView.ts
25.00% <25.00%> (ø)

src/modules/dashboards/dashboardsSaga.ts
52.84% <100.00%> (+0.49%)
:arrow_up:

src/modules/widgets/widgetsSaga.ts
70.12% <100.00%> (+0.79%)
:arrow_up:

src/utils/index.ts
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2ffa1d8...325de42. Read the comment docs.


Replace std::maps with a better implementation

Original issue: https://charm.cs.illinois.edu/redmine/issues/1064

During the 64bit merge, some one-off data structures (such as CkMigratableList) were deleted and replaced by std::map, however the use of std::map was only meant to be temporary (see https://charm.cs.illinois.edu/gerrit/gitweb?p=charm.git;a=commit;h=1791cb5e77c580b4f8c9b67ecbd76a144e4c2cbc for the original commit message).
A better implementation should be found, but one that also works on all of the platforms we care about (so std::unordered_map is out for now due to lack of support on xlc).

Original date: 2016-05-11 21:14:59

Related: https://charm.cs.illinois.edu/redmine/issues/185

Original author: Vipul Harsh
Original date: 2016-05-12 01:44:21

Performance comparisons of map, unordered_map and ckhashtable: https://charm.cs.illinois.edu/private/tms/listlog.php?param=449

Original date: 2016-05-12 18:50:01

It looks like unordered_map is the clear winner in just about every case. Sam suggested using a typedef to use unordered map everywhere it is supported, and typedef to std::map everywhere else (mainly just XLC I think)

Original date: 2016-10-03 22:56:45

Even just doing the typedef for map vs unordered_map would be worth doing soon

Original date: 2016-10-05 19:49:29

Yes, please add that typedef. Note that we'll still need to use old-style typedef syntax instead of modern using syntax, because the case that needs the alternative is IBM XL on Blue Gene, and that doesn't support the new syntax.

Original date: 2016-10-27 23:13:11

Ping. The configure test and consequent typedef for this don't seem like a big deal. Is there some hidden complexity that I'm missing here?

Original date: 2016-11-02 22:39:39

I just missed it being assigned to me. Working on it now. The only "complexity" is that without 'using' we will can't just typedef Map to either std::map or std::unordered_map based on platform. We will need to do a separate typedef for each map in the location manager because they all have different template arguments.

Original date: 2016-11-23 19:15:20

~~https://charm.cs.illinois.edu/gerrit/1978~~ https://github.com/UIUC-PPL/charm/commit/ac84e2be6f63d892bf3aa9a576a16bbc27624db2
I'll open a follow-up bug about resolving which XLC versions/platforms we should be constraining to the slower ordered structures.

Original date: 2016-12-07 16:35:56

Use tr1/unordered_map instead of stl map on BG XLC: ~~https://charm.cs.illinois.edu/gerrit/#/c/2013/~~ https://github.com/UIUC-PPL/charm/commit/806b3277cac6c0e37864c8339e1220a0770a1b54


ScrollSpy offset error

When I use the scroll, is offset by cutting the description of the methods or locating the focus incorrectly selected method.
In this example, when I do click in closeTransaction, cut description.
Can you help with this problem?

This is fixed in 0.5.3 and possibly before.


Currently incompatible with scikit grid search functionality

Integrate hi_lasso into scikit framework fully

Dropping hi_lasso dev, higher priorities are CpG selection and results pipeline dev.


feat: Emergency stop command

This PR includes a ROS2 command to the UR robot that rotates the base of the robot by 90 degrees in its upright position. While the robot in motion, pressing the emergency stop button on the teach pendent will halt the robot.
To reproduce the test sequence follow the commands on the README file on separate terminals . Note, this sequence utilizes the ROS2 package pdf_beamtime. While the robot in motion, press the emergency stop button on the teach pendent.
Expected robot behavior

Robot stops its motion.
The container which executes the pdf_beamtime_server prints the following,

[INFO] [1712338080.639018459] [moveit_robot_model.robot_model]: Loading robot model 'ur'...
[WARN] [1712338080.838124466] [moveit_ros.robot_model_loader]: No kinematics plugins defined. Fill and load kinematics.yaml!
[INFO] [1712338080.882062399] [move_group_interface]: Ready to take commands for planning group ur_arm.
[INFO] [1712338155.966023467] [pdf_beamtime_server]: Current state is HOME.
[INFO] [1712338155.966328446] [pdf_beamtime_server]: Internal state changed from RESTING to RESTING
[INFO] [1712338155.966366801] [pdf_beamtime_server]: Executing state HOME
[INFO] [1712338155.966664882] [move_group_interface]: MoveGroup action client/server ready
[INFO] [1712338155.970541027] [move_group_interface]: Planning request accepted
[INFO] [1712338156.103814386] [move_group_interface]: Planning request complete!
[INFO] [1712338156.104875880] [move_group_interface]: time taken to generate plan: 0.0199934 seconds
[INFO] [1712338156.105087145] [pdf_beamtime_server]: Internal state changed from RESTING to MOVING
[INFO] [1712338156.107853976] [move_group_interface]: Execute request accepted
[INFO] [1712338163.759052632] [move_group_interface]: Execute request aborted
[ERROR] [1712338163.759100464] [move_group_interface]: MoveGroupInterface::execute() failed or timeout reached
[ERROR] [1712338163.759601254] [pdf_beamtime_server]: Goal aborted !

This is great. Can you change the ghcr.io/chandimafernando/erobs-common-img:latest to be an environment variable in the script that you reference. This will make changing it downstream easier with 1 line.

ghcr.io/chandimafernando/erobs-common-img:latest is moved to an environmental variable, GHCR_POINTER. Ref: README


Add GitHub action to parse and check TechEvents.csv

Closing as no longer using .csv to store events


BOOT_IMAGE not set for /proc/cmdline

When using recent zipl for booting an up-to-date Fedora Rawhide system (uses BLS boot entries), I don't see the BOOT_IMAGE variable being set by the bootloader, so it doesn't appear in /proc/cmdline. We get the same behaviour with RHEL-8.1 (work in progress, see bug 1728677) so I suspect the secure boot patches might break it. Looking further ...

Where did you IPL from? DASD or SCSI? Was secure boot enabled?

DASD in z/VM guest, secureboot not used

Thanks, I will have a look.

Looks like stage 2 overwrites the extra_param when loading the stage 3 parameter page. I need to shuffle around the memory area used there.


chore: updates example for latest stack schema

what

I'm starting to POC Spacelift and this module for a client, so figured I'd get the following updated:

Updates stack example to utilize all spacelift.setting + explains them. Utilizes the correct location as well.
Updates surrounding documentation to talk about stacks more and link out to our docs.
Provides some context around imports + globals.yaml
Fixes usage of components.globals to utilize newer top-level vars

why

Keeps documentation up-to-date

/test all

/test all

@Nuru yeah I don't like that we have to define stack configurations in multiple locations, so I've opened https://github.com/cloudposse/docs/pull/536 to deal with that.

@osterman I actually want to close this PR in favor of a new one after https://github.com/cloudposse/docs/pull/536 gets merged. Let me know your thoughts in that PR.

Closing as https://github.com/cloudposse/docs/pull/536 will supersede. I'll put up a follow up PR after that merges.


Set up linting

We should really set up linting e.g. credo with some sensible configuration for the project.
I believe it makes contributions easier as contributors don't have to fix style problems, which also makes reviews easier and also it makes the whole database more consistent.

I completely agree linting would be awesome. It will just take quite a bit of work to fix all of the style issues in the codebase. I wonder if there's a way to ease into using credo without failing all of our builds until we fix all of the style issue? I know https://www.houndci.com takes the approach of only reviewing pull requests so you don't have to update the entire app at once. I wonder if https://ebertapp.io has a similar approach.

ha, good point we could use ebert!
But even without it, you can configure credo with todos so that you already enable a couple of rules and put the others into TODO basically and work on them PR by PR. We did that in one of our projects and it works quite well. Builds absolutely shouldn't be failing :)


Change translateY to 3d

Short description of what this resolves:
Add translate3d instead translateY in refresherController.js
Ionic Version: 1.x
Fixes: #6016

Hi @vitaliy-bobrov,
Thanks for the PR.  Nice work!  Thanks for contributing to Ionic.
Thanks,
Dan


⚠ Bofrost.at has degraded performance

In f36116b, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 2051 ms

Resolved: Bofrost.at performance has improved in 8c0492a after 2 hours, 14 minutes.


feature request: add 'EnforcePartLimits=ALL' to slurm.conf

Currently, if a slurm job requests too many nodes (i.e., more than max_queue_size), it's accepted but sits in the queue and never runs.
Adding EnforcePartLimits=ALL to /opt/slurm/etc/slurm.conf causes the job to be rejected at submission. For example, if max_queue_size=2:
$ sbatch -N 3 job.sh
sbatch: error: Batch job submission failed: Node count specification invalid

Hi @nolta
thanks for proposing this valid enhancement. I think it's reasonable for us to add this to slurm default config. In the meanwhile feel free to modify your cluster slurm config and add that option to it.

coming in next release https://github.com/aws/aws-parallelcluster-cookbook/pull/410


refine the mount file function - [closed]

In GitLab by @sokril on Jan 25, 2019, 08:08
Merges mountfile -> dev

In GitLab by @wknet123 on Nov 13, 2019, 11:37
The test coverage for backend is %  38.96%  , check  console log

In GitLab by @wknet123 on Nov 13, 2019, 11:37
closed


Revert "Utils module fixed."

Reverts XanaduAI/strawberryfields#69

I'm a bit confused about what the status is here, and what any expected actions are. @josh146 ?

I clicked the 'revert merge' button to see what will happen, turns out the result is pretty silly (an inverse PR). So I closed it. Feel free to ignore.


Add MMap - High Cardinality Map on Memcached

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

136 of 147 (92.52%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.3%) to 96.35%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

mmap/mmap.go
55
66
83.33%

Totals

Change from base Build<PHONE_NUMBER>:
-0.3%

Covered Lines:
1795

Relevant Lines:
1863

💛  - Coveralls


add run id to api responses

The field_current_run id needs to be exposed in the api in the following responses

[ ] get signups
[x] get reportbacks
[x] get campaigns

@angaither Would the current_run for a reportback actually be the current run or the run it's actually associated with?

whaaa

If you're doing a GET request for /reportbacks what does "current_run" mean in that context? Wouldn't it just be "campaign_run" that the reportback corresponds to? I don't think it has anything to do with what the current run is... the only place that matters with a reportback is when submitting one and needing to apply the current run id to that uploaded file.

you are correct good sir!

this can be closed now, right?

Not yet. I need to update the old GET signups endpoint

working on it...

@aaronschachter is the GET /signups endpoint used anywhere? I can't find any documentation on it in the API wiki, so not sure what parameters it's implying I can pass, but all I get is:
[
  "Missing required argument parameters"
]

when I hit /api/v1/signups

Kill it with fire. It's code I added before we were using Northstar for the mobile app for Signup Groups, which we've since abandoned:  https://github.com/DoSomething/phoenix/pull/4243

@angaither I think we can close this one now. The only signup related stuff w/ run_nid happens in /campaigns/:id/signup and the run_nid stuff accounted for in the function called dosomething_signup_create() from within the endpoint callback :)


Switching from comma to % so log messages are displayed correctly in a GUI context.

Log output was correct on the command line but displayed '%' characters instead of formatting the arguments with a custom logging Handler object.

no, the current code is correct. We only want to evaluate the string interpolation if the log message is actually handled, that's why we don't do "%" immediately in the string literal, but use the comma.
That's a feature of logging.
You have to use the LogRecord.getMessage() method to get the full message string for a LogRecord instance after merging any arguments following the comma.
https://docs.python.org/3.6/library/logging.html#logging.LogRecord.getMessage
What are you trying to do with your custom logging Handler?

@anthrotype Right, my bad, I'm new to logging. I'm displaying the various log levels in different colors in a TextField. Thanks!


Preview doesn't show an entire component with vite

Steps to reproduce
Use test project: Uploading vite-test-tb.zip… and run preview for Boot.tsx
or do preconditions:

create vite project on type script
add toolbox, antd palette
create new component
run preview
add many components from palette to custom component

AR:
component is not visible entirely on a preview. Top part is absent. Restarting dev server doesn't help

Note that i can't reproduce this with cra-project

It is not ide-toolbox problem, it is incorrect vite template.
Removing place-items: center; from index.css helps to aviod the problem


add nightly binary builds

for beta testers / users who want to live at head, we could add a nightly binary release of the master branch:

nightly travis CI job
check out master head commit
tag it as nightly

make sure to remove any previous nightly tag
see https://stackoverflow.com/a/8044605/6465472
also see https://docs.travis-ci.com/user/deployment-v2/providers/releases/#setting-the-tag-at-deployment-time

deploy to github releases with overwrite: true
ensure this doesn't interfere with our links to lastest release being the latest tagged version release

actually, we probably don't need a nightly CI build, can instead reuse the build on master that happens when a PR is merged.
this build could add a latest tag and create a release as above
then no extra nightly CI required, and latest binary release is always at head of master


Source code?

Do you plan to release the source code for this project or is it encumbered?
I just installed a RAK19007 + RAK4631 digi for test and I am curious what is inside the firmware.

sure I will its nothing that crazy at all but as it is UF2 file and is flashed in another way I guessed no-one wanted it :P

I would like to see if it is possible to add wx and external voltage measurements to it. It would be perfect for some of my solar digi sites.

Ah, didn't realize that RAK made a module with BME860 - it's nice and integrated, but on the other hand a BME280 over I2C is like 5 times cheaper :) Whichever is easier to implement I guess.
External voltage with a voltage divider. I did some research and looks like there are two ADC on RAK4631, one is connected to Vcc rail (over a divider) and could be used as int voltage, the other one is free and we could use it for ext voltage (via an external divider).

Does this FW support the OLED display over I2C?  I would have checked the source but it it is not published yet :)

not yet, but will
been finishing details on the T-echo to have a beta testing FW and I will receive in the meantime new OLED screen to test also and validate


Clean up Sentry user accounts

Background
Users have not been removed from Sentry in the past as part of the leavers process, have gone through all the users, checked their status in slack and created a list of users that have left the organisation. These users have been removed from Sentry and will check they are not on our other tools as well. Have divided the work three ways with Sam and Connor. Also went through the Leaving process with Sam and Connor as part of this ticket. Emailed a group of users whose username and email were obscure, ie couldn't find their account / status on Slack. Some emails bounced straight away, some have asked to be removed and some have asked to keep their access.

Removed:
x47 Sentry (Me / Sam / Connor)
x7 Docker (Me / Sam / Connor)
x1 PagerDuty
x1 Auth0
x1 Pingdom
Sam and Connor got an intro into the process. Connor created a spread sheet that will help the leaving process.
This ticket has shown that the other tools are being well maintained via audits, leavers process and automation. It was mainly Sentry that wasn't being updated. The team know now to check Sentry more when users leave.


User-reported unhelpful error message

Program: https://code.world/#P8xFuYr3ejUpwpcXFgNEc_Q
Message:
program.hs:1:1-12: error:
    Parse error: An equation was expected here.
    Are you missing the equal sign?

The message is exactly correct.

The message is exactly correct.


NLU correctly extracts intent, does not pass to slot via story

Rasa version:
Core: 0.13.7
Core-sdk: 0.13.0
NLU: 0.14.6
Python version:  3.6
Operating system:  Ubuntu 16.04
Issue: NLU correctly identifying intent, not passing to slot via story
Content of configuration file (config.yml):
language: "en"
pipeline:
- name: "nlp_spacy"
- name: "tokenizer_spacy"
- name: "intent_entity_featurizer_regex"
- name: "intent_featurizer_spacy"
- name: "ner_crf"
- name: "intent_featurizer_count_vectors"
- name: "intent_classifier_tensorflow_embedding"
- name: "ner_duckling_http"
  url:  "http://<IP_ADDRESS>:8000"
  dimensions: ["time"]

Content of domain file (domain.yml) (if used & relevant):
intents:
  - greet
  - goodbye
  - get_quote_withinfo
  - get_quote
  - modify_booking
  - thanks
  - inform_city
  - inform_date
  - check_status

actions:
- utter_greet
- utter_goodbye
- utter_welcome
- utter_get_quote
- utter_slot_values
- utter_submit
- utter_ask_source # should this not be included by default, will i be overriding shit?
- utter_changebook
- utter_check_status

entities:
  - source
  - destination
  - day
  - month
  - year
  - time_day
  - new_day
  - new_month
  - date
  - city
  - time

slots:
  source:
    type: unfeaturized
    auto_fill: false
  destination:
    type: unfeaturized
    auto_fill: false
  date:
    type: unfeaturized
    auto_fill: false
  day:
    type: unfeaturized
    auto_fill: false
  month:
    type: unfeaturized
    auto_fill: false
  year:
    type: unfeaturized
    auto_fill: false
  time_day:
    type: unfeaturized
    auto_fill: false

templates:
  utter_greet:
  - text: "Hello! How can I help?"
  - text: "Hiya!"
  utter_goodbye:
  - text: "Goodbye!"
  utter_ask_source:
    - text: "Where are you shipping from?"
  utter_ask_destination:
    - text: "Where are you shipping to?"
  utter_ask_date:
  - text: "When do you want your items to ship from {source}?"
  utter_welcome:
    - text: "You're welcome!"
    - text: "No worries :)"
  utter_get_quote:
    - text: "Okay, lets get a quote going"
  utter_changebook:
    - text: "Okay, let me grab that booking for you"
  utter_slot_values:
    - text: "I am going to run a quote search using the following parameters:\n
            - source: {source}\n
            - destination: {destination}\n
            - date: {day} of {month}\n
            - time: {time_day}"
  utter_submit:
    - text: "All done!"
  utter_default:
    - text: "Totes soz, I don't understand what you mean there, can you rephrase please?"
  utter_ask_month:
    - text: "What day of {year}"
  utter_ask_day:
    - text: "What day of {month}"
  utter_ask_time_day:
    - text: "What time on {day}"

forms:
  - booking_form

**Content of stories.md
## new_quote2
* greet
  - utter_greet
* get_quote{"source":"perth"}
  - utter_get_quote
  - booking_form
  - form{"name": "booking_form"}
  - form{"name": null}
  - utter_slot_values
* goodbye
  - utter_goodbye

I am trying to make a fairly standard variation of the form bot which will make bookings. If the user writes 'make booking' and starts the form then everything works fine. However, I would like to be able to extract information from the very first sentence before an intent has been classified (i.e. before a form is used). It seems this is done via slot filling directly in the stories but i cannot get it to work. In the attached image it shows that NLU is correctly pulling the source city (though admittedly low confidence but that's fine) but the form is still asking for it.
I have tried several variations (i.e. putting the slot filling part in the story after the form is activated) but nothing seems to work, what am i missing?
Thanks!

Thanks for raising this issue, @Ghostvv  will get back to you about it soon.

what version of rasa_core_sdk are you using? it didn't validate already filled slots. We updated forms to add this functionality: https://github.com/RasaHQ/rasa_core_sdk/pull/67
Could you please try a new version

Hello,
The versions I used for that example are listed at the top of the post. I did a pip install --upgrade and am now running
rasa-core                     0.13.8
rasa-core-sdk                 0.13.1
rasa-nlu                      0.14.6
but the issue persists. NLU correctly pulls the entity but the slot remains unfilled. Is there a particular version i should specifically request through pip? I have attached another image showing the same thing happening.
Thanks!

Update: okay I think I have a better idea of my confusion, and I have made a more compact example to illustrate this.
Let's say I have two slots to fill, pickup_city and pickup_street. I have two entities of the same name, so I can extract them directly, but I also want to be able to extract them from a more general intent 'inform_address' which may be used for pickup but also for dropoff. So, for this intent i have two other entities just 'street' and 'city.
What I thought would happen is if i have the following form definition:
` @staticmethod
def required_slots(tracker: Tracker) -> List[Text]:
return ["pickup_city", "pickup_street"]
 def slot_mappings(self):
			return {
				"pickup_city": [ self.from_entity(entity="pickup_city", intent = ["make_booking"]),self.from_entity(entity="city",intent=["inform"])],
				"pickup_street": [self.from_entity(entity="pickup_street", intent = ["make_booking"]),self.from_entity(entity="street", intent = ["inform"])]
			}`

that if the form asked for 'pickup_city' but I gave 'city' and 'street' (as correctly extracted by NLU), then the slot mapping would not only give me pickup_city but also see that it extracted 'street' and map that to 'pickup_street'. However, this does not happen: it does not see that it can fill a slot it knows about by using an entity it has just seen.
For example, if I change it so the required slot it 'street' then this works: I give it a city and a street at the same time and it sees 'street' as the extra slot to fill and does so (though here it would not work because i have not defined 'street' as a slot but i can see it is trying to do the right thing anyway)
So, while I understand i could get around this by having a custom 'validate' function, am I wrong in thinking this sort of thing should happen by default?
Thanks again!

Do you have another action or utterance between a user input and a form? By default, validation is triggered only if the form is called after action_listen

I did have that, though if I remove it i still get the same thing. I have changed the setup quite a bit since the last round, sorry, so here is the new setup.
## intent:get_quote
- get quote
- make quote
- hello, i want to get a quote
- hi i need a quote
- hey i want to find out how much it will cost
- how much will it cost to send stuff from [bayswater](pickup_suburb) to [castlemaine](dropoff_suburb)?
- i want to send some items
- i want to ship some freight
- i want to ship items to [moonah](dropoff_suburb)
- i want to send items to [mildura](dropoff_suburb)
- i need to ship to [petersham](dropoff_suburb)
- i want to send some items from [brisbane](pickup_suburb)
- i want to ship some freight [rye](pickup_suburb)
- i want to send items from [petersham](pickup_suburb) to [brisbane](dropoff_suburb), how much will it cost?
- i need a quote on sending from [new town](pickup_suburb) to [ultimo](dropoff_suburb)
- i want to send items to [petersham](dropoff_suburb) from [brisbane](pickup_suburb), how much will it cost?
- i need a quote on sending to [new town](dropoff_suburb) from [ultimo](pickup_suburb)
- how much will it cost to send stuff to [bayswater](dropoff_suburb) from [castlemaine](pickup_suburb)?

## new_quote1
- utter_intro
* get_quote
  - pickup_address_form
  - form{"name": "pickup_address_form"}
  - form{"name": null}
  - dropoff_address_form
  - form{"name": "dropoff_address_form"}
  - form{"name": null}
  - items_form
  - form{"name": "items_form"}
  - form{"name": null}
  - utter_return_quote

 @staticmethod
 def required_slots(tracker: Tracker) -> List[Text]:
 	return ["pickup_suburb", "pickup_postcode", "pickup_residential"]

 def slot_mappings(self):
	return {
		"pickup_suburb": [ self.from_entity(entity="pickup_suburb", intent = ["make_booking"]),self.from_entity(entity="suburb",intent=["inform_address"])],
		"pickup_postcode": [self.from_entity(entity="pickup_postcode", intent = ["make_booking"]), self.from_entity(entity="postcode", intent = ["inform_address"]), self.from_entity(entity="number", intent = ["inform_address"])],
		"pickup_residential": [ self.from_intent(intent='affirm', value=True), self.from_intent(intent='deny', value=False)]
	}

 # NOTE: this is infact called for each slot at time of filling, not at the end
 def validate(self, dispatcher: CollectingDispatcher, tracker: Tracker, domain: Dict[Text, Any]) -> List[Dict]:

	slot_values = self.extract_other_slots(dispatcher, tracker, domain)
	slot_to_fill = tracker.get_slot(REQUESTED_SLOT)
	if slot_to_fill:
		slot_values.update(self.extract_requested_slot(dispatcher, tracker, domain))
		if not slot_values:
			raise ActionExecutionRejection(self.name(), "Failed to validate slot {0} "
														"with action {1}"
															"".format(slot_to_fill,self.name())
															)

	latest_message = tracker.latest_message['entities']
	updated_slots = {}

	for el in latest_message:
		value = el['value']
		if el['entity'] == 'postcode':
			updated_slots['pickup_postcode'] = value
		elif el['entity'] == 'suburb':
			updated_slots['pickup_suburb'] = value

	for slot, value in updated_slots.items():
		slot_values[slot] = value

	return [SlotSet(slot, value) for slot, value in slot_values.items()]

I have attached an image of the output. Is it possible because i have overwritten the validate() function it is no longer looking for other prefilled stuff? The Core output shows NLU extracting the pickup_suburb correctly but it still does not apply it to the appropriate slot

Thanks!

A thousand apologies, I had changed one of my intents and failed to update my actions.py
For now this seems to work swimmingly so I am happy to close the issue, however it would still be good to know how to utter a message at the start of a form without losing the slot filling aspect.

there is _activate_if_required method, which you can copy and add utterance upon activation there.
there is slot mapping from_trigger_intent that should fill the slot upon form activation depending on user intent that triggered the form

however it would still be good to know how to utter a message at the start of a form without losing the slot filling aspect. - we thinking about the most general way to implement it

Ah okay great, will play with that. For now I just have a custom action
class FillAll(Action):
	def name(self):
		return "action_fillall"

	def run(self, dispatcher, tracker, domain):
		latest_message = tracker.latest_message
		entities = latest_message['entities']
		slots = tracker.slots.keys()

		result = []
		for el in entities:
			if el['entity'] in slots:
				result.append(SlotSet(el['entity'], el['value']))
		return result

which has zero error handling and is lazy but seems to work.
Regarding 'best way' ideally, at least from myself as a user point of view, something akin to the submit() method, basically an init(), would be great.

Hello,
I'm afraid i need to re-open this issue. I went back to a previous version of forms/stories and have come up against this issue again, though it may be me expecting something that isn't meant to happen. Note I have re-read the above stuff to make sure i'm not re-hashing old solutions.
Problem is simple: given the following FormAction (leaving validate() un-overloaded)
class QuoteForm(FormAction):

    def name(self):
        return "quote_form"

    @staticmethod
    def required_slots(tracker: Tracker) -> List[Text]:
        return ["pickup_suburb", "pickup_postcode"  ]

    def slot_mappings(self):
        return {
            "pickup_suburb": [
                self.from_entity(entity="pickup_suburb", intent = ["quote"]),
                self.from_entity(entity="suburb",intent=["inform_address"])],
            "pickup_postcode": [
                self.from_entity(entity="postcode", intent = ["inform_address"]),
                self.from_entity(entity="pickup_postcode", intent = ["quote"]),
                self.from_entity(entity="number", intent = ["inform_number"])]

and story
## quote
* greet
  - utter_greet
* quote
  - quote_form
  - form{"name":"quote_form"}
  - form{"name": null}

I want to be able to respond to suburb and postcode at the same time when suburb is asked for. Specifically, when it asks for suburb  it actually asks for suburb and postcode, so the user has the option to enter both at the same time. I had assumed that the mappings defined above would mean that it would get the postcode even though only suburb was asked for, is that not the case?
An example response i am trying to parse when asked for suburb is " perth 2000" with intent inform_address
I can put a whole bunch of things in validate to deal with this, but it would involve lots of intent checking which should be handled already by the slot mappings (I will have pickup/dropoff and so on)
Thanks!
Z.

Update: Okay I think i have answered my own question, apologies for the threadnaught.
I managed to hack my way to what i wanted but turns out what i wanted isn't what i needed, as i end up overriding slots ahead of time. I think i saw mention of 'groups' being maybe implemented in the future which may solve this problem.
For now I think I  will just define sets of slots that may be set together and others which can only be set when asked for.
Thanks!

so can we close this issue again?

Yes, my apologies, thanks!


30 sec timeout for login with --mode gui parameter

Navigation timeout exceeded 30 000 ms
Is it possible to modify the azure ad pop up login screen timeout, because 30 sec timeout is not so user friendly?  60 sec timeout would be nice

I'm sorry, which timeout are you talking about?

Oh, I'm guessing this is a dup of https://github.com/dtjohnson/aws-azure-login/issues/20

Should be fixed in v1.3.0.


[SUPPORT]When hive queries Hudi data, the query path is wrong

The created hudi table table structure:
CREATE EXTERNAL TABLE guhudi_ro(
_hoodie_commit_time string COMMENT '',
_hoodie_commit_seqno string COMMENT '',
_hoodie_record_key string COMMENT '',
_hoodie_partition_path string COMMENT '',
_hoodie_file_name string COMMENT '',
id bigint COMMENT '',
name string COMMENT '',
birthday bigint COMMENT '',
ts bigint COMMENT '')
PARTITIONED BY (
partition string COMMENT '')
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
WITH SERDEPROPERTIES (
'hoodie.query.as.ro.table'='true',
'path'='hdfs://paat-dev/user/hudi/guhudi')
STORED AS INPUTFORMAT
'org.apache.hudi.hadoop.HoodieParquetInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION
'hdfs://paat-dev/user/hudi/guhudi'
TBLPROPERTIES (
'last_commit_time_sync'='20220114154921888',
'spark.sql.sources.provider'='hudi',
'spark.sql.sources.schema.numPartCols'='1',
'spark.sql.sources.schema.numParts'='1',
'spark.sql.sources.schema.part.0'='{"type":"struct","fields":[{"name":"_hoodie_commit_time","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_commit_seqno","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_record_key","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_partition_path","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_file_name","type":"string","nullable":true,"metadata":{}},{"name":"id","type":"long","nullable":false,"metadata":{}},{"name":"name","type":"string","nullable":true,"metadata":{}},{"name":"birthday","type":"timestamp","nullable":true,"metadata":{}},{"name":"ts","type":"timestamp","nullable":true,"metadata":{}},{"name":"partition","type":"string","nullable":true,"metadata":{}}]}',
'spark.sql.sources.schema.partCol.0'='partition',
'transient_lastDdlTime'='1642146555')
But when I go to query this table, he will go to the xxxx (below) path to find it, so that the data cannot be queried. Why?
hive> select count(1) from guhudi_ro;
Query ID = root_20220114155139_4d8fae2e-3b06-4bdb-94c6-d4da9cdd22c6
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=
In order to set a constant number of reducers:
set mapreduce.job.reduces=
22/01/14 15:51:39 INFO client.RMProxy: Connecting to ResourceManager at lo-t-bd-nn/<IP_ADDRESS>:8032
22/01/14 15:51:39 INFO client.RMProxy: Connecting to ResourceManager at lo-t-bd-nn/<IP_ADDRESS>:8032
22/01/14 15:51:39 INFO hive.HoodieCombineHiveInputFormat: Total number of paths: 1, launching 1 threads to check non-combinable ones.
22/01/14 15:51:39 INFO hive.HoodieCombineHiveInputFormat: Input Format => org.apache.hudi.hadoop.HoodieParquetInputFormat
22/01/14 15:51:39 INFO hive.HoodieCombineHiveInputFormat: CombineHiveInputSplit creating pool for hdfs://paat-dev/tmp/hive/root/b13bb5f0-aabd-4bac-9d72-596b53cfd41a/hive_2022-01-14_15-51-39_382_6318212808265460380-1/-mr-10003/fb4a19a7-08a2-436d-8af3-eab9a59b7be7; using filter path hdfs://paat-dev/tmp/hive/root/b13bb5f0-aabd-4bac-9d72-596b53cfd41a/hive_2022-01-14_15-51-39_382_6318212808265460380-1/-mr-10003/fb4a19a7-08a2-436d-8af3-eab9a59b7be7
22/01/14 15:51:39 INFO hive.HoodieCombineHiveInputFormat: mapreduce.input.fileinputformat.split.minsize=1, mapreduce.input.fileinputformat.split.maxsize=256000000
22/01/14 15:51:39 INFO hive.HoodieCombineHiveInputFormat: Listing status in HoodieCombineHiveInputFormat.HoodieCombineFileInputFormatShim
22/01/14 15:51:39 INFO hive.HoodieCombineHiveInputFormat: Using HoodieInputFormat
22/01/14 15:51:39 INFO utils.HoodieInputFormatUtils: Reading hoodie metadata from path hdfs://paat-dev/tmp/hive/root/b13bb5f0-aabd-4bac-9d72-596b53cfd41a
22/01/14 15:51:39 INFO table.HoodieTableMetaClient: Loading HoodieTableMetaClient from hdfs://paat-dev/tmp/hive/root/b13bb5f0-aabd-4bac-9d72-596b53cfd41a
22/01/14 15:51:39 INFO hadoop.InputPathHandler: Handling a non-hoodie path hdfs://paat-dev/tmp/hive/root/b13bb5f0-aabd-4bac-9d72-596b53cfd41a/hive_2022-01-14_15-51-39_382_6318212808265460380-1/-mr-10003/fb4a19a7-08a2-436d-8af3-eab9a59b7be7
22/01/14 15:51:39 INFO hive.HoodieCombineHiveInputFormat: number of splits 1
22/01/14 15:51:39 INFO hive.HoodieCombineHiveInputFormat: Number of all splits 1
Starting Job = job_1642128573447_0025, Tracking URL = http://lo-t-bd-nn:8088/proxy/application_1642128573447_0025/
Kill Command = /opt/cloudera/parcels/CDH/lib/hadoop/bin/hadoop job  -kill job_1642128573447_0025
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2022-01-14 15:51:44,886 Stage-1 map = 0%,  reduce = 0%
2022-01-14 15:51:48,953 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.18 sec
2022-01-14 15:51:53,011 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 3.65 sec
MapReduce Total cumulative CPU time: 3 seconds 650 msec
Ended Job = job_1642128573447_0025
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1  Reduce: 1   Cumulative CPU: 3.65 sec   HDFS Read: 11212 HDFS Write: 101 HDFS EC Read: 0 SUCCESS
Total MapReduce CPU Time Spent: 3 seconds 650 msec

@gubinjie we need more info about your environment setup to analyze and reproduce. Can you add info like hudi version, environment, other software versions, how table was prepared, what is the reproducing steps, etc.

Version list:
Flink: 1.13.3
CDH:6.3.2
Hive: 2.1.1-cdh6.3.2 rb3393cf499504df1d2a12d34b4285e5d0c02be11
Hudi: 0.10.0-rc2
The Jar package compiled by myself is:
hudi-hadoop-mr-bundle-0.10.0-rc2.jar
hudi-hive-sync-bundle-0.10.0-rc2.jar
Flink startup method:
./yarn-session.sh -s 2 -jm 4048 -tm 4048 -nm ys-hudi01 -d
Flink Sql-Client startup method:
./sql-client.sh embedded -j ../lib/hudi-flink-bundle_2.12-0.10.0-rc2.jar
Create Flink table script:
CREATE TABLE guhudi(
id bigint ,
name string,
birthday TIMESTAMP(3),
tsTIMESTAMP(3),
partition VARCHAR(20),
primary key(id) not enforced -- the uuid primary key must be specified
)
PARTITIONED BY (partition)
with(
'connector'='hudi',
'path' = 'hdfs://paat-dev/user/hudi/guhudi'
, 'hoodie.datasource.write.recordkey.field' = 'id'
, 'write.precombine.field' = 'ts'
, 'write.tasks' = '1'
, 'compaction.tasks' = '1'
, 'write.rate.limit' = '2000'
, 'table.type' = 'MERGE_ON_READ'
, 'compaction.async.enable' = 'true'
, 'compaction.trigger.strategy' = 'num_commits'
, 'compaction.delta_commits' = '5'
, 'changelog.enable' = 'true'
, 'read.streaming.enable' = 'true'
, 'read.streaming.check-interval' = '4'
, 'hive_sync.enable' = 'true'
, 'hive_sync.mode'= 'hms'
, 'hive_sync.metastore.uris' = 'thrift://:9083'
, 'hive_sync.jdbc_url' = 'jdbc:hive2://:10000'
, 'hive_sync.table' = 'guhudi'
, 'hive_sync.db' = 'hudi'
, 'hive_sync.username' = '****'
, 'hive_sync.password' = '*****'
, 'hive_sync.support_timestamp' = 'true'
);
Insert data:
insert into guhudi select 15,'test2',TIMESTAMP '1970-01-01 00:00:01',TIMESTAMP '1970-01-01 00:00:01','part1';
insert into guhudi select 16,'test1',TIMESTAMP '1970-01-01 00:00:01',TIMESTAMP '1970-01-01 00:00:01','part1';
insert into guhudi select 17,'test1',TIMESTAMP '1970-01-01 00:00:01',TIMESTAMP '1970-01-01 00:00:01','part1';
insert into guhudi select 18,'test1',TIMESTAMP '1970-01-01 00:00:01',TIMESTAMP '1970-01-01 00:00:01','part2';
insert into guhudi select 19,'test1',TIMESTAMP '1970-01-01 00:00:01',TIMESTAMP '1970-01-01 00:00:01','part2';
insert into guhudi select 20,'test1',TIMESTAMP '1970-01-01 00:00:01',TIMESTAMP '1970-01-01 00:00:01','part2';
Partial configuration execution:
set execution.checkpointing.interval=30sec;
set sql-client.execution.result-mode=tableau;
=================================================
Flink SQL> select * from guhudi;
2022-01-15 15:59:56,863 INFO org.apache.hadoop.yarn.client.RMProxy[] - Connecting to ResourceManager at lo-t-bd-nn/<IP_ADDRESS>:8032
2022-01-15 15:59:56,864 INFO org.apache.flink.yarn.YarnClusterDescriptor[] - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2022-01-15 15:59:56,864 WARN org.apache.flink.yarn.YarnClusterDescriptor[] - Neither the HADOOP_CONF_DIR nor the YARN_CONF_DIR environment variable is set. The Flink YARN Client needs one of these to be set to properly load the Hadoop configuration for accessing YARN.
2022-01-15 15:59:56,867 INFO org.apache.flink.yarn.YarnClusterDescriptor[] - Found Web Interface lo-t-work1:38477 of application 'application_1642128573447_0031'.
+----+----------------------+--------------------- -------------+-------------------------+------------ -------------+--------------------------------+
| op | id | name | birthday | ts | partition |
+----+----------------------+--------------------- -------------+-------------------------+------------ -------------+--------------------------------+
| +I | 18 | test1 | 1970-01-01 00:00:01.000 | 1970-01-01 00:00:01.000 | part2 |
| +I | 19 | test1 | 1970-01-01 00:00:01.000 | 1970-01-01 00:00:01.000 | part2 |
| +I | 20 | test1 | 1970-01-01 00:00:01.000 | 1970-01-01 00:00:01.000 | part2 |
| +I | 15 | test2 | 1970-01-01 00:00:01.000 | 1970-01-01 00:00:01.000 | part1 |
| +I | 16 | test1 | 1970-01-01 00:00:01.000 | 1970-01-01 00:00:01.000 | part1 |
| +I | 17 | test1 | 1970-01-01 00:00:01.000 | 1970-01-01 00:00:01.000 | part1 |
+----+----------------------+--------------------- -------------+-------------------------+------------ -------------+--------------------------------+
Received a total of 6 rows
=================FlinkSql query ends ===================
After the above execution is completed, the Hive table automatically creates the rt\ro table:
hive> show tables;
OK
guhudi_ro
guhudi_rt
Time taken: 0.101 seconds, Fetched: 2 row(s)
以上rt\ro表表结构为：
hive> show create table guhudi_ro;
OK
CREATE EXTERNAL TABLE guhudi_ro(
_hoodie_commit_time string COMMENT '',
_hoodie_commit_seqno string COMMENT '',
_hoodie_record_key string COMMENT '',
_hoodie_partition_path string COMMENT '',
_hoodie_file_name string COMMENT '',
id bigint COMMENT '',
name string COMMENT '',
birthday bigint COMMENT '',
ts bigint COMMENT '')
PARTITIONED BY (
partition string COMMENT '')
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
WITH SERDEPROPERTIES (
'hoodie.query.as.ro.table'='true',
'path'='hdfs://paat-dev/user/hudi/guhudi')
STORED AS INPUTFORMAT
'org.apache.hudi.hadoop.HoodieParquetInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION
'hdfs://paat-dev/user/hudi/guhudi'
TBLPROPERTIES (
'last_commit_time_sync'='20220115155947483',
'spark.sql.sources.provider'='hudi',
'spark.sql.sources.schema.numPartCols'='1',
'spark.sql.sources.schema.numParts'='1',
'spark.sql.sources.schema.part.0'='{"type":"struct","fields":[{"name":"_hoodie_commit_time","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_commit_seqno","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_record_key","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_partition_path","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_file_name","type":"string","nullable":true,"metadata":{}},{"name":"id","type":"long","nullable":false,"metadata":{}},{"name":"name","type":"string","nullable":true,"metadata":{}},{"name":"birthday","type":"timestamp","nullable":true,"metadata":{}},{"name":"ts","type":"timestamp","nullable":true,"metadata":{}},{"name":"partition","type":"string","nullable":true,"metadata":{}}]}',
'spark.sql.sources.schema.partCol.0'='partition',
'transient_lastDdlTime'='1642149108')
Time taken: 0.088 seconds, Fetched: 31 row(s)
=================================================
hive> show create table guhudi_rt;
OK
CREATE EXTERNAL TABLE guhudi_rt(
_hoodie_commit_time string COMMENT '',
_hoodie_commit_seqno string COMMENT '',
_hoodie_record_key string COMMENT '',
_hoodie_partition_path string COMMENT '',
_hoodie_file_name string COMMENT '',
id bigint COMMENT '',
name string COMMENT '',
birthday bigint COMMENT '',
ts bigint COMMENT '')
PARTITIONED BY (
partition string COMMENT '')
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
WITH SERDEPROPERTIES (
'hoodie.query.as.ro.table'='false',
'path'='hdfs://paat-dev/user/hudi/guhudi')
STORED AS INPUTFORMAT
'org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION
'hdfs://paat-dev/user/hudi/guhudi'
TBLPROPERTIES (
'last_commit_time_sync'='20220115155947483',
'spark.sql.sources.provider'='hudi',
'spark.sql.sources.schema.numPartCols'='1',
'spark.sql.sources.schema.numParts'='1',
'spark.sql.sources.schema.part.0'='{"type":"struct","fields":[{"name":"_hoodie_commit_time","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_commit_seqno","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_record_key","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_partition_path","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_file_name","type":"string","nullable":true,"metadata":{}},{"name":"id","type":"long","nullable":false,"metadata":{}},{"name":"name","type":"string","nullable":true,"metadata":{}},{"name":"birthday","type":"timestamp","nullable":true,"metadata":{}},{"name":"ts","type":"timestamp","nullable":true,"metadata":{}},{"name":"partition","type":"string","nullable":true,"metadata":{}}]}',
'spark.sql.sources.schema.partCol.0'='partition',
'transient_lastDdlTime'='1642149108')
Time taken: 0.037 seconds, Fetched: 31 row(s)
Query whether the data of the hudi table is in hdfs:
[root@lo-t-work1 ~]# hadoop fs -ls /user/hudi/guhudi
Found 3 items
drwxr-xr-x - root supergroup 0 2022-01-15 15:59 /user/hudi/guhudi/.hoodie
drwxr-xr-x - root supergroup 0 2022-01-15 15:58 /user/hudi/guhudi/part1
drwxr-xr-x - root supergroup 0 2022-01-15 15:59 /user/hudi/guhudi/part2
[root@lo-t-work1 ~]# hadoop fs -ls /user/hudi/guhudi/part1
Found 2 items
-rw-r--r-- 3 root supergroup 3948 2022-01-15 15:59 /user/hudi/guhudi/part1/.6dcddda7-2a1b-44b5-a794-3f9058d005e7_20220115155847944.log.1_0-1-0
-rw-r--r-- 3 root supergroup 96 2022-01-15 15:58 /user/hudi/guhudi/part1/.hoodie_partition_metadata
[root@lo-t-work1 ~]# hadoop fs -ls /user/hudi/guhudi/part2
Found 2 items
-rw-r--r-- 3 root supergroup 2961 2022-01-15 15:59 /user/hudi/guhudi/part2/.f4ba4d90-5035-455e-ae99-4932dde39b77_20220115155937926.log.1_0-1-0
-rw-r--r-- 3 root supergroup 96 2022-01-15 15:59 /user/hudi/guhudi/part2/.hoodie_partition_metadata
Check and find that a hudi file has been generated。
But using hive query is impossible to query, it will be queried under other paths, such as the question posted on the first floor。

@gubinjie
Are there only log files in your current Hudi table?

@xiarixiaoyao @xushiyan
thanks for the reply.

In my above steps, I have inserted six pieces of data through flink-sql, and these six pieces of data can also be found through flink-client
As you said, there are only log files in the hudi table, so are these six pieces of data also stored in the .log file? If it is not stored in the .log file, where are these six pieces of data stored?
If it is a Hive problem, is it not possible to use Hive to read hudi files without rewriting the class org.apache.hadoop.hive.common.FileUtils? Or which hive version supports reading hudi file directory?

@xiarixiaoyao can you give some advices here please? thank you.

@gubinjie  if you donot want to modfiy hive code.   could you pls trigger compaction for your table, one compaction done, parquet file will be created, and above problem should not be happen.
problem 2:  now flink only write log file for mor table
problem3:  I discussed this problem with my company's hive experts, we have no way to solve this problem in hudi, since that this check happens before hive calll hudi code.

@xiarixiaoyao
Thank you for your reply
When I add a kafka connector, and then execute insert into 'hudi' select * from 'kafka', ('hudi' and 'kafka' are tables of connector type respectively)
This time there is no problem, there is data appearing.
But I have a question:
If data is not inserted through the kafka connector, do these parameters have no effect: compaction.trigger.strategy, compaction.delta_commits, compaction.delta_seconds?

CC @yihua kafka connect related issue.

As @xiarixiaoyao mentions, compaction should compact log files into base file formats like parquet, which can be then read by Hive.  There are different ways to trigger compaction, e.g., through inline/sync compaction, standalone HoodieCompactor, hudi-cli commands.  @danny0405 does Flink SQL and writer support compaction?
@gubinjie When you refer to Kafka connector, do you mean Flink Kafka connector?  In Hudi, we also provide Kafka Connect Sink for Hudi.


REST Level 0 or API Level 0 ... you decide

Ok, we need to decide whether the mission will be called "REST Level 0" or "API Level 0" as there is inconsistency between launch.openshift.io and the documentation as well as within the documentation itself.  You can search on either title and find results in the docs repo.

"API Level 0" doesn't make sense to me, the term "API" is way broader than just RESTful APIs. The linked commit seems to use "REST API Level 0" which I wholeheartedly agree with.


Disable bracket-pair colorization on JSX comments

Is it possible to disable bracket-pair colorization on the curly braces surrounding JSX comments? It'd be less distracting/confusing if they matched the comment color, whatever the theme is.

We try to keep VS Code lean and we think the functionality you're asking for is great for a VS Code extension. Maybe you can already find one that suits you in the VS Code Marketplace. Just in case, in a few simple steps you can get started writing your own extension. See also our issue reporting guidelines.
Happy Coding!

This is not possible. But an extension could set a decoration (for {/*...*/}) that overrides the color of this.


Documentation is horribly outdated

Several plugins have been added since it was written.

README.md now mentions them all, even though they're not very well described.
A good solution is an online help feature, which is on the TODO.


[BUG] Possible memory leak on accesscontrol/schemas

Rancher Server Setup

Rancher version: >=2.8, possibly earlier too
Installation option (Docker install/Helm Chart): no special instructions needed

Information about the Cluster

Kubernetes version: any
Cluster Type (Local/Downstream): any

Describe the bug
Analysis of heap profiling shows unexpected usage around accesscontrol/schemas. While investigating, I detected that multiple concurrent usage (e.g. driven by UI usage), could cause multiple routines to all try to Add a new entry to the cache. I have no definitive evidence this causes a leak under all circumstances, maybe it needs rapidly-changing roles to invalidate the cache to trigger the problem, but it's still not optimal.

@aruiz14 Does this only happen when we have concurrent users in the UI all trying to add things at the same time? Is it able to be replicated via just the API by chance, do you know?
Also, do the tests you wrote cover the fixes (i.e no additional QA testing needed)?

Hi @rancher-max, it's not easy at all to replicate this issue. The changes applied here (and #47460) aim to refactor the code to handle better concurrency management. I can't think of any specific validations you could run. My suggestion would be to test both of them together and just ensure that access control works as expected.

Test we're planning on running:

Bring up Rancher with three downstreams and a few objects (our baseline setup)
Run the k6 test for access control:
a. Create roles and users multiple times concurrently
b. Login concurrently with all users
Check memory to ensure it is not steadily increasing after user sessions are closed.

This should hopefully tell us something, but if not we will just close the issue as there are the functional tests written around this already and if we notice in the future then we can try to define a better test to replicate.

Unable to reproduce memory leak on Rancher v2.8.0
No persistent memory increases observed on Rancher v2.10.0 from baseline access control tests
Steps for validation

Bring up HA Rancher on v2.10.0-alpha12, install Monitoring and set CATTLE_PROMETHEUS_METRICS=true
Import 3 downstream clusters
Create 1000 ConfigMaps via create_k8s_resources.js taken from https://github.com/rancher/dartboard/blob/main/k6/create_k8s_resources.js

k6 run -e KUBECONFIG=$PATH_TO_KUBECONFIG -e CONTEXT=$CONTEXT -e BASE_URL=$K8S_API_URL -e CONFIG_MAP_COUNT=1000 -e SECRET_COUNT=1 create_k8s_resources.js

Create 50 Users and 50 Roles via create_roles_users.js taken from https://github.com/rancher/dartboard/blob/main/k6/create_roles_users.js , delete users and roles manually via UI, create roles and users again, repeat a few times, adjust vus up to 20 on various runs

k6 run -e KUBECONFIG=$KUBECONFIG -e CONTEXT=$CONTEXT -e COUNT=$COUNT ./create_roles_users.js

Run heap monitor script, verify no memory leaks are present during access control tests

while true; do
	for pod in $(kubectl -n cattle-system get pods -l app=rancher --no-headers -o custom-columns=name:.metadata.name); do
		kubectl exec -n cattle-system ${pod} -c rancher -- curl -s http://localhost:6060/debug/pprof/heap -o heap
		kubectl cp -n cattle-system -c rancher ${pod}:heap ./heap

		go tool pprof -top --trim=false ./heap
	done
done

view heap data with locally via pprof
pprof -http=localhost:8080 heap


ensurepip fails on latest version

From @devkhan on January 17, 2017 20:13
Using latest(2.7.7) IronPython, trying to install pip, the following happens:
$ mono ipy64.exe -X:Frames -m pip install youtube-dl
Traceback (most recent call last):
  File "/Users/.../IronPython-<IP_ADDRESS>/Lib/site.py", line 552, in <module>
  File "/Users/.../IronPython-<IP_ADDRESS>/Lib/site.py", line 534, in main
  File "/Users/.../IronPython-<IP_ADDRESS>/Lib/site.py", line 266, in addusersitepackages
  File "/Users/.../IronPython-<IP_ADDRESS>/Lib/site.py", line 241, in getusersitepackages
  File "/Users/.../IronPython-<IP_ADDRESS>/Lib/site.py", line 231, in getuserbase
  File "/Users/.../IronPython-<IP_ADDRESS>/Lib/sysconfig.py", line 522, in get_config_var
  File "/Users/.../IronPython-<IP_ADDRESS>/Lib/sysconfig.py", line 474, in get_config_vars
  File "/Users/.../IronPython-<IP_ADDRESS>/Lib/sysconfig.py", line 358, in _init_posix
ImportError: No module named _sysconfigdata
Unhandled exception:
Traceback (most recent call last):
  File "/Users/.../IronPython-<IP_ADDRESS>/Lib/runpy.py", line 182, in run_module
  File "/Users/.../IronPython-<IP_ADDRESS>/Lib/runpy.py", line 107, in _get_module_details
ImportError: No module named pip

Mono details:
$ mono -V
Mono JIT compiler version 4.8.0 (mono-4.8.0-branch/9e8f0ee Wed Jan  4 15:06:01 EST 2017)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
	TLS:           normal
	SIGSEGV:       altstack
	Notification:  kqueue
	Architecture:  x86
	Disabled:      none
	Misc:          softdebug
	LLVM:          yes(3.6.0svn-mono-master/8b1520c)
	GC:            sgen

Operating system: macOS Sierra (10.12) Beta 7
Copied from original issue: IronLanguages/main#1566

Agreed

From @matthewblott on January 19, 2017 20:41
Personally I've never been able to get pip to run using Mono - this looks the same as # 1243.

Duplicate of https://github.com/IronLanguages/ironpython2/issues/72


[Issue 16] Adding Lit-Label Mixin

This adds a new mixin for the labels of the different parts of the graph. Changes in this PR include:

Adding new mixin to render the X and Y axis labels
Adding tests to see if labels exist or not

:tada: This PR is included in version 1.3.0 :tada:
The release is available on:

GitHub release
npm package (@latest dist-tag)

Your semantic-release bot :package::rocket: