Host calls exposed via wasm function indirection tables are not re-entrant

Context
This is part of a series of bugs that I spoke to @tyler @pchickey about. We are currently using  Lucet to sandbox libraries in C++ applications. The idea behind this is that using a wasm sandboxed version of the library allows ensuring that a memory safety issue in the library does not automatically result in a memory safety vulnerability in the full application. One of the  consumers of this work is the Firefox web browser.
Problem
Currently the hostcalls exposed via wasm function tables are not re-entrant. By re-entrancy, I mean the following scenario - I have a Wasm function that calls into the host which further calls another wasm function before returning.
For the library sandboxing use case, we use the function indirection table to make this work. There are some ABI translations involved, but this works well. However, if this host call is re-entrant (tries to invoke another WASM function via the Lucet APIs), the Lucet runtime raises an error. This is because the runtime believes we are already executing WASM code.
The reason we are using the function indirection table for host calls is that, we cannot use the same approach as is used in the Lucet code base. I believe lucet specifies all host calls at compile time in a "bindings.json" file. However, in our case, the set of allowed host calls

Can change dynamically
Are not static calls, but are indirect calls (invoked via a function pointer)

One of the workarounds suggested was to have the re-entrant call use Vmctx::get_func_from_idx, which returns a FunctionHandle, that we can then change to a function pointer. I'm still a little confused if this can work... IIUC, this only exposes indexes of functions which are part of the WASM function indirection table. But this means, the re-entrant call cannot invoke a chosen public function exposed by the WASM module... Instead it can only invoke from the subset of functions that happen to be available in the function indirection table.
Actions
I think the best approach here is to support re-entrancy as a part of run_func. @pchickey mentioned that there are some plans for this, which is fantastic. If it helps, I have implemented a patch to make the Lucet APIs re-entrant in my fork of Lucet (See here). Also, if there is something I can help with re this bug, please let me know.

If we added a method to Vmctx such as:
fn get_export_func(&self, sym: &str) -> Result<FunctionHandle, Error>;

Would that solve the issue? There's no fundamental reason it's not there, we just haven't had the need for it ourselves.
While we have been thinking about adding a run_func variant for Vmctx, we are not planning to make run_func reentrant. Most of the work done there is simply not necessary for calling back into guest code from a hostcall; we don't need to set up a new stack, install signal handlers, etc.

Short term, something similar to that would possibly get us close to handling this... It seems what we would need is both a public version of get_export_func as well as a way to call the returned handle - maybe something like a run_export_func that could do the wasm faulted state checks, argument passing etc. but can elide the signal handler setup etc.
However, I guess the longer term answer depends on what must happen in the host calls

Signal handlers - if the signal handler must be disabled during host call invocation, then even the re-entrant version would have to reset signal handlers. It is not yet fully clear to me what the correct decision would be re keeping/removing signal handlers during hostcall invocation either for the general or library sandboxing use case... I will be looking into this (in the context of library sandboxing) and can post any of my findings in case it is helpful
Stack switch - If hostcalls have to do a stack switch either to the embedding process stack or per discussions on #118 , on to a new stack altogether, stack setup is probably something that needs to happen on re-entant calls as well?
Argument passing - I guess whether a re-entrant version of run_func (and for that matter run_func) handles the argument passing depends on whether it is OK for the application/embedding process to rely on the ABI of the wasm function (i.e. the caller can assume the heap is the first parameter followed by WASM arguments following the calling convention of the platform).

Any thoughts on how to proceed welcome!

maybe something like a run_export_func that could do the wasm faulted state checks, argument passing etc. but can elide the signal handler setup etc.

If it's a method on Vmctx, we could elide the state checks as well, since we should always be in a Running state at that point.

Signal handlers - if the signal handler must be disabled during host call invocation, then even the re-entrant version would have to reset signal handlers. It is not yet fully clear to me what the correct decision would be re keeping/removing signal handlers during hostcall invocation either for the general or library sandboxing use case... I will be looking into this (in the context of library sandboxing) and can post any of my findings in case it is helpful

We would definitely appreciate any feedback you have on the signal handling mechanism, but here's an overview of how it works currently:
The Lucet signal handlers are installed whenever any Lucet instance is currently running in the process, because at least with POSIX we can only install one handler per-signal per-process.
The handler implementation is what we use to distinguish signals from guest code trapping vs. signals from something else going wrong in a hostcall, by checking whether the program counter is within the loaded .so.
The upshot for this issue is that we just shouldn't have to worry about installing or uninstalling handlers when we call from the guest into a hostcall, or from a hostcall into a guest; they should only be modified when entering or leaving the closure passed to Instance::with_signals_on().

Stack switch - If hostcalls have to do a stack switch either to the embedding process stack or per discussions on #118 , on to a new stack altogether, stack setup is probably something that needs to happen on re-entant calls as well?

For the cases where we switch back to the embedding/host stack, we don't have to worry about reentrancy because the previous call to run_func will have returned. Regarding #118, we're currently leaning toward a different solution that would allow guest code and hostcalls to continue coexisting on the same stack, so this will probably be a moot concern.

Argument passing - I guess whether a re-entrant version of run_func (and for that matter run_func) handles the argument passing depends on whether it is OK for the application/embedding process to rely on the ABI of the wasm function (i.e. the caller can assume the heap is the first parameter followed by WASM arguments following the calling convention of the platform).

I think the way to handle this is to add a Vmctx::run_func() that will do the right things ABI-wise. It would reintroduce the need to wrap arguments in Val, though, so we could continue to expose the function pointer to unsafely call if you know how to make the ABI happy. Right now, that's not too complicated because it's just the heap pointer for the first argument, but we're considering adding other pinned registers, like for counting instructions.

Ah gotcha, I think I understand the context now. With the above considerations, it sounds like get_export_func and run_export_func (Vmctx::run_func) would be the right fit. In the interest of next steps, could you let me know if this is something I should contribute a patch for or if this is something that is already on your radar?

I believe I'm hitting a similar wall here. Perhaps this helps:
thread 'main' panicked at 'Invalid state: Instance marked as in a hostcall while entering a hostcall.', lucet/lucet-runtime/lucet-runtime-internals/src/instance/execution.rs:394:17
stack backtrace:
   0: std::panicking::begin_panic
             at $HOME/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:519:12
   1: lucet_runtime_internals::instance::execution::KillState::begin_hostcall
             at lucet/lucet-runtime/lucet-runtime-internals/src/instance/execution.rs:394:17
   2: lucet_runtime_internals::instance::Instance::uninterruptable
             at lucet/lucet-runtime/lucet-runtime-internals/src/instance.rs:910:9
   3: lucet_vmctx_grow_memory
             at lucet/lucet-runtime/src/c_api.rs:441:1
   4: sbrk_1419
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
fatal runtime error: failed to initiate panic, error 5

In this case the hostcall invokes the wasm code just fine via get_func_from_idx/transmute. But
then, it panics if wasm invokes another hostcall. In this case sbrk to grow the heap.
Is there another way of doing this? Perhaps I need to yield -> grow the heap -> resume?

There is no way around this restriction at the moment, and we are not planning to work on one. Lucet is in the process of transitioning all of its unique features into the Wasmtime project, and we recommend users transition to Wasmtime.


Symfony not working on Warden

After creating the environment, when I try to connect to DB as:
warden db connect

It's throwing the following error:

ERROR: Command "${WARDEN_DIR}/bin/warden" env exec db mysql -u"${MYSQL_USER}" -p"${MYSQL_PASSWORD}" --database="${MYSQL_DATABASE}" "${WARDEN_PARAMS[@]:1}" "$@" at /opt/warden/commands/db.cmd:33 failed with exit code 1

My .env file looks like:
WARDEN_ENV_NAME=app
WARDEN_ENV_TYPE=symfony
WARDEN_WEB_ROOT=/web

TRAEFIK_DOMAIN=app.test
TRAEFIK_SUBDOMAIN=

WARDEN_DB=1
WARDEN_REDIS=1
WARDEN_RABBITMQ=0
WARDEN_ELASTICSEARCH=0
WARDEN_VARNISH=0

MARIADB_VERSION=10.1
NODE_VERSION=10
COMPOSER_VERSION=1
PHP_VERSION=7.0
PHP_XDEBUG_3=0
RABBITMQ_VERSION=3.8
REDIS_VERSION=5.0
VARNISH_VERSION=6.0

[Update]
Also, it shows the DB has Exit 1 state:

Let me know how can I fix or get the verbose error message.

It worked after running the following commands
warden env down -v
warden env up


Ignore NavigationIDPath for Cancellation

This patches the effect cancellation code to ignore the NavigationIDPath.
The NavigationIDPath is a new (to our codebase, post-1.3 upgrade) that adds keypath-based path components automatically for parent/child reducer relationships such as the .ifLet reducer. Unfortunately, this navigation path, managed as a Dependency, affects the identity of effect cancellation tokens based on the context in which they're constructed.
We discovered that this has actually broken a lot of our effect cancellation code. Due to the performance implications of sending additional actions, we've actually been pushing this pattern—of using helper functions directly as opposed to sending actions—as a best practice, so we need to disable this new behavior for now while we figure out the best way to move forward.

Current dependencies on/for this PR:

bcny/arc

PR #54   👈

This stack of pull requests is managed by Graphite.

Closing in favor of #55


Use [[DefineOwnProperty]] instead of [[Set]] in polyfill

In #2 it was decided to do this change to the algorithm, but the polyfill was not updated. This affects the result of Object.fromEntries([["__proto__", {}]])

In case any future readers wonder how this distinction could be observable — before:
Object.defineProperty(Object.prototype, 'foo', { set(val) { throw new Error('lol') } });
Object.fromEntries(Object.entries({ foo: 1 })); // Uncaught Error: lol

After:
Object.defineProperty(Object.prototype, 'foo', { set(val) { throw new Error('lol') } });
Object.fromEntries(Object.entries({ foo: 1 })); // { foo: 1 }


⚠ NCAA has degraded performance

In 7bbc781, NCAA (https://www.ncaa.com) experienced degraded performance:

HTTP code: 200
Response time: 670 ms

Resolved: NCAA performance has improved in dfba7b7.


Add roi_max_pooling_2d for new interface

add new interface function roi_max_pooling_2d
add doc
add test

this is related to #5285, #5198.

@Hakuyume  kindly ping. if possible, can you review this?

@Hakuyume can you review this PR?

@Hakuyume can you review this?

jenkins, test this please

Jenkins, test this please.

Any updates?

@Hakuyume Could you confirm the changes since your last review?

Jenkins, test this please.

Jenkins CI test (for commit 82e580e7515504dd27c59e27282b4f7606d7f45c, target branch master) failed with status FAILURE.
(For contributors, please wait until the reviewer confirms the details of the error.)

Jenkins, test this please.

Jenkins CI test (for commit d40263d7fdcbd33ecbddce2fb7cc1ba1a90c5cd6, target branch master) failed with status FAILURE.
(For contributors, please wait until the reviewer confirms the details of the error.)

Jenkins failed on the test of BatchNormalization. It is not related to this PR.
@kmaehashi How should I treat this problem?

Should i merge master branch and wait for jenkins test again?

Thanks for resolving conflicts.

Should i merge master branch and wait for jenkins test again?

Jenkins applies this PR against master branch and tests it, so you don't have to do that unless there is a conflict.
Jenkins, test this please.

Jenkins CI test (for commit 302686e849f68c2106398fabb54c7217b93fb7e1, target branch master) succeeded!


[geometry] Templatization of GeometryModel by type in Pinocchio 2.0

I don't find any discussion on this. If it is a known issue, I apologize.
In  Pinocchio 2.0, all classes and algorithms are templated. In particular, one parameter is the scalar type. However, I see GeometryModel and all related stuff are not templated by type, as they only work with double.
I think it only makes sense that GeometryModel also support other scalar types, given everything else does.
I took a look at hpp-fcl, and I see it only supports double, so I guess that's why.
But it also seems the current official version of FCL does support float (check https://github.com/flexible-collision-library/fcl/blob/master/include/fcl/geometry/collision_geometry.h).
Do you agree this is a concern? How should this be handled? I see three options:

Modify hpp-fcl directly to handle different scalar types (how long would it be?)
Switching to FCL official version (possible? advisable?)
Syncing hpp-fcl with the official FCL repository, as suggested in humanoid-path-planner/hpp-fcl#17 . (possible? advisable? too much work? that issue is two years old)

This indeed a known issue. We have already discussed that with @jmirabel and @florent-lamiraux.
They are planning to merge back to the official FCL repository but it will require some tough work.

I will close this issue as there is no feedback. Feel free to open it again.

I talked with @florent-lamiraux . He told he was rather thinking of implementing scalar type parametrization on hpp-fcl and keep it separate from the official FCL distribution


[release-1.3] fix(virtctl,create): Validate names of disks

What this PR does
Manual backport of https://github.com/kubevirt/kubevirt/pull/13547
Before this PR:
virtctl create vm allows to generate VMs with disk names that will lead to rejection of the VM upon creation.
After this PR:
virtctl create vm validates disk names and prevents disk names that will lead to rejection of a VM upon creation.

Fixes https://github.com/kubevirt/kubevirt/issues/13340
Why we need it and why it was done in this way
The following tradeoffs were made:
The following alternatives were considered:
Links to places where the discussion took place:
Special notes for your reviewer

Checklist
This checklist is not enforcing, but it's a reminder of items that could be relevant to every PR.
Approvers are expected to review this list.

[ ] Design: A design document was considered and is present (link) or not required
[ ] PR: The PR description is expressive enough and will help future contributors
[ ] Code: Write code that humans can understand and Keep it simple
[ ] Refactor: You have left the code cleaner than you found it (Boy Scout Rule)
[ ] Upgrade: Impact of this change on upgrade flows was considered and addressed if required
[ ] Testing: New code requires new unit tests. New features and bug fixes require at least on e2e test
[ ] Documentation: A user-guide update was considered and is present (link) or not required. You want a user-guide update if it's a user facing feature / API change.
[ ] Community: Announcement to kubevirt-dev was considered

Release note

virtctl create vm validates disk names and prevents disk names that will lead to rejection of a VM upon creation.

/retest

/test pull-kubevirt-build-s390x-1.3

/retest

/retest-required

/test pull-kubevirt-e2e-k8s-1.31-sig-operator-1.3

/hold cancel

/retest

/hold cancel


Docker image?

Is there any plans for a official docker image?

Hello @gkawamoto thanks for the interest. Yes, there will be an official Docker image up soon


tree-sitter: Add the .slint UI language

[x] Added an entry to CHANGELOG.md if this change could be valuable to users

This needs to add itself to lapce-app/Cargo.toml, similar to how it was done in lapce-ui/Cargo.toml. Also the changelog shouldn't be in the 0.2.8 section since it wasn't sent out in 0.2.8, it should go in unreleased.

Closing since we can't add more grammars for now


View Modes - Single / Facing Pages / "Flow" arrangements

Description
This PR adds a toggle for changing the page arrangement in the Preview pane, and is the last piece of the new Toolbar that I had in mind for the Toolbar.  Using the 3-way toggle, users can choose different visual arrangements for viewing brews.  This setting is set in the toolbar, so it is not part of the brew metadata and neither does it carry over between brews/page reloads.
The three settings:

Single: the same view we've always had-- one page at a time, in a vertical scroll.
Facing: sets the pages to be side by side in pairs, as if displayed in a book.
Flow: sets the pages to fill the horizontal space and wrap to a new row when full-- works well for 'card' layouts.

In addition to these three layouts, another "options" button is added next to that toggle.  The options button opens as a little 'popup' much like the "saving error" notifications, and contains some inputs:

Row Gap/Column Gap: these sliders alter the spacing between pages when using the "Facing" and "Flow" arrangements.  This likely isn't a critical thing to control, but I added it in case folks have different ideas about perfect spacing (for screenshots maybe).
Start on right:  This input is disabled unless you are on the Facing mode, and changes where the first page is situated.  If a user has a Cover Page on page 1, likely it makes the most sense to have "Start on right" set to true.

Additional options could be added, but i honestly couldn't think of anything else.  They should be strictly for modifying the view, and nothing about the actual brew, I think-- though it is tempting to put a "print friendly" button somewhere on the toolbar, I don't think that'll happen.
Related Issues or Discussions

Closes #1379

QA Instructions, Screenshots, Recordings
Toolbar:

Facing Pages:

Flow:

Possible future changes
I am considering changing the "Flow" mode to using Grid rather than Flex.  Grid can still handle what Flex does, but gives more options such as setting a max number of columns/pages in a row, which could then be a setting in the Options box.  I'm not sure this is possible any longer after a quick check.  If someone knows otherwise, let me know.
Another thing:  possibly an additional view mode is the "page flip" from pageflip.js or turn.js
Reviewer Checklist
Please replace the list below with specific features you want reviewers to look at.
*Reviewers, refer to this list when testing features, or suggest new items *

[x] Verify new features are functional

[x] Can toggle between "single", "facing", and "flow" layouts and is visually correct
[x] Adjustments to Options settings carry through to iframe (Gap settings, and "Start on right")
[x] Style is consistent with the rest of the UI

[x] Verify old features have not broken

[x] Toolbar can still be hidden
[x] Test on at least a few existing brews

[x] Test for edge cases / try to break things

[x] Displays well enough (i.e. nothing hidden) in Firefox
[x] Clicks outside of Options box closes the box

[x] Identify opportunities for simplification and refactoring
[ ] Check for code legibility and appropriate comments

Copy this list
- [ ] Verify new features are functional
  - [ ] Can toggle between "single", "facing", and "flow" layouts and is visually correct
  - [ ] Adjustments to Options settings carry through to iframe (Gap settings, and "Start on right")
  - [ ] Style is consistent with the rest of the UI
- [ ] Verify old features have not broken
  - [ ] Toolbar can still be hidden
  - [ ] Test on at least a few existing brews
- [ ] Test for edge cases / try to break things
  - [ ] Displays well enough (i.e. nothing hidden) in Firefox
  - [ ] Clicks outside of Options box closes the box
- [ ] Identify opportunities for simplification and refactoring
- [ ] Check for code legibility and appropriate comments

The Start on right option could instead be implemented as a fourth view setting option ("book"?). Just an idea.

The Start on right option could instead be implemented as a fourth view setting option ("book"?). Just an idea.

I'd write it as "separate cover" or "start by second page" or something like that.

instead be implemented as a fourth view setting option ("book"?)

That is basically what "Facing" is-- it would make more sense I think to just rename "Facing" to "Book", which is fine with me.  I had steered away from "Book" at the time because I thought it might have certain connotations to it that might set expectations for something else, but I'm not sure that was well-founded now.   As a fourth option, I'm not sure it makes any sense.

I'd write it as "separate cover" or "start by second page" or something like that.

I just went with the phrasing from affinity publisher:

I'm not sure  either of those provides any additional clarity over "start on right".  I do think that there will be users wondering "why would I want my pages unbalanced and have the first page all by itself", but I also think that it will be easy enough to click the button to set it to the left.  Or maybe, this particular setting (left or right start) should be a brew metadata property, so authors can decide what works best for their brew.  Or perhaps the author sets the "default"/initial value, and then viewers can still have this option to change it.

I'm not sure either of those provides any additional clarity over "start on right". I do think that there will be users wondering "why would I want my pages unbalanced and have the first page all by itself", but I also think that it will be easy enough to click the button to set it to the left. Or maybe, this particular setting (left or right start) should be a brew metadata property, so authors can decide what works best for their brew. Or perhaps the author sets the "default"/initial value, and then viewers can still have this option to change it.

Start on right or left sounds good, we could also go for start on odd page or start from even. Same thing.
I think this should be in the viewing stuff, not the metadata panel. Definitely.
About that, is this CSS going to override brews' own code to change layout?

It applies inline styles to the .pages element. So if someone is mucking about outside of .page then yes, they’ll have to !important their rules

Review

[x] Verify new features are functional

[x] Can toggle between "single", "facing", and "flow" layouts and is visually correct
[x] Adjustments to Options settings carry through to iframe (Gap settings, and "Start on right")
[x] Style is consistent with the rest of the UI

[x] Verify old features have not broken

[x] Toolbar can still be hidden
[x] Test on at least a few existing brews

[ ] Test for edge cases / try to break things

[ ] Displays well enough (i.e. nothing hidden) in Firefox
[x] Clicks outside of Options box closes the box

[ ] Identify opportunities for simplification and refactoring
[ ] Check for code legibility and appropriate comments

Comments

2 page brews look ridiculous with the facing layout
The config wheel should not be visible in single view mode
The layout toggle should have some indicator to understand it is a toggle (maye that is just me)
Focus in the checkbox looks terrible, maybe we can adjust it

Should the snap zoom to width snap to two pages if that mode is selected?

Will keep reviewing

2 page brews look ridiculous with the facing layout

Yup, that's just the way they are.  If someone is making a 2 page brew, hopefully one of them isn't a cover page.  But if it is, the viewer can switch it to start on the left, or they can use two other viewing modes.

The config wheel should not be visible in single view mode

Perhaps.  I left it as-is because I was contemplating an option on the single view to have horizontal scrolling rather than vertical.  What do you think about that?  For now, i'll make the view settings disabled on Single, though.

The layout toggle should have some indicator to understand it is a toggle (maye that is just me)

I would agree, I think I intended to do that, put it off, and then just got so used to how the toggle worked that I forgot that it might not be natural to others.  I'll think about how to manage that visually.  Perhaps I will put three little dots under it, and have the index of the current mode colored; a quick picture works as better example:

Or, if there were good icons, I could do a set of three buttons (only one active at a time, like a radio input group).  I'm not sure this works well with just text, though, since it would take so much space:

Focus in the checkbox looks terrible, maybe we can adjust it

Agree, I think the checkboxes need a little makeover.  I don't think focus states has really had any...erm...focus in terms of styling up until recently.  I will take a crack at improving it in this particular toolbar while keeping in mind that a future PR may do a more comprehensive restyling of various inputs and their different states.

Should the snap zoom to width snap to two pages if [facing pages] is selected?

I'll let someone else decide this or follow whatever the consensus is.  I'll leave it as the last thing I do, if at all.

Initial value of the sliders does not match the actual value, at all

I'll have to come back to this when i have more time to really think about it. You may ask "what is there to think about", and honestly i'm just not sure it really matters if their initial values line up with their actual values.

Regarding a better way to indicate that there are three options, rather than clicking on a single button to reveal them, what about this?

instead be implemented as a fourth view setting option ("book"?)

That is basically what "Facing" is-- it would make more sense I think to just rename "Facing" to "Book", which is fine with me. I had steered away from "Book" at the time because I thought it might have certain connotations to it that might set expectations for something else, but I'm not sure that was well-founded now. As a fourth option, I'm not sure it makes any sense.

When I was doing more formal layout the term of the day was "spreads"
Also, for future expansion, you need to consider "back to front" books.

In my latest commits I just switched to icons, though I suppose the title of each can be updated to whatever.  But I also didn't think 'spreads' was wholly appropriate either, because I think of spreads as more of a generic term, more akin to what I have been calling 'modes'.   Correct me if I'm wrong, but I thought a 'spread' basically meant 'arrangement', and that a spread could be displaying 3 pages side-by-side, for example a tri-fold brochure.  In which case, I wouldn't want to say that a two-page 'book' layout is the "spread" view, because that is only one type of spread.
I think more generally I can stop saying "View Modes", and instead go with "View Spreads" or just "Spreads" and update my variable names to that convention.

Also, for future expansion, you need to consider "back to front" books.

Eh, i suppose maybe one day, but I haven't ever seen a single demonstrated need for it (aka no user has ever mentioned it), and until then it's not something I am going to think about.  There would need to be like dozens of users suddenly saying they want it before I personally would go down that path.

At this point, with the latest commit, I only have these concerns from @5e-Cleric to address:

[ ] Focus in the checkbox looks terrible, maybe we can adjust it
[ ] Should the snap zoom to width snap to two pages if [facing pages] is selected?

And I'm contemplating whether some of these things should have default values that are set by brew props, so that authors can say "by default, display this as a book spread".   But I'd be happy to leave that as a new PR.

I don't know how I'm how i am going to get around this issue with the "current page number" field showing NaN when really zoomed out.  I might just set a condition that if the value is not a number, then return 1 and just cheese it like that.

Looks good, page navigation is wonky with multiple pages, but that is to be expected, can we make the next page button move 2 if the pages mode is "facing"?

can we make the next page button move 2 if the pages mode is "facing"?

I think I might leave this for a future thing, because the whole page navigation thing as you mention is wonky.  But I expect that the basic implementation would be easy, possibly one line:
<button
					id='next-page'
					className='tool'
					onClick={()=>scrollToPage(pageNum + (spread === 'facing' ? 2 : 1))}
					disabled={pageNum >= totalPages}
				>

(more a note to myself than anything here).

I think this is pretty much ready to be reviewed again.  Thanks @calculuschild for the react changes.

Actually, I think before merging this in, #3845 should get a second look and be merged before/at the same time.  It handles a lot of the "current page" questions that having "facing" and "flow" spreads will raise.

#3845 relies on some functions that will no longer exist as a result of the refactoring in this PR. I will merge this one now since it is reviewed and complete, then we can resolve the conflicts in 3845 to relocate each piece as needed.


nk3 set-config: Add warning and confirmation prompt

With the upcoming firmware release for the Nitrokey 3, changing some configuration values can have side effects, namely a reset of the opcard application.  We will add new commands with additional information and warnings in the future.  For the time being, this patch adds a warning to the set-config command, indicating that it may cause data loss and should only be used for testing and development.
@sosthene-nitrokey We want to release pynitrokey today for some NetHSM fixes.  I’d like to include something like this even if we don’t have the proper commands for changing specific config values yet.  What do you think?

Maybe we can have an allowlist of config changes that don't  have side effects, and only warn for the others ?

I’ve added a whitelist that skips the warning and confirmation for known config values without side effects, currently only fido.disable_skip_up_timeout.


Report error count for make test2

So now we have error count. But if some errors happen, the return value is still 0, right?

The return value should the number of wrong outputs

@EmmaBYPeng What I meant is, after you invoke make test2, if some tests fail, the return value of running this command should be 1, you can see that by echo $?

A successful command returns a 0, while an unsuccessful one returns a non-zero value that usually can be interpreted as an error code. Well-behaved UNIX commands, programs, and utilities return a 0 exit code upon successful completion, though there are some exceptions.

Alright seems I misunderstood your request. But does the current approach work for you?

It looks good! Actually you've solved #135. Now we have both error exit code, and basic summary of error count. That's good for now, thank you!


Fix ArgumentError when uploading to Cloudinary

See https://github.com/rails/rails/pull/34550
Also Cloudinary Gem has been updated to fix security issues with rest-client but not yet released.
Add this to your Gemfile:
# Gemfile
gem "cloudinary", "1.10.1.pre.rc"

Rebase done. Should be good to go!

Cool. Thank you 🥇


Add AWS Logs installer / configuration

This PR adds installing awslogs daemon, and it lead me to update the setup we use for environment variables. Explanation in the comment below.

@shankardevy After this gets merged, you need to update the variables in private_vars_media.yml file.
Before they looked like:
env_vars:
  mix_env: MIX_ENV=prod
  port: PORT=4000
  ...

And now they should look like this:
env_vars:
  mix_env: prod
  port: 4000
  ...

thanks @mosic for notifying me of the changes. can you also upload these files in a shared google folder privately, so that anyone wanting to use devops can access the private folder and have it up and running?

Sure, I sent you a message on Slack with details.


Feature Request: Filter sql query for count

would be nice if the query counter could be configured to filter only specific types of sql statements, like INSERT or UPDATE or SELECT.

Hello @samuelstein
I agree, it is a feature I had in mind but didn't have the time to work on it.
I think it is an easy PR that could be done by using the prepareStatement interceptor method.
Feel free to try it out!

okay, i will give it a try


304 response caching body and prepending to following request header

Hi,
I am seeing a problem in 2.6.0 of OkHTTP (using it with retrofit 1.9.0).
We have an endpoint on our server that will return a 304 response when we ask it for new messages and no new messages are available. The problem is that the endpoint also returns a body (this is contrary to the http spec I know - and I am working to get that fixed).
However what is done with that body is an issue in this library I think. Instead of discarding it - it will be pre-pended to the HEADER of the next response which breaks the next call.
So on the following request I see an error like:
Unexpected status line: {"statusId":304,"subStatusId":0,"message":"No messages found.","path":"http://<IP_ADDRESS>/resource/appmessage/?lastMessageId=1","statusTimeStamp":"2016-02-03 09:12:18","allParameters":[{"key":"lastMessageId","value":"1"}]}HTTP/1.1 200 OK
The first part is the body of the 304 response, followed by the HTTP/1.1 200 OK which should be the first thing in the header.
Full stack trace here:
Unexpected status line: {"statusId":304,"subStatusId":0,"message":"No messages found.","path":"http://<IP_ADDRESS>/resource/appmessage/?lastMessageId=1","statusTimeStamp":"2016-02-03 09:12:18","allParameters":[{"key":"lastMessageId","value":"1"}]}HTTP/1.1 200 OK
02-03 11:12:17.645 2056-5221/com.servicemagic.consumer.debug E/ConsumerRestLoader:     at com.squareup.okhttp.internal.http.StatusLine.parse(StatusLine.java:54)
02-03 11:12:17.645 2056-5221/com.servicemagic.consumer.debug E/ConsumerRestLoader:     at com.squareup.okhttp.internal.http.HttpConnection.readResponse(HttpConnection.java:191)
02-03 11:12:17.645 2056-5221/com.servicemagic.consumer.debug E/ConsumerRestLoader:     at com.squareup.okhttp.internal.http.HttpTransport.readResponseHeaders(HttpTransport.java:79)
02-03 11:12:17.646 2056-5221/com.servicemagic.consumer.debug E/ConsumerRestLoader:     at com.squareup.okhttp.internal.http.HttpEngine.readNetworkResponse(HttpEngine.java:909)
02-03 11:12:17.646 2056-5221/com.servicemagic.consumer.debug E/ConsumerRestLoader:     at com.squareup.okhttp.internal.http.HttpEngine.access$300(HttpEngine.java:93)
02-03 11:12:17.646 2056-5221/com.servicemagic.consumer.debug E/ConsumerRestLoader:     at com.squareup.okhttp.internal.http.HttpEngine$NetworkInterceptorChain.proceed(HttpEngine.java:894)
02-03 11:12:17.646 2056-5221/com.servicemagic.consumer.debug E/ConsumerRestLoader:     at com.squareup.okhttp.internal.http.HttpEngine.readResponse(HttpEngine.java:748)
02-03 11:12:17.646 2056-5221/com.servicemagic.consumer.debug E/ConsumerRestLoader:     at com.squareup.okhttp.Call.getResponse(Call.java:277)
02-03 11:12:17.646 2056-5221/com.servicemagic.consumer.debug E/ConsumerRestLoader:     at com.squareup.okhttp.Call$ApplicationInterceptorChain.proceed(Call.java:234)
02-03 11:12:17.646 2056-5221/com.servicemagic.consumer.debug E/ConsumerRestLoader:     at com.squareup.okhttp.Call.getResponseWithInterceptorChain(Call.java:196)
02-03 11:12:17.646 2056-5221/com.servicemagic.consumer.debug E/ConsumerRestLoader:     at com.squareup.okhttp.Call.execute(Call.java:79)
02-03 11:12:17.646 2056-5221/com.servicemagic.consumer.debug E/ConsumerRestLoader:     at retrofit.client.OkClient.execute(OkClient.java:53)
02-03 11:12:17.646 2056-5221/com.servicemagic.consumer.debug E/ConsumerRestLoader:     at com.servicemagic.consumer.retrofit.InterceptingOkClient.execute(InterceptingOkClient.java:29)
02-03 11:12:17.646 2056-5221/com.servicemagic.consumer.debug E/ConsumerRestLoader:     at retrofit.RestAdapter$RestHandler.invokeRequest(RestAdapter.java:326)
Looking at the code I don't really understand why the body get's put on the next response.

Per RFC 2616:

The 304 response MUST NOT contain a message-body, and thus is always terminated by the first empty line after the header fields.

And per its replacement, RFC 7232:

A 304 response cannot contain a message-body; it is always terminated by the first empty line after the header fields.

OkHttp stops reading the connection's socket after the first \r\n\r\n which indicates termination of the header section. Thus, when the connection is later re-used for subsequent requests, attempting to read the status line of the next response gets the erroneous body of the previous request since those bytes were left sitting on the wire.
If you fix your server to conform to the RFC you'll see OkHttp behave correctly.

You might be able to work-around this by adding a Connection: close header to every request that could suffer this problem. You’ll get worse performance until you remove that.


 It's even slower without instant mode enabled.

Looking through the debug logs, no individual step is unreasonably slow. There's just so many of them it takes more than ten seconds in total. At that rate I could retype it myself faster.


added RDA standard to code

TEST DO Not Merge

Created new staging environment for pr-62. Here are some useful links:
WordPress Admin
Homepage
Article
Listicle
Page
Hub

There are visual regressions!
View the full visual regression test report

⚠ Frontend performance tests failed!
Full Lighthouse performance results:

Page/Branch
pr-62
master
Result

Homepage
92
96
✔

Article
97
98
✔

Listicle
60
98
✘

Page
98
98
✔

Hub
97
94
✔

(Note: Tests are against the NOADS version of the pages!)

Updated staging environment for pr-62. Here are some useful links:
WordPress Admin
Homepage
Article
Listicle
Page
Hub

Congrats! Frontend performance tests passed!
Full Lighthouse performance results:

Page/Branch
pr-62
master
Result

Homepage
95
94
✔

Article
98
97
✔

Listicle
97
97
✔

Page
98
98
✔

Hub
94
94
✔

(Note: Tests are against the NOADS version of the pages!)

There are visual regressions!
View the full visual regression test report

Updated staging environment for pr-62. Here are some useful links:
WordPress Admin
Homepage
Article
Listicle
Page
Hub

There are visual regressions!
View the full visual regression test report

⚠ Frontend performance tests failed!
Full Lighthouse performance results:

Page/Branch
pr-62
master
Result

Homepage
90
96
✘

Article
98
98
✔

Listicle
98
98
✔

Page
98
98
✔

Hub
97
94
✔

(Note: Tests are against the NOADS version of the pages!)

Updated staging environment for pr-62. Here are some useful links:
WordPress Admin
Homepage
Article
Listicle
Page
Hub

There are visual regressions!
View the full visual regression test report

Congrats! Frontend performance tests passed!
Full Lighthouse performance results:

Page/Branch
pr-62
master
Result

Homepage
91
94
✔

Article
96
88
✔

Listicle
97
98
✔

Page
98
98
✔

Hub
97
92
✔

(Note: Tests are against the NOADS version of the pages!)

Merging now, and created https://readersdigest.atlassian.net/browse/DRDT-62 to add back the most critical rules (and discuss about the not-so-critical ones).


Use of reST directives in docstrings, rending in Object Inspector

From<EMAIL_ADDRESS>on 2013-11-07T18:48:54Z
I'm following the numpy docstring standard [ https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt ] and noticed that reST directives to not render in the Object Inspector. Shouldn't they? Or perhaps I didn't specify them correctly?
Spyder 2.2.3, Python 2.7.5 x64, Qt 4.8.4, PyQt4 4.9.6 api2, XP64
Attachment: rest_test.png
Original issue: http://code.google.com/p/spyderlib/issues/detail?id=1637

From<EMAIL_ADDRESS>on 2014-07-17T22:07:42Z
This can be closed afaic -- may have made a better mailing list question.

@ccordoba12 can this be closed then?

Nop, we haven't integrated numpydoc as part sphinxify and we should to provide better rendering of docstring formatted with that standard :-)

What steps would be needed to include numpydoc (or perhaps even napoleon?) in Spyder?
Is it just to include sphinx.ext.napoleon in the configuration override in sphinxify.py ? (Of course depending on a successful import of the module).
How should this dependency be handled? Should it go into the dialog window with the list of optional dependencies?
What other steps would be needed?


Migrate TravisCI and CircleCI to GitHub Actions

Unfortunately GitHub Action wont run from PR of a fork (if there was no GitHub Action already in the repository). Evidence of successful GitHub Action can be seen in: https://github.com/saheerb/mbed-os-example-blinky-baremetal/actions/runs/1245266345

@LDong-Arm  @Patater can you review this please?

@jamesbeyond  already disabled it. I will close and reopen.

The error is from two hours ago. Reopening doesn't seem to remove previous runs of checks that no longer exist. As you confirmed it's now disabled, I'm merging this PR.

Did a squash push to remove the error ;-)


Setup Windows Client

Setup Windows client
~Specifics to be discussed

Possible Options for Windows UI (C#)

.NET Native (Windows App SDK)
Windows Forms (.NET Core)
Windows Presentation Foundation (.NET Framework)
OpenTK (OpenGL + WinForms support)
Vortice.Windows (DirectX)

need layout/design for the user interface


A11y_SQL database projects extension_View And More Action_Update Project From Database_Name Role Value: Name is not defined for the "Server", "Data base" and "Location" combo box.

GitHub Tags
#A11yTCS; #A11ySev2; #DesktopApp; #Name Role Value; #WCAG4.1.2; #Win11; #SH-SQL Azure Data Studio-Win32-March23; #Win32; #SQL Azure Data Studio; #NVDA; #Screenreader; #A11yWCAG2.1; #Closed;
"Please do not close this bug. This bug should only be closed by TCS, C+AI Accessibility or the Divisional Driver"
Environment Details
Application Name: SQL Azure Data Studio
Version: 1.41.2 (user setup)
Microsoft Windows 11 Enterprise 10.0.22621 Build 22621
Screen reader: NVDA
Repro Steps

Launch Azure Data Studio
Tab till extension and search for “SQL Database Projects” and install it.
Tab till database project and press enter. “Database Project” panel will open.
Tab till “View and More Action” button and press enter.
Tab till “Update Project from Database” and press enter.
Tab till "Server", "Data base" and "Location" combo box.
Verify whether the name is not defined for the "Server", "Data base" and "Location" combo box or not.

Actual Result
Name is not defined for the "Server", "Data base" and "Location" combo box.
Expected Result
Name should be defined for the "Server", "Data base" and "Location" combo box.
User Impact
Screen reader user will not be able to get the proper information about the "Server", "Data base" and "Location" combo box.
Attachment

Regressed the issue in the below environment
Version: 1.43.0-insider (user setup)
Commit: afafee844ce2b07eb55bdb6f40ee1ce88c0bbf4e
Date: 2023-03-29T05:28:54.263Z
VS Code: 1.67.0
Electron: 19.1.8
Chromium: 102.0.5005.167
Node.js: 16.14.2
V8: <IP_ADDRESS>-electron.0
OS: Windows_NT x64 10.0.22621

The issue is fixed. Hence closing the bug. Please find the attachment for the reference.


Remove traces of AB3

Codecov Report

Merging #190 (9872141) into master (e43f7ae) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##             master     #190   +/-   ##
=========================================
  Coverage     73.26%   73.26%
  Complexity      584      584
=========================================
  Files            94       94
  Lines          1803     1803
  Branches        232      232
=========================================
  Hits           1321     1321
  Misses          403      403
  Partials         79       79

Impacted Files
Coverage Δ

...ava/seedu/address/logic/commands/ClearCommand.java
100.00% <ø> (ø)

...du/address/logic/commands/ClearFoldersCommand.java
100.00% <ø> (ø)

...va/seedu/address/logic/commands/DeleteCommand.java
100.00% <ø> (ø)

.../logic/commands/DeletePersonFromFolderCommand.java
76.00% <ø> (ø)

...java/seedu/address/logic/commands/EditCommand.java
97.10% <ø> (ø)

.../address/logic/commands/EditFolderNameCommand.java
66.66% <ø> (ø)

...java/seedu/address/logic/commands/ListCommand.java
100.00% <ø> (ø)

...edu/address/logic/commands/ListFoldersCommand.java
100.00% <ø> (ø)

src/main/java/seedu/address/model/AddressBook.java
88.52% <ø> (ø)

src/main/java/seedu/address/model/Model.java
100.00% <ø> (ø)

... and 9 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e43f7ae...9872141. Read the comment docs.


adds missing docs for $permalinkBase

Description
Spoke with @ifox on Discord about $permalinkBase to the docs because it was missing. Here's the PR.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Run the QuickStart unity demo,one error showed up

version:
unity :2020.3.16
openxr:1.0.0
ARR:1.0.21
Problem:

I have fllowed the guide of document, step by step setup the Remote Rendering Scripts account info. please point out what is the problem caused this. thanks

Have found the reason from the official troubleshot: [https://docs.microsoft.com/en-us/azure/remote-rendering/resources/troubleshoot].


🐞 Bug Fix: Tools must be added after extensions

name: 🐞 Bug Fix
about: Tools must be added after extensions
labels: bug

A Pull Request should be associated with a Discussion.
Related discussion: #532

Further notes in Contribution Guidelines
Thank you for your contribution.

Description
This PR aims to move tools installation after extensions.

In case this PR introduced TypeScript/JavaScript code changes:

[ ] I have written test cases for the changes in this pull request
[x] I have run npm run format before the commit.
[x] I have run npm run lint before the commit.
[ ] I have run npm run release before the commit.
[x] npm test returns with no unit test errors and all code covered.

In case this PR edits any scripts:

[x] I have checked the edited scripts for syntax.
[ ] I have tested the changes in an integration test (If yes, provide workflow YAML and link).

https://github.com/shivammathur/setup-php/issues/532#issuecomment-974346750


Core Functionality

A list of core-functionality the bot will need to function and that aren't put within an extension.

[ ] Extension/cog handling
[ ] Error handling

We have the core bot running at the moment, I'd say this is finished.


Implement full TLS1.3 OQS signature support

Warning: This is a big issue. Anyone willing to tackle it should consider this requires substantial OpenSSL3 support (see for example https://github.com/openssl/openssl/issues/10512). Also helpful to get started may be discussions documented here.

email input by @anhu:

My goal is to avoid any source code conflict with any patch that would be accepted into the OpenSSL code base so I think that upstream uptake is definitely a joint goal.
How can I get started helping you? Let me know via github please!

Here's what my plan is (in terms of github issues and in order of resolving): #2, #6, #7, #8, #3 and concluding with #9. From the work on the KEM provider I fully expect upstream changes to be(come) necessary well before concluding #9/https://github.com/openssl/openssl/issues/10512. Thus, depending on how fast we progress on the issues above relative to the OSSL3 release timeline, we may either do upstream PRs or a fork of upstream master. For either option to work out, please be aware that you (probably your employer) need to sign up to this: From my own corporate history I have the hunch that getting this signed may take "longish", so why no get started on this now, @anhu? If you agree with all of this, I'd suggest technical work to begin by looking at this and then "diving into" #2: After we'll have a standalone SIG-capable provider (e.g., doing CMS), I'd expect the TLS parts to be relatively straightforward.

Hi Michael ,
Thanks!  I have already signed up for the OpenSSL Contributors Agreement since we wanted to make 2 minor contributions to the OpenSSL code base:

0a8e6c1fb77612feba350a67dad3e548300785a7
2555285fa5e4248ad4a5a0bc14ae4606443856c2

It likely comes as no surprise to you that we are implementing our own closed source provider.  I am actually currently looking into #2, but for our provider.  Of course, our provider will be different from oqs-proivder's internals.  As such, the code that I will have will be distinctly different than what you will need and will obviously not patch in cleanly. However,  I would be happy to give pointers, helpful hints and warnings about potential dead ends. For example, you won't need to write encoder import methods because there are no other implementations of these algorithms to import from.  When I first started looking at the provider encoder documentation I spent a lot of time trying to figure what I needed to  do for the encoder's import API and finally figured out that it was not necessary. Hopefully, I've prevented you from wasting that time like I did.

Hopefully, I've prevented you from wasting that time like I did.

Quite possibly: I have spent a lot of time scratching my head about the encoder/decoder concepts and how they'd need to use the X509 APIs -- and still do :-( so anything I don't need to wonder about helps.

Same here!!  It does not help that encode_key2any.c is hard to read (for  me) due to macro defines.
I started with text encoder because it seemed simpler. Unfortunately, this means I can't really test it so I don't know that I got it right.
I only implemented

OSSL_FUNC_ENCODER_NEWCTX
OSSL_FUNC_ENCODER_FREECTX
OSSL_FUNC_ENCODER_GETTABLE_PARAMS
OSSL_FUNC_ENCODER_GET_PARAMS
OSSL_FUNC_ENCODER_ENCODE

For me newctx() and freectx() were trivial. encode_key2text.c is a bit easier to follow. Might be a good guide/template  for you as well.
TIP: You'll also need to get openssl's OSSL_FUNC_BIO_VPRINTF function pointer .

I'd like to propose a direction for this issue:
Providers that supply new and unknown sigalgs for TLS 1.3 signature verification should be able to do so via a fully pluggable mechanism called the capabilities query. OpenSSL will query the provider asking it what kind of sigalgs it supports. During this call, the provider will register the name and the TLS 1.3 code point of the sigalgs it supports. That name can be referenced in .conf files or command line parameters to specify supported sigalgs.
This would be similar to the way it currently works for supported groups in TLS 1.3.
Something that needs to also be considered is the OIDs for the signature and public key of the registered sigalgs.  Do the OIDs need to be associated with the codepoint and name as well?

Fully agree: All this "registration" (or reference) information (at the very least name and OID) needs to be accessible to OpenSSL.
The one question I'd have is why this would necessitate a new(?) capabilities query: Wouldn't/shouldn't it be so that name+OID are automatically associated (and thus made available to all OpenSSL piece-parts, incl. signature reference APIs & command line parameters) with the relevant (algorithm's) code the moment the provider is loaded?

Yes.  Name+OID are automatically associated already.
But much like groups, you need to associate the sigalg code points with the name as well. Perhaps the term "code point" is obscure so I'll just explicitly paste the relevant section from the TLS 1.3 spec here:
      enum {
          /* RSASSA-PKCS1-v1_5 algorithms */
          rsa_pkcs1_sha256(0x0401),
          rsa_pkcs1_sha384(0x0501),
          rsa_pkcs1_sha512(0x0601),

          /* ECDSA algorithms */
          ecdsa_secp256r1_sha256(0x0403),
          ecdsa_secp384r1_sha384(0x0503),
          ecdsa_secp521r1_sha512(0x0603),

          /* RSASSA-PSS algorithms with public key OID rsaEncryption */
          rsa_pss_rsae_sha256(0x0804),
          rsa_pss_rsae_sha384(0x0805),
          rsa_pss_rsae_sha512(0x0806),

          /* EdDSA algorithms */
          ed25519(0x0807),
          ed448(0x0808),

          /* RSASSA-PSS algorithms with public key OID RSASSA-PSS */
          rsa_pss_pss_sha256(0x0809),
          rsa_pss_pss_sha384(0x080a),
          rsa_pss_pss_sha512(0x080b),

          /* Legacy algorithms */
          rsa_pkcs1_sha1(0x0201),
          ecdsa_sha1(0x0203),

          /* Reserved Code Points */
          private_use(0xFE00..0xFFFF),
          (0xFFFF)
      } SignatureScheme;

As I'm sure you'll remember with your work with groups, you had  to return your code points during a capabilities query.  The same thing would have to happen with signature algorithms.  OpenSSL would have to do a capabilities query for signatures and it would pass the name of a signature algorithm and then the  provider would respond with the code point that corresponds to that name if it supports that algorithm.
Hopefully, this makes things clearer?

Fixed by https://github.com/openssl/openssl/pull/19312


virustotal discussion

So this is icetea client from cyde's website
https://www.virustotal.com/gui/file/04484b38ea7a8fcb50ed76aab421596a68636b1e82c571a7b230c7fa434f0701
Is that a virus or not? So many flags, but what do you think?
I acknowledge that your autoclicker isnt a virus.

Not related.

lmaooooo ratter


Functions to move out after the multibody switch.

The following functions must be moved out of rbt_utils, into multibody itself as they are utility functions that are system independent. The functions can be added as an extension to the existing drake framework or pushed into Drake itself.

GetBodyIndexFromName
JointsWithinLimits

I believe this is resolved.


Oracle jdbc connection keep-alive

Is there any option for oracle jdbc connection to keep connection alive. It's importat mainly for remote connection behind firewall. "Unused" connection is droped by firewall and dbeaver freezes on CPU.

# @ @

What is it??

But it doesn't work.
I tried with maximum value (32767 s) and it keeps disconeccting after 300 s or so. In my opinion our server is the responsible por dropping connections; but Dbeaver could provide as with an active keeping-alive feature.

You need to set it to a smaller value, e.g. 60.
Keep-alive is the delay between connection ping attempts.  If your server closes connection after 5 minutes of idle then you must set Keep-Alive to a value less than 5 minutes (300 seconds).

Wow, it worked. Thank you very much, @serge-rider. It was frustrating having to wait for the connection to reset, everytime I came back to search anything.


fix: scope CRD update permissions v0.37.x

Fixes #N/A
Description
Scopes down update CRD permissions given to Karpenter.
How was this change tested?
Does this change impact docs?

[ ] Yes, PR includes docs updates
[ ] Yes, issue opened: #
[x] No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>6
Details

0 of 0   changed or added relevant lines in 0 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage decreased (-0.01%) to 78.678%

Files with Coverage Reduction
New Missed Lines
%

pkg/providers/amifamily/ami.go
1
90.5%

Totals

Change from base Build<PHONE_NUMBER>9:
-0.01%

Covered Lines:
5893

Relevant Lines:
7490

💛  - Coveralls


Queue search breadcrumbs

As a ministry coordinator or IAO analyst
I want breadcrumbs when doing a search on the queue
so that I can navigate back to my original search parameters

Assumptions & Scope
What are the assumptions for this story?
Users have asked for to do a search on their queue and easily navigate back to their search results without re-entering the search criteria, column sort, or navigate to a certain page.
Will use material UI component.
What is IN scope?
Breadcrumbs for advanced search
What is NOT in scope?
Breadcrumbs for Advanced Search #1884
Acceptance Criteria
Scenario 1: Search on queue

GIVEN a ministry coordinator or IAO analyst is on a queue (My requests, My Team's Requests, Watching Requests)
AND search keywords have been entered in the search bar
AND/OR column sorts (asc, desc) have been activated
AND search results are listed
WHEN the user clicks on the a request listed
THEN the request view will open
AND a 2 layer breadcrumb will show with Queue Search and the Request Number for the request the user is currently on

Scenario 2: Click Queue Search breadcrumb

GIVEN the user is on a request found as part of an queue search
WHEN the user clicks the Queue Search breadcrumb
THEN the user will be returned to the queue search view (My requests, My Team's Requests, or Watching Requests)
AND their search results will be listed
AND any column sort toggles that were previously active shall remain active
AND the their search parameters will still be selected
AND the pagination will be maintained

...
Dependencies? What is the impact of this dependency? (If so, link dependency in the ticket, make it visible in a team´s backlog)
Validation Rules? (If yes, list here)
Design
@xxx - please link the Design here
Definition of Ready

[ ] Is there a well articulated User Story?
[ ] Is there Acceptance Criteria that covers all scenarios (happy/sad paths)?
[ ] If there is a user interface, is there a design?
[ ] Does the user story need user research/validation?
[ ] Does this User Story needs stakeholder approval?
[ ] Design / Solution accepted by Product Owner
[ ] Is this user story small enough to be completed in a Sprint? Should it be split?
[ ] Are the dependencies known/ understood? (technical, business, regulatory/policy)
[ ] Has the story been estimated?

Definition of Done

[ ] Passes developer unit tests
[ ] Passes peer code review
[ ] If there's a user interface, passes UX assurance
[ ] Passes QA of Acceptance Criteria with verification in Dev and Test
[ ] Confirm Test cases built and succeeding
[ ] No regression test failures
[ ] Test coverage acceptable by Product Owner
[ ] Ticket ready to be merged to master or story branch
[ ] Developer to list Config changes/ Update documents and designs
[ ] Can be demoed in Sprint Review
[ ] Tagged as part of a Release
[ ] Feature flagged if required
[ ] Change Management activities done?

Update AC to include scenario for column sort and pagination

AC Updated to include column sort and pagination requirements. Folded into existing AC

@nkan-aot - it seems to be working as expected, but I am noticing what seems like a major bug that seems to have come with it.... When I follow the breadcrumb back, I can't make any further changes to the search criteria. In fact - it seems to be extending even when I don't use breadcrumbs. For example, I did a search, and realized I spelled the name wrong... but I couldn't correct it - it locked my filter almost immediately

Just realized - it's when I sort a column, then it locks the search filter. That seems to be what's causing the issue

Nice work fixing those issues that came out of it @nkan-aot - scenarios passed, closing ticket


Optimize the file upload client demo

i think go-chassis need to implement a common file upload method

Coverage decreased (-0.2%) to 62.453% when pulling c49611ce1743da0771600868865bbae8766a18bb on little-cui:master into fae4f4d5fe2ceb4cb6ca0d89dc0f0dfcc58b9c54 on ServiceComb:master.


Scaffold a concourse pipeline for our release process

@rhysyngsun can you add more detail to the issue?


Removal of UserSkel.gaeadmin-bone?

According to https://github.com/viur-framework/viur-core/blob/main/src/viur/core/modules/user.py#L626, the BooleanBone gaeadmin of UserSkel is never set, as this part was never ported from ViUR 2 to ViUR 3.
In our recent developments, we also hadn't any problems that this property is missing. Therefore my suggestion is to remove it entirely, as part of pull request #983.

Removed in https://github.com/viur-framework/viur-core/pull/983


Use JDK default security provider when Conscrypt isn't available

Fixes #12907
Motivation

fixes issue with ARM64 platform where Conscrypt isn't available.

Modifications
Call org.conscrypt.Conscrypt.checkAvailability() method using reflection to verify that Conscrypt can be loaded successfully. In a failure case, log the error and fallback to JDK default security provider.

@lhotari Hi, is it expected to log stack trace for the warning message when Conscrypt class is not found?  Thanks.

It's a different matter though that we do not support Conscrypt as a security provider.

Why do you want to disable Conscrypt?

Oh well, we are in enterprise application development space and not in the mobile app or android space.

Why do you want to disable Conscrypt?

Oh well, we are in enterprise application development space and not in the mobile app or android space.

Unfortunately Conscrypt isn't related to mobile apps or android space in Pulsar. Conscrypt is used in Pulsar to improve the performance of Jetty TLS layer. #10372 explains the rationale. The official Jetty documentation recommends to use Conscrypt to achieve good performance with TLS. Embedded Jetty is used as the http server for the Pulsar broker and Pulsar proxy.

Sure, we can evaluate this in our context. Thanks for the detailed info.


FakePlayers Permission Group stays on default, despite group Switch [DiscordIntegration]

Using DiscordIntegration as an example, even If I set it's UUID's group to, say a group called admin, whenever you try to type a command, it'll still check the default group for permissions, not the correct group.
https://mcuuid.net/?q=828653ca-0185-43d4-b26d-620a7f016be6
(I'm only able to give it perms via it's UUID, by "lp user 828653ca-0185-43d4-b26d-620a7f016be6 parent
setprimarygroup bot)

Fixed. For anyone else using LP, it seems DiscordIntegration uses 41c82c87-7afb-4024-ba57-13d2c99cae77
as it's UUID

Related to https://github.com/lucko/LuckPerms/issues/362, except DiscordIntegration is a Forge Server-Side Mod...

I don't understand what the problem is?
Can you give an example of the behaviour you're currently seeing, and what you expect to be happening?

I have created a group called 'bot and have assigned it permissions like "minecraft.command.*" or "minecraft.command.stop". The Issue is that despite this and setting the FakeUser to the bot group (via UUID, since It wont let me put in it's name), when i'd try to run !stop within discord to feed /stop into minecraft, the command returns with Insufficient permissions, even though it's set on the bot group.

LuckPerms won't handle permission checks for fake players, unless Sponge/Forge does anything special to bridge the gap.
I don't know how that system works (if it even exists at all) so can't really provide much help.

As far as I know this isn't a problem LP can solve.
There isn't any official implementation of LP for Forge servers - if you're using LP on Forge, I'm assuming it's via something like Sponge or Cauldron. Afaik it's the responsibility of those projects to bridge the gap.

FakePlayer extends EntityPlayerMP
There should be NO issues checking permissions on a FakePlayer instance.
I see no where in Minecraft Forge, where it distinguish between a normal player and a FakePlayer with regards to commands.

LuckPerms is not a forge mod.

To clarify:
I suggest opening a ticket on the SpongeForge repository to see if support for FakePlayer permission checks can be added in Sponge.


Added Java_2.java file which displays "Hello World!" using a static m…

Added Java_2.java file which displays "Hello World!" using a static method

Merged, thank you!


Fix trailing commas #469

remove trailing comma to be more IE compliant

Looks great I merged it into my fork dsmorse/gridster.js


Trigger scheduler API endpoint fails to save the results to the database on subsequent calls

When trying to run the toy-schedule multiple times trough the API endpoint, I've encountered that the second time I call the schedule, it fails to save the results to the database, see Error below.
In order to reproduce this, I've tested to trigger the schedule twice with the toy-account  (see Setup to reproduce this behavior). This error persists on server restart but is gone on a fresh new database.

:warning: Reminder: you need to use different arguments in the API calls to bypass  the "anti-repeated call feature".

A potential fix consists of delete all the objects in the session. This can be done by setting the argument expunge_session of add_to_session to True, exposing it in the function save_to_db as an optional argument with default value False.
Follow up
Please, let me know if this issue shows up to you, as well. In case this error happens to you as well, I would like to know if the proposed solution is on point or, requires more research to see the origin of this error.

Error
HANDLING RQ SCHEDULING WORKER EXCEPTION: <class 'sqlalchemy.exc.InvalidRequestError'>:Can't attach instance <Sensor at 0x7f0f9a1e5e70>; another instance with key (<class 'flexmeasures.data.models.time_series.Sensor'>, (1,), None) is already present in this session.

Setup to reproduce this behavior
1. Add accounts
1.0. Delete database
make clean-db db_name=flexmeasures

1.1. toy-account
flexmeasures add toy-account

1.2. admin account
flexmeasures add account --name  "admin"

flexmeasures add user --username admin --email<EMAIL_ADDRESS>--account-id 2 --roles=admin

2. Authentication
URL
http://localhost:5000/api/requestAuthToken

HEADERS
Content-Type: application/json

BODY
{
	"email"<EMAIL_ADDRESS>	"password" : "admin"
}

3. Post Measurements
URL
http://localhost:5000/api/v3_0/sensors/data

HEADERS
Content-Type: application/json
Authorization: <auth_token>

BODY
{
	"sensor" : "ea1.2023-06.localhost:fm1.2",
	"start" : "2023-05-14T00:00:00+02:00",
	"duration" : "PT24H",
	"unit" : "EUR/MWh",
	"values" :  [10, 11, 12, 15, 18, 17,
          10.5, 9, 9.5, 9, 8.5, 10,
          8, 5, 4, 4, 5.5, 8,
          12, 13, 14, 12.5, 10, 7]
}

4. Trigger Storage Schedule
URL
localhost:5000/api/v3_0/sensors/1/schedules/trigger

HEADERS
Authorization: <auth_token>
Content-Type: application/json

BODY
{
	"start" : "2023-05-14T00:00:00+02:00",
	"duration" : "PT12H",
	"flex-model" : {
		"soc-unit" : "MWh",
		"soc-at-start" : 0.212,
		"soc-targets" : []
	},
	"flex-context" : {
		"production-price-sensor" : 2,
		"consumption-price-sensor" : 2
	}
}

Thanks for this report @victorgarcia98 - I researched this behavior before (just can't find it right now), and I saw that this only happend if the very same data gets written a second time.
I hope I find my analysis again. Also, I'm not sure where we left this at the time. Maybe we didn't understand the error, but computing the same schedule twice seemed something we would prevent from happening anyways.
I do want to understand what SQLAlchemy really complains about here.

Great!
Indeed, this shouldn't matter when running the scheduler with the same arguments (here make_schedule wouldn't be called due to the new repeated call feature). However, this is happening to me when running the schedule multiple times with different soc-at-start values.
For reproducing the error, I'm using Insomnia to make the calls to the API. If you want, I can export the requests that I'm doing to a file, which should make it easier/faster by importing them to a tool such as Postman.

@Flix6x do you recall where I wrote about this a few weeks ago?

@victorgarcia98 I found my previous research on the issue.
I concluded that "overwriting of pre-existing data might be broken". This means overwriting with data for the same timing, but with different values. The logic which handles that might be in timely-beliefs.

Some potentially relevant thoughts.
When saving beliefs data that may contain such "changed beliefs" (from the same source, at the same belief time, about the same event, but assigning a different value), we should use a dedicated function we made (I think save_to_db) in I think data.utils. I expect this function has relevant logic to either prevent such errors, or raise more informative error messages.
I would actually expect our scheduling logic (both through the API and the CLI) to generate beliefs with a unique belief time each time the logic is executed. That is, somewhere in the validation of the prior field as the belief_time,  it would default to server_now.

After some research, I found the following:
Potential Fixes
Method 1
A potential fix consists of delete all the objects in the session. This can be done by setting the argument expunge_session of add_to_session to True, exposing it in the function save_to_db as an optional argument with default value False.
Instead of expunging all the objects, there's the possibility to expunge just the sensor object. The issue of doing so is that we might lose the data of the Sensor, such as the updated attributes that get updated after the call. Maybe we need to commit that information first in the method persist_flex_model.
Method 2
Another potential fix is to create a fresh new Sensor object after consumption_schedule = scheduler.compute(). It turns out that the object is detaching after the call of compute, regardless of the initial conditions.
Hypothesis
Relationship
On saving the results, the TimedBelief objects try to fetch the Sensor object by doing a DB merge of the sensor table into timed_belief (or subquery). This is where it would probably complain given that the Sensor object is already in the session.
Uncommitted sensor
On compute, we update the SOC depending if we compute the schedule with the same parameters or not. In case this is updated, it will requires us to commit the results to the database.

Thank you for diving deeper into this. I suspect that we no longer need to keep the logic that updates sensor attributes on calling compute, as it is connected to API versions that have now been sunset. So that might be one way forward I could try.

I'm still not able to reproduce. I am running:
Versions:

Python 3.9 (from running python --version)
Postgres: PostgreSQL 13.5 (from running SELECT version(); in postgres terminal
SQLAlchemy 1.4.47
Flask-SQLAlchemy 2.5.1

Preparation
In flexmeasures.cfg:
SQLALCHEMY_DATABASE_URI =<EMAIL_ADDRESS>
Create test_client.py containing:
import asyncio

from flexmeasures_client.client import FlexMeasuresClient as Client

async def my_script():
    await client.post_measurements(
        sensor_id=2,
        entity_address="ea1.2023-06.localhost:fm1",
        start="2023-05-14T00:00:00+02:00",
        duration="PT24H",
        unit="EUR/MWh",
        values=[10, 11, 12, 15, 18, 17,
                10.5, 9, 9.5, 9, 8.5, 10,
                8, 5, 4, 4, 5.5, 8,
                12, 13, 14, 12.5, 10, 7],
    )
    await client.trigger_storage_schedule(
        sensor_id=1,
        start="2023-05-14T00:00:00+02:00",
        duration="PT12H",
        soc_unit="MWh",
        soc_at_start=0.212,
        soc_targets=[],
        consumption_price_sensor=2,
        production_price_sensor=2,
    )
    await client.trigger_storage_schedule(
        sensor_id=1,
        start="2023-05-14T00:00:00+02:00",
        duration="PT11H",  # this changed to bypass the "anti-repeated call feature"
        soc_unit="MWh",
        soc_at_start=0.212,
        soc_targets=[],
        consumption_price_sensor=2,
        production_price_sensor=2,
    )
    await client.trigger_storage_schedule(
        sensor_id=1,
        start="2023-05-14T00:00:00+02:00",
        duration="PT12H",
        soc_unit="MWh",
        soc_at_start=0.213,  # change something else
        soc_targets=[],
        consumption_price_sensor=2,
        production_price_sensor=2,
    )

client = Client(
<EMAIL_ADDRESS>    password="admin",
    host="localhost:5000",
)

asyncio.run(my_script())

First terminal
Bash terminal for setting up and running the server:
make clean-db db_name=toy db_user=toy
flexmeasures add toy-account
flexmeasures add user --username admin --email<EMAIL_ADDRESS>--account-id 2 --roles=admin
# password: admin
flexmeasures run

Second terminal
Bash terminal for running a worker:
flexmeasures jobs run-worker --queue scheduling

Third terminal
Bash terminal for running the client:
python test_client.py

Thanks for such a detailed description!
Running through this locally, I encountered the same error both in Python 3.9.16 and Python 3.10.6 (Postgesql  v12.14 and v13.11 both showing the same).
Trying with @Flix6x, we confirmed the following:

Creating a new sensor object after calling compute fixes the issue but feels hacky.
Expunging all the objects (current_app.db.session.expunge_all()) at the end of the job fixes the issue.

As a side note, @Flix6x proposed to add the decorator as_transaction, as well, in order to rollback in case of failure.

We commented out persist_flex_model to test if it might be the cause, but the error persisted.
We tried to disable lazy loading of attributes in SensorIdField but didn't change the outcome.

New ideas:

DB Isolation Levels : SO post talking about this and why expunge might be solving our issue
Comment by a guy called Ivan Velichko in this SO answer that resembles our problem.

In the process of using the new command flexmeasures add report multiple times, I encountered this same error:
Can't attach instance <Sensor at 0x7ff5ee5208b0>; another instance with key (<class 'flexmeasures.data.models.time_series.Sensor'>, (58,), None) is already present in this session.

As in the scheduling, running the command works fine in the first try but fails in subsequent calls, for the same time period.
Similar to the scheduling case, I tried to expunge all the objects, but it didn't fix it :(
To explore this issue, I run a step-by-step debugging to check the status of the sensor instance object.
To check the status of the sensor instance, we can use the following:
from sqlalchemy import inspect
inspect(sensor).detached

The first report Sensor instance is created in the custom field SensorIdField, as the attribute --sensor-id is deserialized into a proper Sensor object. Inside the _deserialize method, the instance is attached to the session and we can retrieve the beliefs using backreference:
>> sensor.beliefs
[<TimedBelief: at 202...3.463149)>, <TimedBelief: at 202...3.463149)>, ...]

After leaving the _deserialize method, the report Sensor instance is detached, and we can no longer get the beliefs using backref:
>> sensor.beliefs

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/victor/Work/Seita/flexmeasures/venv/lib/python3.10/site-packages/sqlalchemy/orm/attributes.py", line 482, in __get__
    return self.impl.get(state, dict_)
  File "/home/victor/Work/Seita/flexmeasures/venv/lib/python3.10/site-packages/sqlalchemy/orm/attributes.py", line 942, in get
    value = self._fire_loader_callables(state, key, passive)
  File "/home/victor/Work/Seita/flexmeasures/venv/lib/python3.10/site-packages/sqlalchemy/orm/attributes.py", line 978, in _fire_loader_callables
    return self.callable_(state, passive)
  File "/home/victor/Work/Seita/flexmeasures/venv/lib/python3.10/site-packages/sqlalchemy/orm/strategies.py", line 863, in _load_for_state
    raise orm_exc.DetachedInstanceError(
sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <Sensor at 0x7f1b9db976a0> is not bound to a Session; lazy load operation of attribute 'beliefs' cannot proceed (Background on this error at: https://sqlalche.me/e/14/bhk3)

With this in consideration, the solution is to reattach the report Sensor Instance to the session in this by either of the following options:

db.session.add(sensor)
result.sensor = Sensor.query.get(sensor.id)

Two thoughts:

Does removing the backref fix the issue? Maybe we don't need it, because we usually use sensor.search_beliefs (because we don't typically need to fetch all beliefs for a sensor).
Could we extend theSensorIdField with some post-deserialization method, where we add the sensor to the session?

Instead, a more DRY solution is to add in the sensor = Sensor.query.get(sensor.id) inside the TimedBelief class constructor.
Tested this solution for both cases and it works :fireworks:


Updated head-of-house-neumann SpriteSheet File Name

For issue #90

Does this need a corresponding update in a reference somewhere?

Does this need a corresponding update in a reference somewhere?

I don't believe so! Originally, the image for the sprite wasn't loading. I updated the filename based on the fileName property in the config file, which fixed it for me after reloading the level it's in


Fixed the reinstallation of bun

All Submissions:

[x] Have you followed the guidelines stated in CONTRIBUTING.md file?
[x] Have you checked to ensure there aren't any other open Pull Requests for the desired changed?

Type of change
Please delete options that are not relevant.

[x] Bug fix (non-breaking change which fixes an issue)

Description:
Add a check_bun_version() function to check if the version is >= than the required one. Then I used that function to check the prerequisit on the install_bun() function.
Closes #2753

Closing as same behavior is done through #4006


Add conda to the list of requirements to search for

If conda is not in the list of requirements, the scripts exit too early in case pyenv is not installed and the module is configured with:
zstyle ':prezto:module:python' skip-virtualenvwrapper-init 'on'
zstyle ':prezto:module:python' conda-init 'on'
Tested on Amazon EC2 Linux for Deeplearning AMI 47.0 and MacOS 11.4

Thank you @tblaschke ✨


Eliminate all uses of os.system(), shell=True

Removing from draft insofar as this has given a clean bill of health during static checking; however, insofar as I'm not locally set up to use all the code it touches, caution / a test pass is urged when merging.

Awesome, thanks a lot. This seems all good except for the inline comment I added.

Thanks a lot, the changes to lmql.stream should be okay.


Fix ConcurrencyTest to permit jvm to connect using agent in JDK11.

Connecting to a JVM agent for the same process is no longer allowed in JDK11.
The test suite is attempting to do this for org.commonjava.indy.mem.data.ConcurrencyTest and causes an IOException.

Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.346 sec - in org.commonjava.indy.mem.data.MemoryRepositoryManagementTest
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.371 sec <<< FAILURE! - in org.commonjava.indy.mem.data.ConcurrencyTest
org.commonjava.indy.mem.data.ConcurrencyTest(org.commonjava.indy.mem.data.ConcurrencyTest)  Time elapsed: 0.371 sec  <<< ERROR!
java.io.IOException: Can not attach to current VM
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.412 sec - in org.commonjava.indy.mem.data.MemoryGroupManagementTest
Results :
Tests in error:
ConcurrencyTest.org.commonjava.indy.mem.data.ConcurrencyTest » IO Can not atta...
Tests run: 15, Failures: 0, Errors: 1, Skipped: 0

PR #1547 has been created to fix this.

The PR has been merged in.


Update sQLT phenotypeId field in V2D credset

In the v2d_credset dataset there is a field called phenotype_id. The API expects this to be a ENSG ID, and is used to map between datasets. This is incorrect and it results in us dropping ~25m of ~60m records. sQTLs only account for a portion of this data-loss, with the remainder caused by a more general issue (ticket pending). In the sQTL data this field includes additional metadata regarding the splice; for example phenotypeId: ‘chr9:13110740:13112024:clu_56974:ENSG00000107186’.
When the API tries to resolve this it cannot find a gene with that ID and does not return the expected data.
This problem relates to opentargets/issues#2682 regarding data loss, and impacts the FE development work (see opentargets/issues#2623).
The proposed solutions for both of these tickets work around the issue, rather than addressing the problem. The phenotype_id field should not be relied upon to contain an ENSG ID. An additional column needs to be added to the credset data called gene_id.
A new field needs to be added by the genetics team in preparation of the credset dataset geneId which includes the ENSG ID extracted from the sqtl entries, eg.
{
"phenotype_id":  "chr9:13110740:13112024:clu_56974:ENSG00000107186"
"gene_id": "ENSG00000107186"
}

We would expect the field gene_id never to be null for qtl studies, and always null for gwas studies.
The FE can then choose how this information is displayed, whether all the sQTLs or just the most significant are displayed.
Tasks

[ ] Update data generation steps to include the new field gene_id and update the phenotype_id field so that only the field gene_id is present. (Jack)
[ ] Update the SQL loading scripts to include new fields (somewhere here) (Jarrod)
[ ] Update the API to include a new field to expose a field geneId of type Option[String] (Ricardo)
[ ] Update the API to change parameter phenotypeId to geneId on qtl enpoints, eg. qtlRegional
[ ] Update the API to query qtl data using the field geneId. (see components/Backend.scala method qtlCredibleSet)
[ ] [Optional] Update the API to log errors when there is a database error. It currently silently falls back to an empty list with no notification.
[ ] Any relevant FE work to use the new field to display sQTL metadata. (Ricardo)

There should be no update necessary to the ETL, as this data is passed through directly from the data team to the database. This might fall under some ETL QA though where we want to confirm all the mappings are valid.
Background
To unblock the API work I have executed the following query against the development database to create the necessary new field for the API to query:
ALTER TABLE v2d_credset
ADD COLUMN IF NOT EXISTS gene_id Nullable(String)
ALTER TABLE v2d_credset
UPDATE gene_id = phenotype_id
WHERE startsWith(phenotype_id, 'ENSG')

Notes

We're using lower camel case in the API, and snake case in the data / DB. This it to ensure consistency with current field contexts.

Acceptance tests
Data validation

[ ] The query select gene_id from v2d_credset where gene_id is not null returns valid results.
[ ] select count(gene_id) from v2d_credset where data_type = 'gwas' and gene_id is not null returns 0
[ ] The query select count(gene_id) from v2d_credset where startsWith(gene_id, 'chr') returns 0
[ ] select count(gene_id) from v2d_credset where startsWith(gene_id, 'ENSG) returns the same value as select count() from v2d_credset where data_type != 'gwas'

API validation

[ ] The following query returns a valid entry:

query sqtl_query {
  qtlCredibleSet(studyId: "GTEx-sQTL",
    variantId: "19_218383_G_T",
    geneId: "ENSG00000105556",
    bioFeature: "Heart_Atrial_Appendage"){
		spliceId
  }
}

For the time being the credible set (pre ETL) has been patched in the same way as reported in #2670 for coloc.
New output: gs://genetics-portal-dev-staging/finemapping/220224_merged/credset
Old output: gs://genetics-portal-dev-staging/finemapping/220224_merged/credset_obsolete
The snippet:
credset_path = 'gs://genetics-portal-dev-staging/finemapping/220224_merged/credset_obsolete'
phenotype_gene_path = 'gs://ot-team/dochoa/phenotype_id_gene_luts/'

credset = spark.read.json(credset_path)
phenotype_gene_lut = spark.read.csv(phenotype_gene_path, sep='\t', header=True).select(F.col('phenotype_id').alias('right_phenotype_id'), F.col('gene_id').alias('right_gene_id'))

credset_fix = (
    credset.join(
        phenotype_gene_lut,
        credset['phenotype_id'].eqNullSafe(phenotype_gene_lut['right_phenotype_id']),
        how='left',
    )
    .drop('right_phenotype_id')
    .withColumn(
        'gene_id',
        F.when(F.col('right_gene_id').isNull(), F.regexp_extract('phenotype_id', r'(ENSG\d+)', 1)).otherwise(
            F.col('right_gene_id')
        ),
    )
    .drop('right_gene_id', 'phenotype_id')
    .distinct()
)

credset_fix.repartition(200).write.option('compression', 'gzip').json('gs://genetics-portal-dev-staging/finemapping/220224_merged/credset')

Some checks:
credset.count()
>> 60090436

credset_fix.count() # Again, slight increase due to the 1:M phenotype/gene relationship
>> 60095556

credset_fix.filter(F.col('type') != 'gwas').filter(F.col('gene_id').isNull()).count()
>> 0

credset_fix.filter(F.col('type') == 'gwas').filter(F.col('gene_id').isNull()).count()
>> 5077645

credset_fix.schema == credset.schema
>> False

In this case the fix results in a different schema because we are dropping phenotype_id in favour of gene_id.
credset_fix.printSchema()
>> root
 |-- bio_feature: string (nullable = true)
 |-- is95_credset: boolean (nullable = true)
 |-- is99_credset: boolean (nullable = true)
 |-- lead_alt: string (nullable = true)
 |-- lead_chrom: string (nullable = true)
 |-- lead_pos: long (nullable = true)
 |-- lead_ref: string (nullable = true)
 |-- lead_variant_id: string (nullable = true)
 |-- logABF: double (nullable = true)
 |-- multisignal_method: string (nullable = true)
 |-- postprob: double (nullable = true)
 |-- postprob_cumsum: double (nullable = true)
 |-- study_id: string (nullable = true)
 |-- tag_alt: string (nullable = true)
 |-- tag_beta: double (nullable = true)
 |-- tag_beta_cond: double (nullable = true)
 |-- tag_chrom: string (nullable = true)
 |-- tag_pos: long (nullable = true)
 |-- tag_pval: double (nullable = true)
 |-- tag_pval_cond: double (nullable = true)
 |-- tag_ref: string (nullable = true)
 |-- tag_se: double (nullable = true)
 |-- tag_se_cond: double (nullable = true)
 |-- tag_variant_id: string (nullable = true)
 |-- type: string (nullable = true)
 |-- gene_id: string (nullable = true)


Replace Spring dependency management by a single gradle file

While working on push stuff I figure out Spring dependency management doesn't work properly with Gradle transitive dependencies.
It's ignoring the transitive dependencies and on push, it causes some dependency conflicts. Since Google Services plugin doesn't found any dependency it's going to inject it from your on.
Android dependency 'com.google.firebase:firebase-iid' has different version for the compile (11.4.2) and runtime (12.0.1) classpath.

This PR moves all our dependencies version for a single file like most of the libraries do.
Related issues:

AGDROID-799

LGTM


Everything should happen in a transaction by default

This can likely be accomplished using TransactionRepository:  https://github.com/typeorm/typeorm/blob/master/docs/transactions.md#transaction-decorators

I've added a pattern for doing transactions in this example: https://github.com/goldcaddy77/warthog/blob/main/examples/11-transactions/src/user.service.ts
I've opted not to put too much magic in because I want folks to have control over how transactions work.  Note that BaseService also supports passing in a Transaction Manager so that you can have it process within the same transaction.


Added Watermark (credits) 2

Fixed this issue as well https://github.com/SintcoLTD/music-player/pull/3

That's a pretty nice update.

That's a pretty nice update.

Thanks!


lastUpdated displays incorrect date

What version of starlight are you using?
0.13.1
What version of astro are you using?
3.6.3
What package manager are you using?
npm
What operating system are you using?
macOS
What browser are you using?
Firefox
Describe the Bug
I set lastUpdated in frontmatter to 2023-03-24:

And get March 23, 2024 as the output on the page:

But the <time> element includes the correct date.
The underlying component uses toLocaleDateString:
https://github.com/withastro/starlight/blob/64d25c125c80375ff7b2cbf749f3c35d49b8bc98/packages/starlight/components/LastUpdated.astro#L14
Which uses the user-agent's timezone. So I'm at -5 offset, so it'll always show the wrong day on my computer.
Link to Minimal Reproducible Example
http://stackblitz.com/sorry-i-can-submit-a-pr-this-is-pretty-well-described-right
Participation

[X] I am willing to submit a pull request for this issue.

Ah, good catch. Would love a PR to make this build time zone agnostic!

Opened a PR: https://github.com/withastro/starlight/pull/1170


Is it possible to use two different linking philosophies simultaneously?

I'm slowly adopting setup initialized models and looking to use the new philosophy for linking variables that @whoburg recently introduced me to so that I can use material models (e.g. compressivestress <= steel['compressiveyieldstrength'], where steel is an instance of a constraintless model).
But my overarching model is still a LinkedConstraintSet. Does it have to be one or the other? The idea of converting everything at once terrifies me, so I would like to do it piecewise, but I'm currently getting substitution conflict errors because steel != concrete.

I'm at somewhat of a crossroads here so any thoughts are much appreciated!

It doesn't have to be one or another, especially for this usecase, so I'm not sure why you're seeing that error that steel != concrete?

I think it's because the models in which the material models are referenced are themselves used in a LinkedConstraintSet, resulting in
ValueError: substitution conflict: could not link because youngsmodulus_Steel was set to 2e+11 but youngsmodulus_Concrete was set to<PHONE_NUMBER>0.0

Oh, yeah. Solutions: 1) give them different names. 2) Convert your linked models piece-by-piece to setup methods, then implement this.

You can switch away from linkedconstraintsets in a piecewise fashion, but you can't add new identically-named variables in low-level models before you've switched away.

Gotcha. Thanks!

Wait. You totally have an exclude kwarg in LinkedConstraintSet that should work here, right?

For this particular case, yup! That increases maintenance problems instead of decreasing them, though. :p

I think it's a reasonable bandaid for now. Thanks for implementing this!


Aeronautica Imperialis support

If there's any chance it would be great if it could be supported. Great work on the other system implementations!

I'll have a look.  You are the first to request Aeronautica support.


errors while using ibazel

Freshly installed bazel and ibazel and run bazel run @yarn//:yarn all seems OK exit code 0
Problem come when I want run things with ibazel like

am i doing something wrong ?

Hmm, I don't see any more clues from that output, it still dies with signal 6 and no clue why.
@gregmagolan is looking at changing the npm_install rule right now, let's debug this more after that lands, just in case we manage to fix it by accident.

Let's continue at https://github.com/bazelbuild/rules_nodejs/issues/101 where we track more closely


System purge

This PR starts a big System context rebuilt in anticipation of Inbox.
In this PR we:

Transform System aggregates into projections and post events instead of commands.
Strip system entities of their power to dispatch (or deny dispatching of) events and commands to non-event-sourced domain entities.

Currently, idempotency tests are disabled, since there are no more idempotency checks except for IdempotencyGuard for aggregates.
The System context receives not handled events (e.g. EventCausedCommand). Those will be required later for integration with trancing services.

@armiol, PTAL.

@armiol, PTAL again.


In the inspector, when looking at a GemStone proxy object, navigating to a dictionary object should always open a proxy

At the moment we either get a proxy, or a serialised instance.


changed beam decoder stopping condition

stopping condition of the beam decoder now works as follows:
-- case return only the first beam (stop_early=True): it will stop when the first beam is 'unbeatable' by the other alive beams. (basically the previous behavior)
-- case return all beams (stop_early=False): it will stop when all the first N beams are 'unbeatable' by the other alive beams.
Note that checking that the last beam (in the top N) is unbeatable implies that all the top N beams are unbeatable as well.
Note w.r.t. the test changes: I duplicated testNotGreedyBeamTwo so that there is now a version with stop_early=True and one with stop_early=False. The first checks only the first beam, the latter will check all the beams.
Other two tests changed as well: they were checking the beam output length (checking it was equal to INPUT_LENGTH + decode_length) - disabled these checks given that this is not true anymore.

Any update on this PR?

@f-lng  - It seems to fail the pylint check.
I am not exacly sure why (it doesn't say) - any help appreciated there.
I think the code is working though - all the tests are passing.
(and I am using it since long time now)
Not sure who should take a look in accepting/commenting the MR...
maybe @rsepassi ?
(I think I saw your name in the decoder code comments)

Pylint test fails because of https://travis-ci.org/tensorflow/tensor2tensor/jobs/392427948#L1727
463 beam_search._is_finished: Unused argument 'finished_in_finished'
I haven't tried to understand the code, but if the parameter is not needed it should be deleted as _is_finished is an underscore method.

thanks @martinpopel - there are  other parameters there that are not used in that method.
(and I think they cannot be removed because they are inside a tf while loop)
But I noticed their names start with unused_ , so, I renamed the finished_in_finished into unused_finished_in_finished, and it seems the pylint check is now passing.
There is now a problem with an autoencoder unit test:
https://travis-ci.org/tensorflow/tensor2tensor/jobs/397916033#L1489
but it seems a problem unrelated to this PR - see last comment here:
https://github.com/tensorflow/tensor2tensor/pull/899
(failing on the same test)

closing for CLA problems - re-opened here: https://github.com/tensorflow/tensor2tensor/pull/965


Reworked controls

Finally done! Controller movement is a little buggy if the joystick is moving fast but I will consider it as an autowalk feature for now ;)

Done

Done

Awesome work, thank you!


Fortune branch

I'm confused, you didn't change any files 🤔. What was your intention with this PR?

I'm confused, you didn't change any files 🤔. What was your intention with this PR?
I'm happy to answer any questions that you might have 😁

Hey, I was trying to update my local branch so I can make some changes. it seems I didn't go the right way?

hey @fort3,
Theres a part in the readme calledKeep your banch updated!! that should help you out mate. 🙂


feat: add toFin_shiftLeft

We choose to define the shift on natural numbers and then coercing back to fin with Fin.ofNat' to maintain defeq.
While shifting is defined on Fin in core, the type signature is Fin -> Fin -> Fin, while the type signature we want is Fin -> Nat -> Fin.
An alternative rhs using the Fin's definition of shift would be:
x.toFin <<< (Fin.ofNat' n (Nat.two_pow_pos _))

but this is not a def-eq anymore, so we choose the current implementation instead.

awaiting-review

I am skeptical of this as a @[simp] lemma. Could we add the lemma for now without it, and separately discuss the simp normal forms for toFin_X type lemmas?


Can I output the xgboost R dmatrix to file and then load it by xgboost4j ?

Environment info
Operating System:
Compiler: Centos
Package used (python/R/jvm/C++):
xgboost version used:
java : ml.dmlc.xgboost4j.0.7
R : > packageVersion("xgboost") : ‘0.4.0’
If you are using R package, please provide

sessionInfo()
R version 3.1.3 (2015-03-09)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 7 x64 (build 7601) Service Pack 1

locale:
[1] LC_COLLATE=Chinese (Traditional)_Taiwan.950  LC_CTYPE=Chinese (Traditional)_Taiwan.950
[3] LC_MONETARY=Chinese (Traditional)_Taiwan.950 LC_NUMERIC=C
[5] LC_TIME=Chinese (Traditional)_Taiwan.950
attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base
other attached packages:
[1] xgboost_0.4-0
loaded via a namespace (and not attached):
[1] chron_2.3-45     data.table_1.9.6 grid_3.1.3       lattice_0.20-30  magrittr_1.5
[6] Matrix_1.1-5     stringi_0.5-5    stringr_1.0.0    tools_3.1.3
Steps to reproduce

r : xgb.DMatrix.save(dtrain, 'xgb.DMatrix.data')
java:  DMatrix trainMat = new DMatrix("/home/stream/Desktop/stupidjava/mvn2017/HelloXgb4J
/src/data/xgb.DMatrix.data");
java will occurs :
[15:57:17] dmlc-core/include/dmlc/logging.h:235: [15:57:17] src/data/data.cc:51: Check failed: (version) == (kVersion) MetaInfo: invalid format
Exception in thread "main" ml.dmlc.xgboost4j.java.XGBoostError: [15:57:17] src/data/data.cc:51: Check failed: (version) == (kVersion) MetaInfo: invalid format
at ml.dmlc.xgboost4j.java.JNIErrorHandle.checkCall(JNIErrorHandle.java:48)
at ml.dmlc.xgboost4j.java.DMatrix.(DMatrix.java:83)
at com.myMaven.app.App.main(App.java:36)
What have you tried?

I have tried that model outputed by R can be reused in java , I'm wondering if it's possible using R

Hi , R install with xgboost using the same compile version  as java solve this problem!!


[review] Installation process

I would suggest not to be explicit about where on the system a user should install qtools. I think just changing the README so that the instructions are a bit more general could avoid confusion and some other issues. Something like this maybe? Or alternatively using setuptools/pypi.
README content:
...
Clone the repository to your local directory:
mkdir -p <path_to_installation_folder> && cd <path_to_installation_folder>
git clone https://github.com/mpurg/qtools

Add this line to your ~/.bashrc or ~/.bash_profile:
source <path_to_installation_folder>/qtools/qtools_init.sh

...

Also this line can cause issues if PATH contains hardlinks:
https://github.com/mpurg/qtools/blob/baf501d25ed532b943dc7b74cbefdc92936d26d7/qscripts-cli/qscripts_config.py#L121