chore: move entity loader utils into their own object

Why
This is the second part of https://github.com/expo/entity/pull/238. From the summary:

In a follow-up, I'll look into moving the common methods (the ones in knownLoaderOnlyDifferences) to a different utils class since it no longer makes sense for them not to be on the loader class. But to limit the scope of this PR, they are kept in the NonEnforcingEntityLoader class for now.

This PR is this follow-up. It creates a utils concept that the loaders use but also that end-users can use in rare cases (we use them in rare cases in Expo server code).
The reason for this is that it doesn't make sense to have a call of the form (what they are after #238):
AssetEntity.loader(viewerContext, queryContext).nonEnforcing().invalidateEntityAsync(asset)
Instead, this PR makes it:
AssetEntity.loader(viewerContext, queryContext).utils().invalidateEntityAsync(asset)
Before both PRs, it was:
AssetEntity.loader(viewerContext, queryContext).invalidateEntityAsync(asset)
But I think it's better to not have them on the top-level loader for cleanliness.
Test Plan
Run all tests.

I prefer AssetEntity.loader(viewerContext, queryContext).nonEnforcing().invalidateEntityAsync(asset) over AssetEntity.loader(viewerContext, queryContext).utils().invalidateEntityAsync(asset) because utils gives me no information about what is going on.
Most of the calls in www are of form AssetEntity.loader(viewerContext, queryContext).enforcing().invalidateEntityAsync(asset) anyways, so the user has to be aware of the Result concept and there is no reason to abstract it away with utils().

I think the reason I prefer them not be top-level is that they are so rarely used compared to loader methods, and shouldn't be used by normal applications except in exceptional cases. Some data from Expo backend:

~1200 instances of loader methods
~3 instances of invalidateEntityAsync calls
2 instances of invalidateFieldsAsync, both in tests

So to me the API that makes the most sense in the end (for now, we can't put enforcing loader methods on top-level yet due to implicit breaking change) is:

Top-level loader has just three methods: enforcing, withAuthorizationResults, and utils (can be named something else)
Each of those loaders only has loadX methods (no invalidateEntityAsync).

I prefer AssetEntity.loader(viewerContext, queryContext).nonEnforcing().invalidateEntityAsync(asset) over AssetEntity.loader(viewerContext, queryContext).utils().invalidateEntityAsync(asset) because utils gives me no information about what is going on.

This is a good point. "utils" in general is a pretty poor term. But in this case it's a bunch of utility methods that are both used by the loaders themselves as well as (super rarely) used by the application. They don't have a "enforcing" or "withAuthorizationResult" property (the methods in utils don't use results or throw errors or load entities), so in this case I think they belong elsewhere. And "utils" is all I got as far as creativity lol.

Most of the calls in www are of form AssetEntity.loader(viewerContext, queryContext).enforcing() anyways, so the user has to be aware of the Result concept and there is no reason to abstract it away with utils().

I think theres a slight misunderstanding in the set of methods being put in utils. It's just things like invalidateEntityAsync, but none of the load/result-based-load methods, those still live in their loaders.
So after this PR it is:

Entity.loader(vc, qc).enforcing().loadX:  enforcing loader methods
Entity.loader(vc, qc).withAuthorizationResults().loadX: non-enforcing loader methods (wraps authorization and construction errors in Result)
Entity.loader(vc, qc).utils().invalidateEntityAsync: other methods related to loaders but very rarely used outside of loaders.


retained value not incrementing

hi, this is my esp monitor reading :
when i hard boot it using EN
Build:Feb  7 2021
rst:0x1 (POWERON),boot:0xc (SPI_FAST_FLASH_BOOT)
SPIWP:0xee
mode:DIO, clock div:1
load:0x3fcd5810,len:0x438
load:0x403cc710,len:0x918
load:0x403ce710,len:0x25f4
entry 0x403cc710
E (71) esp_core_dump_flash: Incorrect size of core dump image: 81858548
Deep sleep test: Booted, retained value 0, sleep in 2 seconds
After 10 seconds this is what i get :
ESP-ROM:esp32c3-api1-20210207
Build:Feb�ESP-ROM:esp32c3-api1-20210207
Build:Feb�ESP-ROM:esp32c3-api1-20210207
Build:Feb�ESP-ROM:esp32c3-api1-20210207
Build:Feb�ESP-ROM:esp32c3-api1-20210207
Build:Feb  7 2021
this is me pressing gpio 1 :
rst:0x5 (DSLEEP),boot:0xc (SPI_FAST_FLASH_BOOT)
ESP-ROM:esp32c3-api1-20210207
Build:Feb  7 2021
rst:0x5 (DSLEEP),boot:0xc (SPI_FAST_FLASH_BOOT)
After 10 sec :
ESP-ROM:esp32c3-api1-20210207
Build:Feb  7�ESP-ROM:esp32c3-api1-20210207
Build:Feb�ESP-ROM:esp32c3-api1-20210207
Build:Feb�ESP-ROM:esp32c3-api1-20210207
Build:Feb�ESP-ROM:esp32c3-api1-20210207
i dont get any incremented value each boot.

Did you change the code to make it boot on either the timer or the GPIO? Because the sample code will always go back to deep sleep, there's a comment where you can return from the stub, and then after booting you can inspect the counter.
Another note is that this was written with IDF 4.4.x and might need small changes for IDF 5.x
(currently doing a new project with newer IDF so I should check if it needs changes)

Okay okay my bad , I think I got it , i misunderstood your code , your code doesn't print the incremented value unless you boot into the firmware using return, right ?
Another thing , is it possible to use printf in wakeup stub ?

not sure where your reply went, but to print from the stub you need to include (I think)
#include "rom/ets_sys.h"
and use ets_printf() to print, but beware that the string needs RTC_RODATA_ATTR and you will need to loop a bit to create a delay to get the string out before the deep sleep kicks in again (I did 100000 loops IIRC), and in that loop you need
REG_WRITE(TIMG_WDTFEED_REG(0), 1);
or the watchdog boots you :)

sorry for the inconvenience, i forgot to change the account and replied with a different one and later deleted it.
okay , thank you for the help .
I want to ask a different question :
why does this code https://gist.github.com/igrr/54f7fbe0513ac14e1aea3fd7fbecfeab gives this error :
/Users/so20466256/espc3/main/main.cpp:34:10: fatal error: soc/rtc_io_reg.h: No such file or directory
#include "soc/rtc_io_reg.h"
^~~~~~~~~~~~~~~~~~
compilation terminated.
this works perfectly fine in esp32 but in esp32c3 i m facing the error mentioned above.
Yes, i did try adding  set_rtc_memory_crc(); below  REG_WRITE(RTC_ENTRY_ADDR_REG, (uint32_t)&wake_stub); but i dont think it has anything to do with that.
thanks in advance

Check the IDF doc for the target you use and IDF version, they change how things are done, including header files, between targets and IDF versions.
Espressif is not so fun in this aspect, porting and upgrading has many breaking changes :(

idf example doesnt work either , bcz "#include "esp_wake_stub.h"" this file doesnt exist ,i think i thats bcz i m using esp idf 4.4 , i think i will use your example , thanks for the help.

okay , one last ques, if u r using esp idf 4.4 , can u check if this example is getting compile on your machine:
https://github.com/espressif/esp-idf/tree/a7fbf452fa181452f266b16c49063e2ff069f119/examples/system/deep_sleep_wake_stub/main

hey , sorry to disturb again , but do u know how to use adc in stub ?
I tried implementing it simply using "analogRead()" while defining LDR value using RTC_DATA_ATTR, but get this error in the terminal :
Guru Meditation Error: Core 0 panic'ed (Illegal instruction)
Core 0 register dump:
PC      : 0x420057ae  RA      : 0x500000da  SP      : 0x3fcde6f0  GP      : 0x00000000
0x420057ae: __analogRead at /Users/so20466256/Embedded_Projects/project-
name/components/arduino/cores/esp32/esp32-hal-adc.c:153
later , tried using this way "  LDR_Val = SENS.sar_meas_start1.meas1_data_sar;
SENS.sar_meas_wait2.force_xpd_sar = SENS_FORCE_XPD_SAR_PD; "
but SENS is not declared , probably bcz its not for espc3

heh, I'm not the Espressif helpline ;)
Code to be used in the stub has to run from RTC ram (which keeps power), that RAM is limited and if ADC needs lots of helpers those also need to be in RTC ram. So your best shot is to use direct register read/write and not IDF functions.
Also most powerdomains are switched off during deepsleep so you need to call esp_sleep_pd_config() to make sure ADC isn't powered down


coverage badge

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 88.81%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
746

Relevant Lines:
840

💛  - Coveralls


Reserve parking spots so no other car can steal it

While debugging #218 I discovered that cars struggle to reach their precalculated parking spot. I think it would be good if cars can reserve parking spots such that other vehicles can't steal them until the first car finally approaches it.
Related to #176

It could also open the door for 'permit holders only' / 'patrons only' / 'residents only' parking feature...?

Just been doing some pondering on parking (see #346 and #342) and I'm increasingly less sure about the 'reserve parking for every car that is driving to parking'.
First, there are issues with despawning - what if if a parking spot is reserved and the car going there never makes it? Would we end up with an infestation of 'reserved but never used' parking spaces in the city?
Second, it just seems too convenient, too perfect. Like everyone would know in advance whether they can park at destination. That's like really organised, but from gameplay perspective it feels mechanical.
So, what I propose is that this issue (#231) gets refactored in to a feature that lets user click on a building or road and say who can park there as per my comment above. The purpose being I want to increase likelihood that certain cims will be able to find parking, but I'm not guaranteeing that they find parking.
Roughly:

#346 deals with general parking needs
User can set a building or road to limit its parking to:

People visiting that building (residents, shoppers, employees, whatever)
People visiting some other building nearby (discussed in #47)
People from or travelling to a certain city/industry/campus/park district

#346 would need some way to handle that


fix: (c sshnpd) allocate threaded memory outside of the thread

- What I did

Allocated atkeys outside of the thread and are freed outside of the thread.

- How to verify it
new leak summary (after closing thread)
notice that the 585 bytes that were usually suppressed are no longer lost.
==28467== LEAK SUMMARY:
==28467==    definitely lost: 0 bytes in 0 blocks
==28467==    indirectly lost: 0 bytes in 0 blocks
==28467==      possibly lost: 272 bytes in 1 blocks
==28467==    still reachable: 121,622 bytes in 201 blocks
==28467==         suppressed: 0 bytes in 0 blocks

- Description for the changelog
fix: no longer lose 585B every time SSH session is established

running into a 16 KB leak after establishing SSH connection, converting to draft while I investigate

After establishing an SSH session with 2 fresh atSigns, we get 16KB lost.
Threaded leak summary
==68541== LEAK SUMMARY:
==68541==    definitely lost: 16,384 bytes in 2 blocks
==68541==    indirectly lost: 0 bytes in 0 blocks
==68541==      possibly lost: 272 bytes in 1 blocks
==68541==    still reachable: 121,622 bytes in 201 blocks
==68541==         suppressed: 0 bytes in 0 blocks

Main process leak summary
==68485== LEAK SUMMARY:
==68485==    definitely lost: 16,384 bytes in 2 blocks
==68485==    indirectly lost: 0 bytes in 0 blocks
==68485==      possibly lost: 0 bytes in 0 blocks
==68485==    still reachable: 0 bytes in 0 blocks
==68485==         suppressed: 0 bytes in 0 blocks

Full valgrind logs: https://gist.github.com/JeremyTubongbanua/a49a783ea400506d871239873224d1a8

After establishing an SSH session with 2 fresh atSigns, we get 16KB lost.
Threaded leak summary
==68541== LEAK SUMMARY:
==68541==    definitely lost: 16,384 bytes in 2 blocks
==68541==    indirectly lost: 0 bytes in 0 blocks
==68541==      possibly lost: 272 bytes in 1 blocks
==68541==    still reachable: 121,622 bytes in 201 blocks
==68541==         suppressed: 0 bytes in 0 blocks

Main process leak summary
==68485== LEAK SUMMARY:
==68485==    definitely lost: 16,384 bytes in 2 blocks
==68485==    indirectly lost: 0 bytes in 0 blocks
==68485==      possibly lost: 0 bytes in 0 blocks
==68485==    still reachable: 0 bytes in 0 blocks
==68485==         suppressed: 0 bytes in 0 blocks

Full valgrind logs: https://gist.github.com/JeremyTubongbanua/a49a783ea400506d871239873224d1a8

The reason for this leak is at_c related. See #328 for an upcoming fix.
Next commit will most likely be a commit hash change for atsdk so that the C Daemon will use the latest trunk hash for the memory leak fix.


chart-bar directive examples

Does anyone have an example of using the  directive? Looking to use this for a work project, but don't see any examples of how this should be used. Thanks!

Did you have a look at this?
http://jettro.github.io/c3-angular-directive/
Also in the code there is an example section with a lot of samples.

Yeah definitely. I'm trying to change the bar ratio of a chart but haven't found examples that show using the chart-bar directive anywhere. I see it for changing the ratio of a donut graph, but not for a bar graph. Sorry if I'm missing something

Ah, sorry, now I understand what you mean. Will add the sample to the examples folder, but this is the code:
<c3chart bindto-id="chart1" show-labels="true">
    <chart-column column-id="bar1"
                  column-name="Bar 1"
                  column-color="green"
                  column-values="30,200,100,400,150,250"
                  column-type="bar"/>
    <chart-bar ratio="1"/>
</c3chart>

Yeah that's definitely how I've been trying to use it. I guess I'm running into problems having those attributes take effect. I'm getting the same behavior with chart-donut not having any effect. I'll investigate this further and see what's going on. Thanks for your help.

I think I have solved why this isn't working for me.  The zip files on the directive size are for version 0.3.1. This caught me because we aren't using bower yet where I work. Should've just gotten then github code!  Anyways, just wanted to bring this to your attention.

Hmm, not sure what you mean, did I leave very old versions somewhere? Of
course not my intention, maybe I forgot something.
On Wed, Oct 28, 2015 at 3:10 AM, Kris Brown<EMAIL_ADDRESS>wrote:

I think I have solved why this isn't working for me. The zip files on the
directive size are for version 0.3.1. This caught me because we aren't
using bower yet where I work. Should've just gotten then github code!
Anyways, just wanted to bring this to your attention.
—
Reply to this email directly or view it on GitHub
https://github.com/jettro/c3-angular-directive/issues/73#issuecomment-151700139
.

--
Jettro Coenradie
http://www.gridshore.nl

The first time I got the code if was the zip file from http://jettro.github.io/c3-angular-directive/#download

Ah good point, will change it, thanks.
On Wed, Oct 28, 2015 at 8:17 PM, Kris Brown<EMAIL_ADDRESS>wrote:

The first time I got the code if was the zip file from
http://jettro.github.io/c3-angular-directive/#download
—
Reply to this email directly or view it on GitHub
https://github.com/jettro/c3-angular-directive/issues/73#issuecomment-151961830
.

--
Jettro Coenradie
http://www.gridshore.nl


Initial Position

Is there any way to set the initial position of the draggable element?

Please update to v1.3.0. We can use [position] to set the initial offset.

Which doesn't work at all.
Setting position let's the coordinates explode on drag. Moving the mouse 1mm and whoops, x coord sitting at 8000+.

@ventr1x the position input has an exported type definition: IPosition. Use it wisely and this inconsistent issue can be complained by the compiler.


MWPW-135488 Add homepage to query index

Add homepage to query index

Resolves: MWPW-135488

Test URLs:
N/A

Codecov Report

Merging #145 (345823c) into main (96250b4) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #145   +/-   ##
=======================================
  Coverage   97.57%   97.57%
=======================================
  Files           8        8
  Lines         702      702
=======================================
  Hits          685      685
  Misses         17       17

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Fix up Error reporting on IO.chain

Right now IO does not report an Error if the function passed does not return an IO.
It should throw inside of the chain function instead of failing when the run function does not exist.

Submitted PR#51 to address this.

Merged PR#51.
Closing.


Added a way to set segment width distribution

Content distribution for the segmented control

Fixed: The segmented control item has a fixed width at totalWidth / 6, where 6 is maximum number of segment items.
HalfFixed: The segmented control item has a width equal to totalWidth / 6, if number of segment items > 2, and totalWidth / 4 otherwise.
Flexible: The segmented control item has a width equal to totalWidth / segmentCount

Closing, the examples image layout are still broken.


parser+typechecker+tests: Support import list imports

Adds support for from <file> import {<list of names>} to import only a subset of the items declared.

I haven't looked at the actual PR yet, but please update the tests to follow the format from: 99a337b5734ac8f8c74ecdcb545dfe9cabde956d

From the discord chat:
"Something I was trying to figure out how to do was to import from a file that isn't the main file. Say something like:
// main.jakt
from foo import {Foo}
from bar import {Bar}

// foo.jakt
from bar import {Bar}

so you don't have to be the main file to be able to import from other files
does your PR support this or maybe something we could add later?"
Current plan is to look into aliasing so that we can handle situations like the above without duplicating definitions in the codebase.

@jntrnr Another thought is I can close this PR and instead add proper modules. It currently functions more like an include instead of modules.

@jntrnr Another thought is I can close this PR and instead add proper modules. It currently functions more like an include instead of modules.

If you've got an idea for how to do this, go for it! I was looking into it, but I'm going to busy for the next week and a bit, so don't have time to work on it myself. The current system was very much a hack to get things rolling.

Closing to go in a different direction to solve this issue.


Allow "rebonding" during "unbonding" period

As the title says - Would like to be able to "rebond" to the initial "unbond" period (1, 7, or 14 day) anytime after "Unbond All" is chosen and before unbonding is complete.  Obviously, after rebonding, if you unbonded again, the clock restarts for the full unbonding period...
There may be a conscious reason why this option was not baked in from the beginning of the design - but if not, would love to see this feature.

Agreed, I think this would be a great feature!

In addition to rebonding, you should be able to bond to the longer time preference without the need to unbond.
For example, should be able to bond to 14 days from 1 or 7 days without unbonding but wouldn’t be allowed to switch from 14 to 7 or 1 day without unbonding

@pysel can you link the PR to this issue please?

I think it is linked already: https://github.com/osmosis-labs/osmosis/pull/4713/
@czarcas7ic

We have decided we are not going to do this


Works fine with local-render, but does not generate correctly.

I followed the instructions and use hugo to generate index.html and other files at the end. The format did not seem right:

However, it renders perfectly by hugo server -w

How can I make the generated one look like the same as the locally rendered one?

@jcguu95 Weird! I'll take a look.
Any errors in the console when you generated?

Not of I'm aware of. I should provide the procedures I followed -- being very new to Hugo, I might have misunderstood something.
cd ~
mkdir website-local
cd website-local
hugo new site .

echo "# website-global" >> README.md
echo "public/" >> .gitignore
git init
git add .
git commit -m "first commit"

# Then manually replace `config.toml` by that in the theme package.

# We can now host it locally by running `$ hugo server -w` Read the
# message and go to the indicated local url. It should be rendered
# fine.

hugo new post/hello.md
cd content/post/
vim hello.md

# Manually change "draft: true" to "draft: false" for publishing and
# add content.

cd ../..
hugo -d . # this will generate the site
git add .
git commit -m "generated!"

# Then manually create a repo "website-global" in github and get its url.
git remote add origin https://github.com/jcguu95/website-global.git
git push origin master # Enter username and password

# Go to setting of the repo and change to master-branch for github page.

Forget about the "git" thing. I expect the procedure above should generate a good html with associating files. If that's not the case -- perhaps I missed something -- would you mind pointing it out..? I am very excited to get my hands on hugo-classics and make some sites with it :) Thank you!

Ahh -- I see what's up.
@jcguu95 You might find this interesting: https://gohugo.io/hosting-and-deployment/hosting-on-netlify/
Netlify is how I host my personal website. It's great! Free, very fast CDN that works well with Hugo.
The pages you've generated need to be built and served. Clicking on the HTML and expecting to see the rendered copy won't work. Give the above guide a try. 👍

Ah! I thought Hugo generates a complete site! Is this an issue just for this theme package, or it's just like this in general in Hugo?

@jcguu95 It would be for any Hugo site. 😄


Fix links to formance.com instead of numary

Signed-off-by: Hugo Hamon<EMAIL_ADDRESS>

Thanks @hhamon for you PR ❤

Thank you @hhamon, really appreciated! 👍


fix column type of metadata to be same as filter type

Description
Fix column type of metadata to be same as filter type
Related Issue :

How Has This Been Tested?

Management - Metadata - Column Details
Check that the column type of the metadata is the same as the type of the filter.

Need additional checks?
Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[ ] I have read the CONTRIBUTING document.
[ ] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have added tests to cover my changes.

Additional Context

deploy to 3

deploy to 3


Added faq page

issue: #632
i have added the faq page and related questions
@tushargupta1504 please review and allow merge

proof:
https://github.com/user-attachments/assets/9526ae3e-5266-4393-9048-ced76cdf572e

@tushargupta1504 please review and update..

Hey @tushargupta1504 ??

@tushargupta1504 please check and update

@tushargupta1504 please update and merge


AppImage files have broken icon

created AppImages using uruntime have blank icon as compared to created with appimagetool.

what file system did you use?

If DwarFS, then this is not a uruntime problem, but the fact that the file manager (its thumbnail) does not know how to work with DwarFS to get the icon

GitKraken-10.5.0.glibc2.25-x86_64-UR.AppImage is built using --appimage-mksquashfs uruntime flag

@mmtrt
show all the actions and commands that you did to reproduce the problem.
because it's not reproducible for me.

and show your AppDir

./uruntime-appimage-squashfs-x86_64 --appimage-mksquashfs squashfs-root/ GitKraken-10.5.0.glibc2.25-x86_64-UR.AppImage -comp zstd

cat'ing the uruntime  to built appimage seem to solve the issue.


Connected buttons to new retirement flow and loading spinner

Description
Wired up buttons to new retirement flow and used next/link where previously omitted.
Fixed a <p> nesting error in the CustomizableModal to allow for formatted titles w/o nesting error
Included a spinner for while web3 is initializing. With the new wire-ups and using Link it was difficult to find a case where this spinner actually showed.
Related Ticket

Resolves #1174
Checklist

[X] I have run npm run build-all without errors
[X] I have formatted JS and TS files with npm run format-all

I'll leave the first two items for 0xMakka but yes I noticed the requirement for a beneficiary name in order to show/download the PDF when I was testing
Re: modal standardizing, the top one looks like it's from the first iteration of the retirement flow that used the pre-existing Modal component (that shouldn't be in the current retire flow? if it is, where is it?). The reviews were that it was too far off the design so I created a modal that was more customizable to fit the design (https://www.figma.com/file/I5hXFeW1DKjV7agHq7T4UZ/Portfolio?type=design&node-id=247-10947&t=O8YxVEdk72TWMjKx-0), which is the second image. Open to whatever works, but as far as I can tell the top image isn't in the current implementation(?).

@psparacino the top image was from the retire from pool listing that @ladytrekker just implemented https://github.com/KlimaDAO/klimadao/issues/1189

@jabby09 I've opened up a PR with a fix for item [1]. In this case I wasn't showing the button if the beneficiary name was missing, I've now removed this check.
With regards [2] - I've tried reproducing this but can't. Is this happening for every retirement for you, and is there anything you are doing in particular for this to happen or it's just not scrollable immediately when you load the page?

With regards [2] - I've tried reproducing this but can't. Is this happening for every retirement for you, and is there anything you are doing in particular for this to happen or it's just not scrollable immediately when you load the page?

@0xMakka @psparacino i discussed this with @Atmosfearful - when I described the situation he seemed to know what is was, something about the modal and not clearing something


Castbox Support

Request for support of scrobbling via Castbox.

hmm... not sure what to say. If Castbox does not negotiate with MediaSessionManager I will need to build custom notification code which won't be standardized and could fall apart at any time.
I will try myself though.

hmm. It may not share information correctly even though it get detect by Simple Scrobbler.

hmm. It may not share information correctly even though it is detected by Simple Scrobbler.

yeah, that's why I turned to you :c

@zorgel
If the data is sent incorrectly from Castbox to Simple Scrobbler then this is most likely and issue with Castbox and contacting their developers is your best bet.


Update apple-sso-plugin.md for rebranding

Rebrand, “Windows Account” extension to the “Microsoft Single Sign On”. And adding additional information.

@henrymbuguakiarie
Can you review the proposed changes?
When the changes are ready for publication, add a #sign-off comment to signal that the PR is ready for the review team to merge.
#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

#sign-off

@ejahja Can we resolve the merge conflicts and get this merged?
@henrymbuguakiarie Could you review plz?

@iambmelt - i resolved the merge conflicts as well as made some fixes to the validation issue.
note that PRMerger will only accept sign off from the listed 'author' - @henrymbuguakiarie . However, if you are a Microsoft employee authorized to update this content, you can send a request to<EMAIL_ADDRESS>for this PR to be merged.  Include your role, and an explanation of why the PR should be merged without approval from the listed author.

@iambmelt - i resolved the merge conflicts as well as made some fixes to the validation issue.
note that PRMerger will only accept sign off from the listed 'author' - @henrymbuguakiarie . However, if you are a Microsoft employee authorized to update this content, you can send a request to<EMAIL_ADDRESS>for this PR to be merged. Include your role, and an explanation of why the PR should be merged without approval from the listed author.

Thanks @Jak-MS -- let me ping Henry for action


Live AlphaModel Return% After Re-Deploy or Live Error

Expected Behavior
Live algorithm is running, based on the new AlphaModel framework.  Equity is $1000, Unrealized is $1.50.  Net profit is $3.00.  Return% is 0.45%.
Live Error occurs:  The maximum number of attempts for brokerage cash sync has been reached.
There is currently nothing the algo writer can do about the above live error, so he/she re-deploys the AlphaModel.
Live algorithm successfully re-deploys and shortly resumes reporting the following statistics:  Equity is $1000, Unrealized is ~$1.50.  Net profit is ~$3.00.  Return% is ~0.45%.  Where ~ means there is some allowance for market moves between the live error and the re-deploy.
Actual Behavior
Live algorithm is running, based on the new AlphaModel framework.  Equity is $1000, Unrealized is $1.50.  Net profit is $3.00.  Return% is 0.45%.
Live Error occurs:  The maximum number of attempts for brokerage cash sync has been reached.
There is currently nothing the algo writer can do about the above live error, so he/she re-deploys the AlphaModel.
Live algorithm successfully re-deploys 2 minutes after the live error and shortly resumes reporting the following statistics:  Equity is $1000, Unrealized is ~$1.50.  Net profit is $0.00.  Return% is -34.00%.  In the 2 minutes of downtime, there were no major market moves.
Potential Solution
Persist live algorithm state and use it for re-establishing statistics.  This should be done by the platform, and not in the AlphaModel.
Reproducing the Problem

Write down account details.
Start a live AlphaModel.
Stop and Re-deploy the Alpha Model.
Wait 2 minutes.
Upon successful re-deploy, the Return% will be off, and the Net Profit will be reset to 0.

System Information
Was just running Alphas on www.quantconnect.com.  (This happened on a Server 2048 and a Server 1024.)
Checklist

[ x ] I have completely filled out this template
[ x] I have confirmed that this issue exists on the current master branch
[ x ] I have confirmed that this is not a duplicate issue by searching issues

[ x ] I have provided detailed steps to reproduce the issue

This is partially related to #1040, but #1040 was more about algo writers having a spot to persist their own statistics.  This is more about core platform statistics, I think.

Hi @fcnorman thank you for the issue but this relates to the QuantConnect.com web platform not LEAN. Github issues are for issues relating to the LEAN engine ony.


Prometheus Release 2.2.0  Memory Leak ?

Proposal
Use case. Why is this important?
Nice to have' is not a good use case :)
Bug Report
What did you do?
Not thing
What did you expect to see?
using memory and releasing it
What did you see instead? Under which circumstances?
using memory and not releasing it
Environment

System information:

#uname -srm
Linux 3.10.0-327.ali2010.rc7.alios7.x86_64 x86_64

Prometheus version:

/prometheus $ prometheus --version
prometheus, version 2.2.1 (branch: HEAD, revision: 94e4a4321761f83207ad11542ee971e7d5220e80)
  build user:       root@sz-bcs-a151
  build date:       20180508-12:56:09
  go version:       go1.9.4

Alertmanager version:

/alertmanager $ alertmanager --version
alertmanager, version 0.14.0 (branch: HEAD, revision: 30af4d051b37ce817ea7e35b56c57a0e2ec9dbb0)
  build user:       root@37b6a49ebba9
  build date:       20180213-08:16:42
  go version:       go1.9.2

Prometheus configuration file:

# Source: prometheus/templates/server-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-server-0
  namespace: monitoring
  labels:
    component: "server"
    addonmanager.kubernetes.io/mode: Reconcile
data:
  prometheus.yml: |
    global:
      scrape_interval: 30s
      #scrape_timeout: 10s
      external_labels:
        zone: sz-uc

    #remote_write:
     #- url: http://remote-storage-adapter-service.monitoring.xxxx.local:9201/write

    alerting:
      alertmanagers:
      - scheme: http
        static_configs:
        - targets:
          - prometheus-alertmanager-service.monitoring.xxxx.local:9093

    rule_files:
      - "/etc/rules/*.yml"
      - "/etc/rules/alerts"

    scrape_configs:

      - job_name: 'prometheus-server'
        static_configs:
        - targets:
          - xxxx:9090
          - xxxx:9090

      - job_name: 'kubernetes-apiservers'
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        static_configs:
        - targets:
          - xxxx:6443
          - xxxx:6443
          - xxxx:6443
        metric_relabel_configs:
          - source_labels: ['__name__']
            regex: 'apiserver_request.*'
            action: keep

      - job_name: 'kubernetes-etcds'
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        static_configs:
        - targets:
          - xxxx:2379
          - xxxx:2379
          - xxxx:2379

      - job_name: 'kubernetes-controller-manager'
        scheme: http
        static_configs:
        - targets:
          - xxxx:10252
          - xxxx:10252
          - xxxx:10252

      - job_name: 'kubernetes-kube-scheduler'
        scheme: http
        static_configs:
        - targets:
          - xxxx:10251
          - xxxx:10251
          - xxxx:10251

      - job_name: 'kubernetes-state-metrics'
        scheme: http
        static_configs:
        - targets:
          - prometheus-kube-state-metrics.monitoring.xxxx.local:8085
        metric_relabel_configs:
          - action: labeldrop
            regex: instance
          - action: labeldrop
            regex: container_id
          - action: labeldrop
            regex: image_id
          - source_labels: ['__name__']
            action: drop
            regex: '(kube_persistentvolume_status_phase|kube_pod_status_phase)'

      - job_name: 'kubernetes-nodes'
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubernetes_sd_configs:
        - api_server: xxxx.local:6443
          role: node
        relabel_configs:
          - action: labelmap
            regex: __meta_kubernetes_node_label_(.+)
          - target_label: __address__
            replacement: xxxx.local:6443
          - source_labels: [__meta_kubernetes_node_name]
            regex: (.+)
            target_label: __metrics_path__
            replacement: /api/v1/nodes/${1}:4194/proxy/metrics
        metric_relabel_configs:
          - source_labels: ['__name__']
            action: keep
            regex: '(container_cpu_usage_seconds_total|container_memory.*|container_network_.*|container_fs.*|machine.*)'
          - source_labels: ['__name__']
            action: drop
            regex: container_memory_failures_total
          - source_labels: [pod_name]
            replacement: "$1"
            target_label: pod
          - action: labeldrop
            regex: id
          - action: labeldrop
            regex: name
          - action: labeldrop
            regex: beta_kubernetes_io_arch
          - action: labeldrop
            regex: beta_kubernetes_io_os

      - job_name: 'kubernetes-cadvisor'
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubernetes_sd_configs:
        - api_server: xxxx.local:6443
          role: node
        relabel_configs:
          - action: labelmap
            regex: __meta_kubernetes_node_label_(.+)
          - target_label: __address__
            replacement: xxxx.local:6443
          - source_labels: [__meta_kubernetes_node_name]
            regex: (.+)
            target_label: __metrics_path__
            replacement: /api/v1/nodes/${1}:10255/proxy/metrics
        metric_relabel_configs:
          - source_labels: ['__name__']
            regex: kubelet_volume_stats_.*
            action: keep

      - job_name: 'kubernetes-service-endpoints'
        kubernetes_sd_configs:
        - api_server: xxxx.local:6443
          role: endpoints
        relabel_configs:
          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
            action: keep
            regex: true
          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
            action: replace
            target_label: __scheme__
            regex: (https?)
          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
            regex: (.+)
          - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
            action: replace
            target_label: __address__
            regex: (.+)(?::\d+);(\d+)
            replacement: ${1}:${2}
          - action: labelmap
            regex: __meta_kubernetes_service_label_(.+)
          - source_labels: [__meta_kubernetes_namespace]
            action: replace
            target_label: kubernetes_namespace
          - source_labels: [__meta_kubernetes_service_name]
            action: replace
            target_label: kubernetes_name
        metric_relabel_configs:
          - source_labels: ['__name__']
            regex: node_hwmon_.*
            action: drop
          - source_labels: ['__name__']
            regex: http_.*
            action: drop
          - source_labels: ['__name__']
            regex: node_vmstat_.*
            action: drop

      - job_name: 'kubernetes-services'
        metrics_path: /probe
        params:
          module: [http_2xx]
        kubernetes_sd_configs:
        - api_server: xxxx.local:6443
          role: service
        relabel_configs:
          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
            action: keep
            regex: true
          - source_labels: [__address__]
            target_label: __param_target
          - target_label: __address__
            replacement: blackbox
          - source_labels: [__param_target]
            target_label: instance
          - action: labelmap
            regex: __meta_kubernetes_service_label_(.+)
          - source_labels: [__meta_kubernetes_namespace]
            target_label: kubernetes_namespace
          - source_labels: [__meta_kubernetes_service_name]
            target_label: kubernetes_name

      - job_name: 'kubernetes-pods'
        kubernetes_sd_configs:
        - api_server: xxxx.local:6443
          role: pod
        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
            action: keep
            regex: true
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
            regex: (.+)
          - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
            action: replace
            regex: (.+):(?:\d+);(\d+)
            replacement: ${1}:${2}
            target_label: __address__
          - action: labelmap
            regex: __meta_kubernetes_pod_label_(.+)
          - source_labels: [__meta_kubernetes_namespace]
            action: replace
            target_label: kubernetes_namespace
          - source_labels: [__meta_kubernetes_pod_name]
            action: replace
            target_label: kubernetes_pod_name
  rules: ""

Alertmanager configuration file:

insert configuration here (if relevant to the issue)

Logs:

insert Prometheus and Alertmanager logs relevant to the issue here

(pprof) top 20
Showing nodes accounting for 8.79GB, 93.68% of 9.38GB total
Dropped 318 nodes (cum <= 0.05GB)
Showing top 20 nodes out of 86
      flat  flat%   sum%        cum   cum%
    2.45GB 26.14% 26.14%     2.52GB 26.85%  github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*jsonDecDriver).DecodeString /root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/json.go
    0.76GB  8.10% 34.23%     0.76GB  8.10%  github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.(*decbuf).uvarintStr /root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/encoding_helpers.go
    0.69GB  7.40% 41.63%     0.69GB  7.40%  github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.newReader.func2 /root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/index.go
    0.69GB  7.38% 49.00%     2.74GB 29.23%  github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.fastpathT.DecSliceStringV /root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/fast-path.generated.go
    0.57GB  6.06% 55.06%     0.85GB  9.02%  github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.codecSelfer1234.decSliceNodeCondition /root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go
    0.50GB  5.32% 60.39%     3.24GB 34.55%  github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.codecSelfer1234.decSliceContainerImage /root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go
    0.47GB  5.02% 65.40%     0.74GB  7.85%  github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.(*Reader).readSymbols /root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/index.go
    0.43GB  4.61% 70.02%     0.45GB  4.84%  github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.codecSelfer1234.decResourceList /root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go
    0.37GB  3.92% 73.94%     0.37GB  3.92%  github.com/prometheus/prometheus/pkg/labels.(*Builder).Labels /root/gopath/src/github.com/prometheus/prometheus/pkg/labels/labels.go
    0.30GB  3.24% 77.18%     0.30GB  3.24%  github.com/prometheus/prometheus/pkg/textparse.(*Parser).Metric /root/gopath/src/github.com/prometheus/prometheus/pkg/textparse/parse.go
    0.23GB  2.46% 79.64%     0.23GB  2.46%  github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*decbuf).uvarintStr /root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/encoding_helpers.go
    0.23GB  2.44% 82.08%     0.46GB  4.90%  github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*walReader).decodeSeries /root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/wal.go
    0.21GB  2.29% 84.37%     0.21GB  2.29%  reflect.unsafe_New /root/working/go/src/runtime/malloc.go
    0.18GB  1.92% 86.29%     0.18GB  1.92%  github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/chunkenc.NewXORChunk /root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/chunkenc/xor.go
    0.18GB  1.92% 88.21%     0.26GB  2.73%  github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.fastpathT.DecMapStringStringV /root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/fast-path.generated.go
    0.15GB  1.61% 89.82%     0.15GB  1.61%  github.com/prometheus/prometheus/retrieval.(*scrapeCache).trackStaleness /root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go
    0.11GB  1.22% 91.05%     0.11GB  1.22%  github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.newMemSeries /root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go
    0.10GB  1.02% 92.07%     0.10GB  1.02%  github.com/prometheus/prometheus/retrieval.(*scrapeCache).addRef /root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go
    0.08GB   0.9% 92.97%     0.08GB   0.9%  github.com/prometheus/prometheus/pkg/pool.(*BytesPool).Get /root/gopath/src/github.com/prometheus/prometheus/pkg/pool/pool.go
    0.07GB  0.71% 93.68%     0.07GB  0.71%  github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*Decoder).string /root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/decode.go (inline)

goroutine-stacks.txt

I think I’ve seen a memory profile like that before, can you just to try remove the scrape configs with “role: node” and see if it disappears? (I’ve seen leaks somehow related to node objects being deserialized, never found the reason for it though)

@brancz just remove the role: node? (--config.file=/etc/config/prometheus.yml): parsing YAML file /etc/config/prometheus.yml: role missing (one of: pod, service, endpoints, node) .
I remove the kubernetes-service-endpoints  target which  has  about 150 node-exporter  and it's fine.
Is it the to many retrieval.(*scrapePool)   to cause OOM ?

@krasi-georgiev I've never encountered memory leak.
@piaoyu Could you increase your prometheus memory limit to see if it grows unboundedly?

@cofyc   Looks like it also memory  grows  always .

goroutine.txt

Can you confirm, that when you do kubectl get pod <your-prometheus-pod> -oyaml, that it actually says that your container has been OOMKilled?
In my case the graph of the container usage looks similar but I think we're just seeing mmap memory being freed.
Concretely I've been running a setup where the container_memory_usage_bytes looks similar to yours. My Prometheus container has a memory limit of 1Gi:

However heap usage (reported by the go runtime) seems stable and not ever growing:

Also the process has been running for >24h, so besides Kubernetes not reporting an OOMKill to me, the process itself is also confirming that.
Could you try these queries and validate this against your setup?

@brancz

container_memory_usage_bytes  not the  real memory but container_memory_rss does

I've also just found this bug: works fine without the k8 discovery, but as soon as I add that it, it OOM.

I recently upgraded from 2.1 to 2.2.1 and am getting the same thing. I tried removing the k8 cluster from the config but the same thing ends up happening. It uses up all the memory, then starts going into swap before OOM.

@krasi-georgiev  I check the logs it really gets OOM killed.

I've also experienced the same problem and it has stopped when I removed the k8 cluster from the config.

I'm looking into this, and trying to reproduce this issue in my cluster.
/prometheus $ prometheus --version
 prometheus, version 2.2.1 (branch: HEAD, revision: 94e4a4321761f83207ad11542ee971e7d5220e80)
  build user:       root@sz-bcs-a151
  build date:       20180508-12:56:09
  go version:       go1.9.4

@piaoyu From output of prometheus --version, you are using a custom build, could you try to use quay.io/prometheus/prometheus:v2.2.1 instead? If it still get OOM killed, let me know.

@grantstephens Could you provide output of prometheus --version?

prometheus, version 2.2.1 (branch: HEAD, revision: bc6058c81272a8d938c05e75607371284236aadc)
  build user:       root@149e5b3f0829
  build date:       20180314-14:15:45
  go version:       go1.10

Let me know if you need any more information.

@djsly it would helpful if you could share:

The exact Prometheus version.
System logs about the OOM events.
Prometheus logs.
graphs of the following metrics

container_memory_usage_bytes for the Prometheus pod(s).
prometheus_tsdb_head_series
rate(prometheus_tsdb_head_series_created_total[1m])

for everyone in this issue I just realised that if you are testing the binary from the release page this doesn't include the refactoring so make sure you build a new one from master or the 2.2 branch.
https://github.com/prometheus/prometheus/tree/release-2.2
I can build the binary for anyone willing to test it.

@piaoyu did you read my comment above?
quay.io/prometheus/prometheus:v2.2.1 is before the refactoring which is suppose to fix the problem

v2.3.0 is out so try that.
quay.io/prometheus/prometheus:v2.3.0

@krasi-georgiev  OK .  I will try quay.io/prometheus/prometheus:v2.3.0

My prometheus was sucking up a ton of memory and disk. More than I expected. I had storage.tsdb.retention set to 10 days, and gave the prometheus docker 50G disk and 4G of memory. It wasn't happy. I tried upgrading to 2.3.0 and I think that made things even worse.
Doing more research I found https://www.robustperception.io/which-are-my-biggest-metrics/ which pointed out that I had a custom metric with 166070 entries. I think I will fix my metrics and see if that fixes my prometheus stability issue. Adding it here in case others have this issue too.
topk(10, count by (__name__, job)({__name__=~".+"}))

Element
Value

containerpilot_events{job="consul"}
166070

consul_catalog_service_node_healthy{job="consul"}
150

consul_health_service_status{job="consul"}
141

go_gc_duration_seconds{job="consul"}
105

ex
containerpilot_events{job="consul"}

Element
Value

containerpilot_events{code="Metric",instance="<IP_ADDRESS>:9090",job="consul",service="containerpilot",source="tb_process_load|0.000024106811388486265"}
1

containerpilot_events{code="Metric",instance="<IP_ADDRESS>:9090",job="consul",service="containerpilot",source="tb_process_load|0.000024114556647921814"}
1

containerpilot_events{code="Metric",instance="<IP_ADDRESS>:9090",job="consul",service="containerpilot",source="tb_process_load|0.000024234685599017417"}
1

containerpilot_events{code="Metric",instance="<IP_ADDRESS>:9090",job="consul",service="containerpilot",source="tb_process_load|0.000024337671686951137"}
1

@dfredell thanks for the report, but can you open a new issue as this is getting to long.
@piaoyu can you please confirm if 2.3 fixed the problem for you so we can close this one?

@krasi-georgiev  seem like 2.3 fixed the problem for 12hours  memory watch.

Nice , closing this as resolved!
Feel free to reopen or file a new one if you notice any other problems.
Thanks for the team work :+1:


Where should pre-trained models put

Hello,Thanks to your excellent work,Could you tell me the project data structure,so I can download pre-trained models and put them to right place.

Hello, you can place the downloaded models anywhere you want. There is a parameter which specify the path to the pre-trained model when running training or inference.
If you want to continue training for the pre-trained model the model,opt,ema all three files are required and also the name of the files should be adjusted in the following way {name}_000000.{sufix}, the 000000 represents the current iteration.
If you need any additional help, feel free to ask.
Link for paper related to this code


BUG: optimize.shgo: ignores a constraint

Describe your issue.
How is it that the constraint x[0]+x[2]-1=0 is not respected in the returned 4-uple?
Reproducing Code Example
from scipy import optimize
from math import log

bounds = ((0, 1), (0, 1), (0, 1), (0, 1))

def finite_resources(x):
    return x[0] + x[2] - 1

cons = ({'type': 'eq', 'fun': finite_resources})

def global_satisfaction(x):
    return -(log(1+x[0])/(1+x[1]*10000) + log(1+x[2])/(1+x[3]*10000))

results = dict()
print(optimize.shgo(global_satisfaction, bounds, constraints=cons))

Error message
message: Optimization terminated successfully.
 success: True
     fun: -1.3862943611198906
    funl: [-1.386e+00]
       x: [ 1.000e+00  0.000e+00  1.000e+00  0.000e+00]
      xl: [[ 1.000e+00  0.000e+00  1.000e+00  0.000e+00]]
     nit: 1
    nfev: 18
   nlfev: 5
   nljev: 1
   nlhev: 0

SciPy/NumPy/Python version and system information
warnings.warn("Install `pyyaml` for better output", stacklevel=1)
{
  "Compilers": {
    "c": {
      "name": "gcc",
      "linker": "ld.bfd",
      "version": "10.2.1",
      "commands": "cc"
    },
    "cython": {
      "name": "cython",
      "linker": "cython",
      "version": "3.0.11",
      "commands": "cython"
    },
    "c++": {
      "name": "gcc",
      "linker": "ld.bfd",
      "version": "10.2.1",
      "commands": "c++"
    },
    "fortran": {
      "name": "gcc",
      "linker": "ld.bfd",
      "version": "10.2.1",
      "commands": "gfortran"
    },
    "pythran": {
      "version": "0.16.1",
      "include directory": "../../tmp/pip-build-env-znxdftlb/overlay/lib/python3.12/site-packages/pythran"
    }
  },
  "Machine Information": {
    "host": {
      "cpu": "x86_64",
      "family": "x86_64",
      "endian": "little",
      "system": "linux"
    },
    "build": {
      "cpu": "x86_64",
      "family": "x86_64",
      "endian": "little",
      "system": "linux"
    },
    "cross-compiled": false
  },
  "Build Dependencies": {
    "blas": {
      "name": "scipy-openblas",
      "found": true,
      "version": "0.3.27.dev",
      "detection method": "pkgconfig",
      "include directory": "/opt/_internal/cpython-3.12.4/lib/python3.12/site-packages/scipy_openblas32/include",
      "lib directory": "/opt/_internal/cpython-3.12.4/lib/python3.12/site-packages/scipy_openblas32/lib",
      "openblas configuration": "OpenBLAS 0.3.27.dev DYNAMIC_ARCH NO_AFFINITY Zen MAX_THREADS=64",
      "pc file directory": "/project"
    },
    "lapack": {
      "name": "scipy-openblas",
      "found": true,
      "version": "0.3.27.dev",
      "detection method": "pkgconfig",
      "include directory": "/opt/_internal/cpython-3.12.4/lib/python3.12/site-packages/scipy_openblas32/include",
      "lib directory": "/opt/_internal/cpython-3.12.4/lib/python3.12/site-packages/scipy_openblas32/lib",
      "openblas configuration": "OpenBLAS 0.3.27.dev DYNAMIC_ARCH NO_AFFINITY Zen MAX_THREADS=64",
      "pc file directory": "/project"
    },
    "pybind11": {
      "name": "pybind11",
      "version": "2.12.0",
      "detection method": "config-tool",
      "include directory": "unknown"
    }
  },
  "Python Information": {
    "path": "/opt/python/cp312-cp312/bin/python",
    "version": "3.12"
  }
}

If you lower the coefficients of x[1] and x[3] in global_satisfaction, shgo succeeds with the constraints satisfied.  For example, with
def global_satisfaction(x):
    c = 5000
    return -(log(1+x[0])/(1 + c*x[1]) + log(1+x[2])/(1 + c*x[3]))

shgo succeeds and the equality constraint is satisfied (to within a small tolerance).
I haven't looked into the SHGO alorithm, so I don't know why increasing the sensitivity of x[1] and x[3] would cause it to return an incorrect result.

It seems that shgo only works with inequality constraints. If you specify the 'eq' constraint it will still interpret it as an inequality. In your example x[0] + x[2] - 1 = 1 + 1 - 1 = 1 > 0 and the algorithm thinks everything is fine. If you try to circumvent this behaviour by specifying two inequality constraints (g >= 0 and g <= 0) then shgo will most likely not get any valid points from the sampling stage and fail.


Start kwok controller with static pod when the runtime is kind

What type of PR is this?
/kind feature
What this PR does / why we need it:
Which issue(s) this PR fixes:

Fixes #98
Special notes for your reviewer:
Does this PR introduce a user-facing change?

Run kwok controller with static pod when the runtime is kind by kwokctl create cluster

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

Here is the test verification process
➜  ~ kwokctl create cluster --name kwok --runtime kind --kind-node-image kindest/node:v1.21.1
Creating cluster                                                    cluster=kwok
Starting cluster                                                    cluster=kwok
Creating cluster "kwok-kwok" ...
 ✓ Ensuring node image (kindest/node:v1.21.1) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-kwok-kwok"
You can now use your cluster with:

kubectl cluster-info --context kind-kwok-kwok

Have a nice day! 👋
Cluster is ready                                                    cluster=kwok
You can now use your cluster with:

    kubectl config use-context kwok-kwok

Thanks for using kwok!
➜  ~ kubectl config use-context kwok-kwok
Switched to context "kwok-kwok".
➜  ~ k -n kube-system get po | grep kwok-controller
kwok-controller-kwok-kwok-control-plane           1/1     Running   0          33s
➜  ~ >....
      key: kwok.x-k8s.io/node
      value: fake
status:
  allocatable:
    cpu: 32
    memory: 256Gi
    pods: 110
  capacity:
    cpu: 32
    memory: 256Gi
    pods: 110
  nodeInfo:
    architecture: amd64
    bootID: ""
    containerRuntimeVersion: ""
    kernelVersion: ""
    kubeProxyVersion: fake
    kubeletVersion: fake
    machineID: ""
    operatingSystem: linux
    osImage: ""
    systemUUID: ""
  phase: Running
EOF
node/kwok-node-0 created
➜  ~ kubectl get node -o wide
NAME                      STATUS                     ROLES                  AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE       KERNEL-VERSION     CONTAINER-RUNTIME
kwok-kwok-control-plane   Ready,SchedulingDisabled   control-plane,master   91s   v1.21.1   <IP_ADDRESS>    <none>        Ubuntu 21.04   5.10.76-linuxkit   containerd://1.5.2
kwok-node-0               Ready                      agent                  9s    fake      <IP_ADDRESS>    <none>        <unknown>      <unknown>          <unknown>
➜  ~ >....
  template:
    metadata:
      labels:
        app: fake-pod
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: type
                    operator: In
                    values:
                      - kwok
      # A taints was added to an automatically created Node.
      # You can remove taints of Node or add this tolerations.
      tolerations:
        - key: "kwok.x-k8s.io/node"
          operator: "Exists"
          effect: "NoSchedule"
      containers:
        - name: fake-container
          image: fake-image
EOF
deployment.apps/fake-pod created
➜  ~ k get po
NAME                        READY   STATUS    RESTARTS   AGE
fake-pod-759cdcff4c-8hzjh   1/1     Running   0          4s
fake-pod-759cdcff4c-9r2rz   1/1     Running   0          4s
fake-pod-759cdcff4c-cfbx4   1/1     Running   0          4s
fake-pod-759cdcff4c-ckscn   1/1     Running   0          4s
fake-pod-759cdcff4c-fgmnh   1/1     Running   0          4s
fake-pod-759cdcff4c-l75vx   1/1     Running   0          4s
fake-pod-759cdcff4c-qkp85   1/1     Running   0          4s
fake-pod-759cdcff4c-qts55   1/1     Running   0          4s
fake-pod-759cdcff4c-w5mnr   1/1     Running   0          4s
fake-pod-759cdcff4c-w5wmn   1/1     Running   0          4s

/ok-to-test

@wzshiming I tested it and when docker exec -it kwok-kwok-control-plane mv /etc/kubernetes/manifests/kube-controller-manager.yaml /etc/kubernetes/manifests/kube-controller-manager.yaml.bak is executed, it deletes the old pod and starts a new pod
➜  ~ k -n kube-system get po
NAME                                              READY   STATUS    RESTARTS   AGE
coredns-558bd4d5db-c4sxf                          0/1     Pending   0          15m
coredns-558bd4d5db-x7wlq                          0/1     Pending   0          15m
etcd-kwok-kwok-control-plane                      1/1     Running   0          15m
kindnet-97jkb                                     1/1     Running   0          13m
kindnet-zrtg6                                     1/1     Running   0          15m
kube-apiserver-kwok-kwok-control-plane            1/1     Running   0          15m
kube-controller-manager-kwok-kwok-control-plane   1/1     Running   0          15m
kube-proxy-2f2mg                                  1/1     Running   0          15m
kube-proxy-hclf7                                  1/1     Running   0          13m
kube-scheduler-kwok-kwok-control-plane            1/1     Running   0          15m
kwok-controller-kwok-kwok-control-plane           1/1     Running   0          15m
➜  ~ docker exec -it kwok-kwok-control-plane mv /etc/kubernetes/manifests/kube-controller-manager.yaml /etc/kubernetes/manifests/kube-controller-manager.yaml.bak
➜  ~ k -n kube-system get po
NAME                                              READY   STATUS    RESTARTS   AGE
coredns-558bd4d5db-c4sxf                          0/1     Pending   0          17m
coredns-558bd4d5db-x7wlq                          0/1     Pending   0          17m
etcd-kwok-kwok-control-plane                      1/1     Running   0          17m
kindnet-97jkb                                     1/1     Running   0          16m
kindnet-zrtg6                                     1/1     Running   0          17m
kube-apiserver-kwok-kwok-control-plane            1/1     Running   0          17m
kube-controller-manager-kwok-kwok-control-plane   1/1     Running   0          29s
kube-proxy-2f2mg                                  1/1     Running   0          17m
kube-proxy-hclf7                                  1/1     Running   0          16m
kube-scheduler-kwok-kwok-control-plane            1/1     Running   0          17m
kwok-controller-kwok-kwok-control-plane           1/1     Running   0          17m

/test pull-kwok-e2e-test-main

/approve
/lgtm


add scroll button

Solves issue #32

Added button to scroll to the top of page.

Images

Hi @grace-omotoso, I've renamed the branch


Playbook build fail with python3: command not found

I'm building the the playbook and it failed with the below error:
TASK [install python] **********************************************************
Thursday 18 November 2021  11:07:23 -0500 (0:00:00.024)       0:00:00.024 *****
skipping: [default]

TASK [gather facts] ************************************************************
Thursday 18 November 2021  11:07:23 -0500 (0:00:00.059)       0:00:00.084 *****
fatal: [default]: FAILED! => {"changed": false, "module_stderr": "Shared connection to <IP_ADDRESS> closed.\r\n", "module_stdout": "/bin/sh: python3: command not found\r\n", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 127}

PLAY RECAP *********************************************************************
default                    : ok=0    changed=0    unreachable=0    failed=1    skipped=1    rescued=0    ignored=0

Thursday 18 November 2021  11:07:23 -0500 (0:00:00.761)       0:00:00.846 *****
===============================================================================
gather facts ------------------------------------------------------------ 0.76s
install python ---------------------------------------------------------- 0.06s
Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.

the section install python in bootstrap.yml seem to be skipped, I have Mac OSX Monterey in my laptop, I'm not sure if it has something to do with this. I'm wondering if anyone has the same issue  ?
Vagrant 2.2.14

I'm running Monterey and am not experiencing this.
I see the default python in my $PATH is /usr/bin/python which is Python 2.7, and I have a python3 installed via Home-brew at /user/local/bin/python3.

My python setup is also the same as well:

... Aslo I see this warning as well, maybe the anisible version is not compatible  ?
Vagrant gathered an unknown Ansible version:

ansible [core 2.11.6]
  config file = /Users/huynhng7/development/islandora-playbook/ansible.cfg
  configured module search path = ['/Users/huynhng7/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/Cellar/ansible/4.8.0/libexec/lib/python3.10/site-packages/ansible
  ansible collection location = /Users/huynhng7/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.10.0 (default, Oct 22 2021, 13:39:57) [Clang 13.0.0 (clang-13<IP_ADDRESS>)]
  jinja version = 3.0.2
  libyaml = True

and falls back on the compatibility mode '1.8'.

Alternatively, the compatibility mode can be specified in your Vagrantfile:
https://www.vagrantup.com/docs/provisioning/ansible_common.html#compatibility_mode

Here are my ansible and vagrant versions:
$ ansible-playbook --version
ansible-playbook [core 2.11.6]
  config file = None
  configured module search path = ['/Users/aoneill/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/Cellar/ansible/4.8.0/libexec/lib/python3.10/site-packages/ansible
  ansible collection location = /Users/aoneill/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.10.0 (default, Oct 13 2021, 06:45:00) [Clang 13.0.0 (clang-13<IP_ADDRESS>)]
  jinja version = 3.0.2
  libyaml = True
cmtctest:islandora_install_profile_demo aoneill$ vagrant --version
Vagrant 2.2.19
cmtctest:islandora_install_profile_demo aoneill$ brew info ansible
ansible: stable 4.8.0 (bottled), HEAD
Automate deployment, configuration, and upgrading
https://www.ansible.com/
/usr/local/Cellar/ansible/4.8.0 (43,136 files, 408.6MB) *
  Poured from bottle on 2021-11-04 at 11:19:08
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/ansible.rb
License: GPL-3.0-or-later
==> Dependencies
Build: pkg-config ✔, rust ✘
Required: libyaml ✔<EMAIL_ADDRESS>✔<EMAIL_ADDRESS>✔, six ✔
==> Options
--HEAD
	Install HEAD version
==> Analytics
install: 60,377 (30 days), 160,138 (90 days), 559,344 (365 days)
install-on-request: 59,303 (30 days), 157,069 (90 days), 549,327 (365 days)
build-error: 43 (30 days)

I thought that it was testing if python was installed in the Ubuntu box, not on your host machine?

PLAY [bootstrap] ***************************************************************

TASK [install python] **********************************************************
Thursday 18 November 2021  14:16:51 -0500 (0:00:00.026)       0:00:00.026 *****
skipping: [default]

TASK [gather facts] ************************************************************
Thursday 18 November 2021  14:16:51 -0500 (0:00:00.061)       0:00:00.087 *****
fatal: [default]: FAILED! => {"changed": false, "module_stderr": "Shared connection to <IP_ADDRESS> closed.\r\n", "module_stdout": "/bin/sh: python3: command not found\r\n", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 127}

PLAY RECAP *********************************************************************
default                    : ok=0    changed=0    unreachable=0    failed=1    skipped=1    rescued=0    ignored=0

Thursday 18 November 2021  14:16:51 -0500 (0:00:00.694)       0:00:00.782 *****
===============================================================================
gather facts ------------------------------------------------------------ 0.69s
install python ---------------------------------------------------------- 0.06s
Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.

Yes, according to the log above  the task [Install python] (i looked at the bootstrap.yml) is skipping, means the install python commands in this task fail to run ?
I also run vagrant ssh to the fail built box above, try to run sudo apt-get install -y python3 and  I got :
sudo: apt-get: command not found

Maybe apt-get is not installed in the ubuntu box by default. I made only chance for the v.gui = true for Mac Monterey after I cloned the playbook

No I'm pretty sure apt-get is installed in Ubuntu by default.
The fact that it is looking for python3 makes me wonder if the test on this line should be
raw: test -e /usr/bin/python3 || (apt-get update; apt-get install -y python3;)

I changed the line to the one you suggested,  ... I still got the same error above.
(base) huynhng7@LIB-C02D38MSML85 islandora-playbook % git diff
diff --git a/Vagrantfile b/Vagrantfile
index a86f163..fd40681 100644
--- a/Vagrantfile
+++ b/Vagrantfile
@@ -23,6 +23,7 @@ $vagrantUser = "vagrant"
 Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
   config.vm.provider "virtualbox" do |v|
     v.name = "Islandora 8 Ansible"
+    v.gui = true
   end

   config.vm.hostname = $hostname
diff --git a/bootstrap.yml b/bootstrap.yml
index 067ac83..ef07a7b 100644
--- a/bootstrap.yml
+++ b/bootstrap.yml
@@ -9,7 +9,7 @@
     # python isn't installed by default on ubuntu so we need
     # to do that with the raw command before
     - name: install python
-      raw: test -e /usr/bin/python || (apt-get update; apt-get install -y python3;)
+      raw: test -e /usr/bin/python3 || (apt-get update; apt-get install -y python3;)
       register: output
       changed_when:
         - output.stdout != ""

When you vagrant ssh in to the box, what does
which python

and
which python3

return?

As below ....

(base) huynhng7@LIB-C02D38MSML85 islandora-playbook % vagrant ssh
Last login: Thu Nov 18 21:43:42 2021 from <IP_ADDRESS>
[vagrant@islandora8 ~]$ which python
/usr/bin/python
[vagrant@islandora8 ~]$ which python3
/usr/bin/which: no python3 in (/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/vagrant/.local/bin:/home/vagrant/bin)
[vagrant@islandora8 ~]$

Maybe @alxp can weigh in here, as he had the PR that switched the install from python to python3. Maybe now python3 is a requirement of ansible?

Maybe we should switch to python-is-python3 it install if the path is the issue.
raw: test -e /usr/bin/python3 || (apt-get update; apt-get install python-is-python3;)

I made the change when updating the base box to Ubuntu 20.04, who's default python interpreter is Python 3, the test is a little misaligned now I think. If I log in to a fresh box built with islandora_distro = ubuntu/focal64:
vagrant@islandora8:~$ which python
vagrant@islandora8:~$ which python3
/usr/bin/python3
There is a fairly new Ubuntu package called 'python_is_python3' which has in its package description:

In Ubuntu, all python packages use explicit python3 or python2 interpreter and do not use unversioned /usr/bin/python at all ... No packages may declare dependencies on this package.

So the test for 'python' should probably be changed to Python 3. @kylehuynh205  If you change test -e /usr/bin/python to test -e python3 how does it go?

there's a variable "ansible_python_interpreter" set in inventory/vagrant/group_vars/database.yml, the test should more appropriately use that one.

I changed as you suggested, but still got the same error. and the [install python] task still be skipped.
diff --git a/bootstrap.yml b/bootstrap.yml
index 067ac83..a28fe50 100644
--- a/bootstrap.yml
+++ b/bootstrap.yml
@@ -9,7 +9,7 @@
     # python isn't installed by default on ubuntu so we need
     # to do that with the raw command before
     - name: install python
-      raw: test -e /usr/bin/python || (apt-get update; apt-get install -y python3;)
+      raw: test -e python3 || (apt-get update; apt-get install -y python-is-python3;)
       register: output
       changed_when:
         - output.stdout != ""

In inventory/vagrant/group_vars/database.yml, I see  ansible_python_interpreter: python3, how should I set the test with test -e python3 ? sorry I'm still getting to know more with Anisible.

Oh sorry test-e needs the absolute path.
Perhaps to be proper we should do something like
test -e `which "{{ ansible_python_interpreter }} "
But failing that you can try
test -e /usr/bin/python3

But thinking about why it is skipping the step,
the condition in the task is:
      when: islandora_distro|default('') is match("ubuntu/")

Do you have something in your Vagrantfile or unix environment changing that variable?

I checked the Virtualbox settings (screenshot below) and find the box is actually centos 7 instead of ubuntu, maybe that's why it's skipping.

I have only one change in Vagrantfile which is v.gui=true for the new MacOSX Monterey.  I'm not sure how the box become Centos instead.
index a86f163..fd40681 100644
--- a/Vagrantfile
+++ b/Vagrantfile
@@ -23,6 +23,7 @@ $vagrantUser = "vagrant"
 Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
   config.vm.provider "virtualbox" do |v|
     v.name = "Islandora 8 Ansible"
+    v.gui = true
   end

   config.vm.hostname = $hostname
diff --git a/bootstrap.yml b/bootstrap.yml
index 067ac83..0cd9ef7 100644
--- a/bootstrap.yml
+++ b/bootstrap.yml

Maybe you should delete your entire islandora-playbook directory and re-clone the playbook. It seems like something has gotten out of sync for you. I'd also go into VirtualBox and delete the machine in there.

I deleted islandora-playbook and the VM in virtual box, then cloned the fresh one, but it still pull the Centos 7 when vagrant up, so i improvised a bit,
Run "vagrant ssh" then run sudo yum update and then, sudo yum install -y python3 and exit, run vagrant up --provision. the installation proceed up until :

TASK [install python] **********************************************************
Friday 19 November 2021  10:33:00 -0500 (0:00:00.029)       0:00:00.029 *******
skipping: [default]

TASK [gather facts] ************************************************************
Friday 19 November 2021  10:33:00 -0500 (0:00:00.061)       0:00:00.090 *******
fatal: [default]: FAILED! => {"changed": false, "module_stderr": "Shared connection to <IP_ADDRESS> closed.\r\n", "module_stdout": "/bin/sh: python3: command not found\r\n", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 127}

PLAY RECAP *********************************************************************
default                    : ok=0    changed=0    unreachable=0    failed=1    skipped=1    rescued=0    ignored=0

then did the same thing with vagrant ssh but sudo pip3 install pymysql, and exit continue with vagrant up --provision, and things continue proceed but stopped at:
TASK [geerlingguy.drupal : Check if site is already installed.] ****************
Friday 19 November 2021  11:31:30 -0500 (0:00:00.043)       0:05:23.107 *******
ok: [default]

TASK [geerlingguy.drupal : Configure database correctly if using PostgreSQL.] ***
Friday 19 November 2021  11:31:30 -0500 (0:00:00.498)       0:05:23.605 *******
skipping: [default]

TASK [geerlingguy.drupal : Install Drupal with drush.] *************************
Friday 19 November 2021  11:31:30 -0500 (0:00:00.028)       0:05:23.634 *******
fatal: [default]: FAILED! => {"changed": true, "cmd": ["/var/www/html/drupal/vendor/drush/drush/drush", "site-install", "standard", "-y", "--root=/var/www/html/drupal/web", "--site-name=Islandora 8", "--account-name=admin", "--account-pass=islandora"<EMAIL_ADDRESS>"delta": "0:00:00.533545", "end": "2021-11-19 16:31:31.640654", "msg": "non-zero return code", "rc": 1, "start": "2021-11-19 16:31:31.107109", "stderr": " [notice] Starting Drupal installation. This takes a while.\n [notice] Performed install task: install_select_language\n [notice] Performed install task: install_select_profile\n [notice] Performed install task: install_load_profile\n [notice] Performed install task: install_verify_requirements\n\nIn install.core.inc line 970:\n                                                                               \n  Resolve all issues below to continue the installation. For help configuring  \n   your database server, see the installation handbook, or contact your hosting provider.

The database server version 5.5  \n  .68-MariaDB is less than the minimum required version 10.3.7.

                                         \n                                                                               ", "stderr_lines": [" [notice] Starting Drupal installation. This takes a while.", " [notice] Performed install task: install_select_language", " [notice] Performed install task: install_select_profile", " [notice] Performed install task: install_load_profile", " [notice] Performed install task: install_verify_requirements", "", "In install.core.inc line 970:", "                                                                               ", "  Resolve all issues below to continue the installation. For help configuring  ", "   your database server, see the installation handbook, or contact your hosting provider.

The database server version 5.5  ", "  .68-MariaDB is less than the minimum required version 10.3.7.

                                         ", "                                                                               "], "stdout": "\n You are about to:\n * Create a sites/default/settings.php file\n * CREATE the 'drupal8' database.\n\n // Do you want to continue?: yes.                                              ", "stdout_lines": ["", " You are about to:", " * Create a sites/default/settings.php file", " * CREATE the 'drupal8' database.", "", " // Do you want to continue?: yes.                                              "]}

RUNNING HANDLER [geerlingguy.php : restart webserver] **************************
Friday 19 November 2021  11:31:31 -0500 (0:00:00.911)       0:05:24.545 *******

RUNNING HANDLER [Islandora-Devops.matomo : restart apache] *********************
Friday 19 November 2021  11:31:31 -0500 (0:00:00.000)       0:05:24.546 *******

PLAY RECAP *********************************************************************
default                    : ok=112  changed=26   unreachable=0    failed=1    skipped=83   rescued=0    ignored=0

Friday 19 November 2021  11:31:31 -0500 (0:00:00.000)       0:05:24.547 *******
===============================================================================
geerlingguy.drupal : Install dependencies with composer require (this may take a while). - 170.17s
geerlingguy.apache : Ensure Apache is installed on RHEL. --------------- 35.76s
geerlingguy.php : Ensure PHP packages are installed. ------------------- 27.04s
geerlingguy.drupal : Generate Drupal project with composer package in /tmp/composer-project (this may take a while). -- 24.81s
geerlingguy.git : Ensure git is installed (RedHat). -------------------- 14.50s
install extra packages centos ------------------------------------------- 7.82s
geerlingguy.php-mysql : Install PHP MySQL dependencies (RedHat). -------- 5.08s
geerlingguy.drupal : Move Drupal project files to drupal_composer_install_dir (this may take a while). --- 2.85s
geerlingguy.repo-remi : Install remi repo. ------------------------------ 2.63s
geerlingguy.apache : Add apache vhosts configuration. ------------------- 1.36s
gather facts ------------------------------------------------------------ 1.30s
geerlingguy.mysql : Copy my.cnf global MySQL configuration. ------------- 1.05s
geerlingguy.mysql : Remove anonymous MySQL users. ----------------------- 1.04s
geerlingguy.repo-remi : Import remi GPG key. ---------------------------- 1.01s
Gathering Facts --------------------------------------------------------- 0.98s
geerlingguy.drush : Install Drush. -------------------------------------- 0.93s
geerlingguy.drupal : Install Drupal with drush. ------------------------- 0.91s
geerlingguy.mysql : Ensure MySQL is started and enabled on boot. -------- 0.87s
geerlingguy.composer : Download Composer installer. --------------------- 0.85s
geerlingguy.apache : Ensure Apache has selected state and enabled on boot. --- 0.83s
Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.

This seems to be another error, I'm wondering does it happen before ?

I found the issue, so I'm not aware of the environment ISLANDORA_DISTRO set to CentOS in my labtop, so it installed Centos instead
Anyways, the playbook built and works as expect.


Support PodMan Configuration for Local RHEL Deployment

I am going to move this to the backlog since our deployment on LinuxOne may not utilize Podman


Pre-compile each layer individually

After some feedback we found some runtimes require ability to individually pre-compile layers.  This attempts to address that ability.
This also fixes a bug that was introduced when adding pre-compilation where the media types were being overwritten.

@jsturtevant I have tested these changes out with a branch of the Spin containerd shim with pre-compilation support and it works great. I've tested Spin apps with multiple components and static assets, as well. To me, this is looking ready to take out of draft. Thank you for expanding the implementation!

@jsturtevant through more testing, I did find some odd behavior. After pulling an app, I can rerun the shim over and over and get expected behavior (reuses precompiled components); however, if i repull the image (ctr image pull) and try to run (ctr run) it fails to execute. It looks like after the image is repulled, runwasi is erroneously asking the spin shim to precompile again despite knowing it already was precompiled. Then the shim fails, potentially because it is trying to precompile cwasm at that point:
Feb 23 13:17:51 kagold-ThinkPad-X1-Carbon-6th containerd[496568]: time="2024-02-23T21:17:51.312229693Z" level=info msg="Shim successfully started, waiting for exit signal..."
Feb 23 13:17:51 kagold-ThinkPad-X1-Carbon-6th containerd[496568]: time="2024-02-23T21:17:51.317710774Z" level=info msg="found manifest with WASM OCI image format"
Feb 23 13:17:51 kagold-ThinkPad-X1-Carbon-6th containerd[496568]: time="2024-02-23T21:17:51.318314715Z" level=info msg="layer sha256:35dd7338635af15592327986c45e75efc2b258f1d92e4282ced7668d827e5741 has pre-compiled content: sha256:8d7374edd04e514d5c7b35a1b4a2a334716e1bcae596dfc0944e23b921bf6cc6 "
Feb 23 13:17:51 kagold-ThinkPad-X1-Carbon-6th containerd[496568]: time="2024-02-23T21:17:51.323980565Z" level=info msg="layer sha256:65bc286f8315746d1beecd2430e178f539fa487ebf6520099daae09a35dbce1d has pre-compiled content: sha256:dabc42e0d769727d23126b7908e0ea9dd4a93a7cd8867cb36615cbc990776ba8 "
Feb 23 13:17:51 kagold-ThinkPad-X1-Carbon-6th containerd[496568]: time="2024-02-23T21:17:51.335170453Z" level=info msg="precompiling layers for image: ghcr.io/kate-goldenring/spin-hello-and-kv-explorer:latest"
Feb 23 13:17:51 kagold-ThinkPad-X1-Carbon-6th containerd[496568]: time="2024-02-23T21:17:51.335195337Z" level=info msg="Precompiling layer with Spin Engine: "sha256:35dd7338635af15592327986c45e75efc2b258f1d92e4282ced7668d827e5741""
Feb 23 13:17:51 kagold-ThinkPad-X1-Carbon-6th containerd[496568]: time="2024-02-23T21:17:51.335225245Z" level=info msg="Precompiling layer "sha256:35dd7338635af15592327986c45e75efc2b258f1d92e4282ced7668d827e5741""
Feb 23 13:17:51 kagold-ThinkPad-X1-Carbon-6th containerd[496568]: time="2024-02-23T13:17:51.502392136-08:00" level=info msg="shim disconnected" id=two namespace=default
Feb 23 13:17:51 kagold-ThinkPad-X1-Carbon-6th containerd[496568]: time="2024-02-23T13:17:51.502458979-08:00" level=warning msg="cleaning up after shim disconnected" id=two namespace=default
Feb 23 13:17:51 kagold-ThinkPad-X1-Carbon-6th containerd[496568]: time="2024-02-23T13:17:51.502475509-08:00" level=info msg="cleaning up dead shim" namespace=default
Feb 23 13:17:51 kagold-ThinkPad-X1-Carbon-6th containerd[496568]: time="2024-02-23T13:17:51.507305203-08:00" level=error msg="copy shim log" error="read /proc/self/fd/13: file already closed" namespace=default

@kate-goldenring did anything in the image change that caused the image sha to change?

@jsturtevant the layers of the image are the same to ctr but for some reason runwasi is seems to be getting different layers. Checking if it is already precompile in the shim is an optional workaround that gets it working again:
    fn precompile(&self, layer: &WasmLayer) -> Option<Result<Vec<u8>>> {
        log::info!(
            "Precompiling layer with Spin Engine: {:?}",
            layer.config.digest()
        );

        match layer.config.media_type() {
            MediaType::Other(name) => {
                log::info!("Precompiling layer {:?}", layer.config.digest());
                if name == "application/vnd.wasm.content.layer.v1+wasm" {
                    if let Some(_) = self.wasmtime_engine.detect_precompiled(&layer.layer) {
                        log::info!("Already precompiled");
                        return None
                    }
                    let component =
                        spin_componentize::componentize_if_necessary(&layer.layer).unwrap();
                    Some(self.wasmtime_engine.precompile_component(&component))
                } else {
                    None
                }
            }
            _ => None,
        }
    }

ah, I see so we are passing the precompiled bits to the engine to recompile.  Makes sense it will crash, Let me see if we can fix it up in the shim library code but being defensive here might not be a bad option.

@jsturtevant to give more context, when evaluating whether to recompile(let mut needs_precompile = can_precompile && !image.labels.contains_key(&precompile_id);), after logging, it is clear that  !image.labels.contains_key(&precompile_id); is evaluating to false despite the fact that the labels are correctly displayed on the image during a ctr content ls

The change is pretty minimal on the shim implementor side.

6be79082379979b7e9d56bcd38c93e4bf64cde75

To @kate-goldenring's point:

@jsturtevant to give more context, when evaluating whether to recompile(let mut needs_precompile = can_precompile && !image.labels.contains_key(&precompile_id);), after logging, it is clear that image.labels.contains_key(&precompile_id); is evaluating to false despite the fact that the labels are correctly displayed on the image during a ctr content ls.

Any thouhgts why this would be happening? Nothing about the image changes in between pulls.

To @kate-goldenring's point:

@jsturtevant to give more context, when evaluating whether to recompile(let mut needs_precompile = can_precompile && !image.labels.contains_key(&precompile_id);), after logging, it is clear that image.labels.contains_key(&precompile_id); is evaluating to false despite the fact that the labels are correctly displayed on the image during a ctr content ls.

Any thouhgts why this would be happening? Nothing about the image changes in between pulls.

looking into this today

@jsturtevant to give more context, when evaluating whether to recompile(let mut needs_precompile = can_precompile && !image.labels.contains_key(&precompile_id);), after logging, it is clear that  image.labels.contains_key(&precompile_id); is evaluating to false despite the fact that the labels are correctly displayed on the image during a ctr content ls

I am seeing the label on the image disappear when using ctr i pull.  This doesn't seem to be the case when we use ctr import which is used by the tests.  Working on a fix and a test to cover the scenario.

I've stored the "pre-compiled" flag on the content instead of the image, since the image is mutable (I think @cpuguy83 pointed this out previously but I didn't fully understand the implications at the time).
Note that if runwasi detects that one or more of the pre-compiled components are removed from the cache it will re-request a compilation so its good to keep the check for compilation in the shim. I don't really know a valid scenario where this would happen besides a user going in and deleting it manually.  I will see if we can come up with a way to handle this when we improve the api towards process_layers.

The image reference (which we read from the container object) is mutable, is what I was referring to.
But yes I think its still best to have it on the content that is being compiled rather than the image itself which is often referring to an image index and not a specific image manifest.

I was looking into the why the test failed here, I was able to reproduce it locally although flaky and identified that it is due to the way the test is importing images.  In the failed test the image gets imported twice, I expected this to be a no-op but the annotations in the image are stored in a hashmap which results in a different image digest causing the flake (since the flag isn't found on the content but the individual layers are the same).
This comes back to my comment in https://github.com/containerd/runwasi/pull/492#issuecomment-1965541517 which is to say that we are not passing information around if we know whether it is compiled to the shim today but can possible address it with a better API.

Thank you @jsturtevant for all the iterations of this. This really speeds up the performance of the containerd shim being able to precompile all layers in one call and preserve those precompiled layers.

I pushed one last change based on your feedback.  Thanks for trying out all the changes, the feedback and being patient as I adjusted the API based on the usage

I may have bumped into an issue that will turn out a blocker for the current approach — while running a Wasm component built with SpiderMonkey, the resulting precompiled component size is  larger than the maximum message size in gRPC, which means saving it to the content store fails:
ERRO[2024-03-06T12:52:14.701165146+01:00] (*service).Write failed                       error="rpc error: code = ResourceExhausted desc = grpc: received message larger than max (43404572 vs. 16777216)" expected="sha256:617aa5799c43c511336f0c40708e8612dd804011eb9f713a9dbcb7340e8ff9c7" ref=precompile-runwasi.io/precompiled/spin/17767192358106208183 total=43404352
time="2024-03-06T11:52:14.703413103Z" level=warn msg="Error obtaining wasm layers for container sl.  Will attempt to use files inside container image. Error: response stream error: status: ResourceExhausted, message: "grpc: received message larger than max (43404572 vs. 16777216)", details: [], metadata: MetadataMap { headers: {} }"

Is there a way where runwasi could write the file in the store directly, then use the containerd API to add the correct annotations?

Is the message not getting chunked? If not, perhaps that is the path we should pursue.

it looks like the write failed when saving the new content. Will need to implement streaming for larger content when doing the write. Right now it writes and commits in one action: https://github.com/containerd/runwasi/blob/2171b3d841752b9dc0ac5314479059d23e9f6df8/crates/containerd-shim-wasm/src/sandbox/containerd/client.rs#L198-L201
I don't think this is a blocker on this PR and I would be inclined to merge this and make those changes separately as this changeset already has quite a bit going on.

Chunking sounds great, and I agree that we can do this in a follow-up.
Thanks for the quick update!

Okay, going to merge this and figure out the comments later.


Update contributors list

:warning: This is a Pull Request Template :warning:

Checkout all the things you've done by placing an x within the square braces below.

Proposed Changes

[x] I've read the CODE OF CONDUCT and abide to it.
[x] I've read the CONTRIBUTING.md
[x] I've forked the repository.
[x] I've created a branch and made my changes in it.
[x] I understand opening a PULL REQUEST doesn't mean it will be merged for sure.

Hi Josh, my bad cause I didn't see the fact that the names were alphabetically ordered.
I will squash the commits in the PR when merging so that history is clean.


RabbitListener annotation: Messages are treated several times with several application instances

We have an instance of rabbitmq in a docker container and we have several instances of our application in some containers which are pulling the same queue.
We are using the annotation @RabbitListener to pull the queue, the problem is that each instance of our application succeed to read the same message from the queue but the expected behaviour is that if a listener is reading one message the other listener cannot read it.

Might be you actually have multiple queues bound to the same same exchange, so each message sent to that exchange is duplicated into every queue?

No it is a direct exchange and there is only one queue configured for the exchange

Time to share a simple reproducible project with us? Generally speaking it’s impossible with the single queue.

I try to reproduce the problem locally in a sample project but the two consumers are always read exactly half of the messages (even if the spring boot jar is running inside a docker container).
There is probably another problem on my side.
Thank you


Initial condition for joint first order from stationary distribution

To make the 1st order methods more comparable, we could try the following change to https://github.com/HighDimensionalEconLab/HMCExamples.jl/blob/main/src/turing_models.jl#L42-L66
Which I think is what you have in mind @donskerclass ?
@model function rbc_joint_1(z, m, p_f, α_prior, β_prior, ρ_prior, cache, settings, x0_ignored)
    α ~ truncated(Normal(α_prior[1], α_prior[2]), α_prior[3], α_prior[4])
    β_draw ~ Gamma(β_prior[1], β_prior[2])
    ρ ~ Beta(ρ_prior[1], ρ_prior[2])
    β = 1 / (β_draw / 100 + 1)
    p_d = (; α, β, ρ)
    (settings.print_level > 1) && @show p_d
    T = size(z, 2)
    ϵ_draw ~ MvNormal(m.n_ϵ * T, 1.0)
    ϵ = reshape(ϵ_draw, m.n_ϵ, T)
    sol = generate_perturbation(m, p_d, p_f, Val(1); cache, settings)
    (settings.print_level > 1) && println("Perturbation generated")

    if !(sol.retcode == :Success)
        (settings.print_level > 0) && println("Perturbation failed $(sol.retcode)")
        @addlogprob! -Inf

    else
        (settings.print_level > 1) && println("Calculating likelihood")
        # Simulate and get the likelihood.
        x0 ~ MvNormal(sol.x_ergodic_var) # draw the initial condition
        problem = LinearStateSpaceProblem(sol, x0, (0, T), observables=z, noise=ϵ)
        @addlogprob! solve(problem, DirectIteration()).logpdf
    end
    return
end

Note that all I did was take the x0 from a fixed value and draw it after the ergodic distribution's covariance matrix was calculated.  Also note that the x0_ignored can be dropped from the arguments at the top if we go down that route).
That said, while this compiles I don't know how well it does in terms of the sampling itself.  Note that
But this might not work especially well and I think it woudl take some tinkering.  The things to remember to do with it:

we have to change the initial condition files to add in the x0 initial draws or else it might not work especially well.  Since the x0 ~ is the last variable, it means just putting two 0 at the end of the rbc_1_joint_burnin.csv etc. files to start with.  More for the fvgq of course.
Flip on the https://github.com/HighDimensionalEconLab/HMCExamples.jl/blob/main/src/rbc_1_joint_defaults.txt#L18  to calculate the ergodic distribution and probably turn off the perturb_covariance to 0.  At least for rbc
Maybe try doing a using DistributionsAD  in the script once just to ensure it isn't using the wrong gradients.
After you get that working, I think you can go back to removing the x0 argument in the joints for https://github.com/HighDimensionalEconLab/HMCExamples.jl/blob/main/src/turing_models.jl#L42  etc.  and then remove it from teh accompanying https://github.com/HighDimensionalEconLab/HMCExamples.jl/blob/main/src/estimate_rbc_1_joint.jl#L21  etc.  No need to pass in that vector of zeros

If you want to explore this, we can come up with a game plan for James to try it out.

See #80