Add an e2e test for running instructlab distributed training on RHOAI

Description

RHOAIENG-14225 - Automate an E2E test scenario for running instructlab distributed training on RHOAI
Source : https://github.com/redhat-et/ilab-on-ocp/tree/main/standalone
How Has This Been Tested?

Pre-requisites :  Document

Login to cluster and install RHOAI operator with Training Operator component enabled
Install NFS Provisioner, Redhat Authorino and Redhat Openshift Serverless operators from OperatorHub
Create  storage class to setup a storage with ReadWriteMany access :
Login to cluster and execute below commands

$ git clone https://github.com/OCP-on-NERC/nerc-ocp-config.git && cd nerc-ocp-config
$ kubectl apply -k  csi-driver-nfs/overlays/nerc-ocp-test
$ kubectl apply -k  nfs/overlays/nerc-ocp-test
```
5. Create test-namespace and Service-account linked with it
apiVersion: v1
kind: ServiceAccount
metadata:
  name: test-sa
  namespace: test-namespace

Create model server using single model serving platform, VLLM Serving  Runtime for Kserve serving runtime with accelerator( only supported on GPU- In my case used NVIDIA A100-SXM4-40GB for testing ) and the storage bucket data connection. (Check Model route and token authentication enabled with above created service account)

Note: If unable to use accelerator profile while deploying model server, check whether you have GPU available in cluster and once gpu is deployed and driver are installed on node then either create accelerator profile or run oc delete configmap migration-gpu-status -n redhat-ods-applications;oc rollout restart deployment.apps/rhods-dashboard -n redhat-ods-applications

Setup an S3 bucket consisting an tar archive of (SDG-Data, Model and Taxonomy)

>> export ODH_NAMESPACE=redhat-ods-applications \
AWS_ACCESS_KEY_ID=<secret-access-key> \
AWS_SECRET_ACCESS_KEY=<s3-secret-key> \
AWS_STORAGE_BUCKET=myrhelai \
AWS_DEFAULT_REGION=us-east-1 \
AWS_DEFAULT_ENDPOINT=https://s3.us-east-1.amazonaws.com \
SDG_OBJECT_STORE_DATA_KEY=rhelai.tar.gz \
SDG_OBJECT_STORE_VERIFY_TLS=True \
JUDGE_ENDPOINT=https://<model-name>-<namespace-name>.apps.<cluster-base-url>.com \
JUDGE_NAME=judge-eval-model \
TEST_NAMESPACE=rhelai-ilab-dw-training-test \
TEST_SERVICE_ACCOUNT=<created-service-account-name> \
JUDGE_SERVING_MODEL_API_KEY=<model-server-service-account-token-secret>

>> go test timeout 180m -v ./tests/odh -run TestInstructlabTrainingOnRhoai

Merge criteria:

[ ] The commits are squashed in a cohesive manner and have meaningful messages.
[ ] Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
[ ] The developer has manually tested the changes and verified that the changes work

Please follow this PR for further updates : https://github.com/opendatahub-io/distributed-workloads/pull/257


Add NGiNX Status endpoint to Healthcheck server

Thanks for the contribution! Please ensure your commits follow our style guide. This code will be tested once a Deis maintainer reviews it.

Codecov Report

Merging #335 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #335   +/-   ##
=======================================
  Coverage   56.47%   56.47%
=======================================
  Files           6        6
  Lines         425      425
=======================================
  Hits          240      240
  Misses        159      159
  Partials       26       26

Impacted Files
Coverage Δ

nginx/config.go
56.86% <ø> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b8e207e...63d807b. Read the comment docs.

I added the manual testing label. @pixeleet can you confirm that this is working? I haven't actually tried it; have only looked at the diff.

I can confirm that it works in a simple nginx implementation with the exact same code, I'd have to go and install this version of the router, and I have a long way to go before I'm ready to go 2.13 or what's the latest and greatest Deis release.

@pixeleet... if you do a stock install of deis/workflow in minikube or a development cluster, the command make deploy executed from the root of this repo will build the router from source, build it into a Docker image, and then swap that for the existing router in your cluster.

Thanks for the contribution! Please ensure your commits follow our style guide. This code will be tested once a Deis maintainer reviews it.

Alternatively you can also do a kubectl -n deis edit deploy deis-router and change the image name to quay.io/deisci/router:git-d16b111. That's the image jenkins just built and pushed to test this PR.

@bacongobbler Can't seem to pull the image. Do I need to setup secrets to access your Quay Repo?
18s        33s         2         deis-router-1387728611-mcb4z    Pod       spec.containers{deis-router}    Normal    Pulling      {kubelet ip-10-0-10-73.eu-central-1.compute.internal}    pulling image "quay.io/deisci/router:git-d16b111"
15s        31s         2         deis-router-1387728611-mcb4z    Pod       spec.containers{deis-router}    Warning   Failed       {kubelet ip-10-0-10-73.eu-central-1.compute.internal}    Failed to pull image "quay.io/deisci/router:git-d16b111": image pull failed for quay.io/deisci/router:git-d16b111, this may be because there are no credentials on this request.  details: (Tag git-d16b111 not found in repository quay.io/deisci/router)
15s        31s         2         deis-router-1387728611-mcb4z    Pod                                       Warning   FailedSync   {kubelet ip-10-0-10-73.eu-central-1.compute.internal}    Error syncing pod, skipping: failed to "StartContainer" for "deis-router" with ErrImagePull: "image pull failed for quay.io/deisci/router:git-d16b111, this may be because there are no credentials on this request.  details: (Tag git-d16b111 not found in repository quay.io/deisci/router)"

4s        30s       2         deis-router-1387728611-mcb4z   Pod       spec.containers{deis-router}   Normal    BackOff      {kubelet ip-10-0-10-73.eu-central-1.compute.internal}   Back-off pulling image "quay.io/deisci/router:git-d16b111"
4s        30s       2         deis-router-1387728611-mcb4z   Pod                                      Warning   FailedSync   {kubelet ip-10-0-10-73.eu-central-1.compute.internal}   Error syncing pod, skipping: failed to "StartContainer" for "deis-router" with ImagePullBackOff: "Back-off pulling image \"quay.io/deisci/router:git-d16b111\""

Just built it locally and YOLO'd it into production, I'm seeing stats coming in from Nginx into Sysdig.
Things seem to have not broken :P

@bacongobbler can you LGTM?

LGTM3


Add translations, add tx

What does this PR do and why?
Provide a description of what this PR does. Link to any relevant GitHub issues, Notion tasks or Discord discussions.
Screenshots or screen recordings
If applicable, provide screenshots or screen recordings to demonstrate the changes.
Acceptance checklist

[ ] I have evaluated the Approval Guidelines for this PR.
[ ] I have tested this PR in all supported browsers.
[ ] I have tested this PR in dark mode and light mode (if applicable).

Current dependencies on/for this PR:

main

PR #1654

PR #1655

PR #1656   👈

This comment was auto-generated by Graphite.

@jmilldotdev looks like some whacky merge conflicts in the translations files.
Give it a git checkout main -- src/locales and run extract again. Should be way easier in future, thanks for bearing with the current process <3


False positive issues

When i used your tool,it gives false positives to the website i scanned.I thought it is zero false positive.Kindly improve on the tool because of your reputation.

There is no such tool that will give you 0% false positives. That's why you use more than one tool for the job.
This is not an issue, nor a problem related to repo. The tool is awesome

See https://github.com/fullhunt/spring4shell-scan/blob/master/spring4shell-scan.py#L160 - the tool says that unless it gets a HTTP 200 or HTTP 404, the system is vulnerable. This is a very bad assumption. How about a WAF or any other layer returning a 403 (access denied) or 400 (bad request)?
Applied following diff for better debugging:
diff --git a/spring4shell-scan.py b/spring4shell-scan.py
index 7c911a7..f25dd89 100755
--- a/spring4shell-scan.py
+++ b/spring4shell-scan.py
@@ -158,6 +158,7 @@ def test_url_cve_2022_22965(url):
data={parameter: value},
proxies=proxies)
if r.status_code not in (200, 404) and verify_base_request(url, "POST") != r.status_code:
+ print(r.status_code, url, {parameter: value})
return True
except Exception as e:
cprint(f"EXCEPTION: {e}")
@@ -171,6 +172,7 @@ def test_url_cve_2022_22965(url):
params={parameter: value},
proxies=proxies)
if r.status_code not in (200, 404) and verify_base_request(url, "GET") != r.status_code:
+ print(r.status_code, url)
return True
except Exception as e:
cprint(f"EXCEPTION: {e}")

Now we can see what is claimed vulnerable:
Run this payload in curl:
> curl -i -XPOST https://www.example.com/ -d "{'class.module.classLoader[2akirzr]': '2akirzr'}"
HTTP/2 403
server: Varnish
retry-after: 0
accept-ranges: bytes
date: Fri, 22 Apr 2022 13:49:45 GMT
content-length: 0


SquareOkHttp3-4.4.1 signing

just date to trigger push

/azp run

Just to let you know that version 4.4.1 was successfully published in https://www.nuget.org/packages/Square.OkHttp3/4.4.1 🙂
Could you please also publish https://github.com/xamarin/XamarinComponents/pull/1046 and https://github.com/xamarin/XamarinComponents/pull/997?
Thanks 👍

@4brunu Thanks for the feedback.
We switched to new signing pipeline and approval took 48 hrs or so...
I will publish as soon as I can, gotta do some urgent stuff


How to delete map and timeline from menu?

Trying to figure out how to delete the Map and Timeline from the main nav menu

Figured it out!


Hide tooltip when anchor element is removed from the DOM

Closes #924.
The MutationObserver API is supposed to be super optimized, so this should scale fine, but I still worry if there isn't a more efficient way to do this.

I don't know yet if there is a better way to do this, but the code looks good to me, feel free to merge when you want :)


SegmentedDictionary Initial Implementation and Tests

When a MemoryGraph has many addresses, the internal Dictionary instance it uses to keep track of them might run out of space due to its internal array's limitations. Hence, this PR adds a new SegmentedDictionary class that will not run into this issue. The original implementation of the class is located in the dotnet/roslyn repo and can be found here.
The new class should have a similar performance to the Dictionary class it is replacing. However, we will continue to use the old class unless we detect that we might run into the memory issue.
This PR adds also the Generic Dictionary tests from the dotnet/runtime repo to guarantee that the new implementation is behaving as expected.

Overall, looks good.  A couple of requests:

Can you move the new files into a new SegmentedDictionary subdirectory of FastSerialization?  This will help to keep track of what files were copied.
Can you move the tests from PerfView.Tests to TraceEvent.Tests?  This is where the other FastSerialization tests live.  They should also live in a SegmentedDictionary subdirectory as well.


write: Added --compress option to gzip compress the resulting image

This adds a --compress option to the write command which will gzip compress the
image when writing.
Currently, it only does gzip. The ACI spec references gzip, bzip2, and xz.
Go has a built in gzip writer, but only a bzip2 reader. XZ is possible, however
it requires linking to libxz and then requiring it at runtime.
This changes also reorders some of the file closing within the write function.
It moves the closes from being done in line to being defered. The gzip.Writer
exposes a Close function, and want to ensure the Close options are done in
the proper order, and done across all three. Since the compression is optional,
it is easier with just defering the close.
The ACI may be removed on a failure, however it is fine to remove the file
before closing, it will clean up properly.
@jonboulle @dgonyeo

Other than my comments on defaulting to compress the ACI, LGTM.

Updated to simply always compress. Removed the parameter and passing of the compress parameter. Give it another look over.

LGTM


Update README.md

Fixes #1900.

I'm closing this off as I don't see the change of removing a single punctuation mark as a necessary modification. Especially when the punctuation would be otherwise consistent with all of the other list items.


[Documentation] The tmpdir_stagger_phase_ options are very hard to understand

I had been trying a lot of things with tmpdir_stagger_phase_ before finally understanding them thanks to @jkbecker in chat .  In my case, they allow me to do what I want, but I had no idea they did.
There's a few reasons why they are confusing to me:

The documentation and the tool itself uses : to separate major minor,  but the comment just says before tmpdir_stagger_phase_major tmpdir_stagger_phase_minor is less which is confusing.
They are based around "tmpdir", but i my case I have a single temp dir, and I am really only thinking about avoiding overloading my CPU, so it wasn't intuitive that it could do what I want (why is phase limit per tmpdir?)
The default of 1 seems very conservative, perhaps this is because it makes sense for people with many temp drives, but for people like me with a single temp drive, wouldn't something like 4 here be a better default?
the tmpdir_stagger_phase_major tmpdir_stagger_phase_minor are defined as the start of the next point beyond the limited window, this is a bit strange to wrap ones head around.  EG if i want to limit what's in phase 1, I set it to 2:1.  Perfectly fine once I understand it, but didn't lead to an intuitive reading.
The monitored/limited window always starts with 0:0 (because plotman can't predict when a job will enter a non 0:0 window start) but that isn't really clear
Given the way the major phases work (that they are intensive in different ways), is there any legitimate reason to tune the phase_minor here? Minor is always just the table.  I feel like the inclusion of it makes it seem more complex and harder to understand
I feel like the other options in the scheduling section are more intuitive and important, perhaps they come first, and this optional phase staggering is commented out and only used by people who wrap their head around it.
tmpdir_stagger_phase_limit is marked as optional, but tmpdir_stagger_phase_x are not, which makes me think they work independently?

I think that in the long run the option names could be changed, or the whole stagger limit by phase window could be reconsidered, (for instance simply a phase_1_max_jobs option would be so clear), I think updating the comments in the default yaml file will help a lot.  To kickstart the discussion I've quickly come up with the following draft.

I don't want to open a PR right away because it seems like a better discussion topic and I don't want to jump to conclusions.
Here's a first draft at a comment which could try and clearly and conciesly paint a picture of how these work:
        # The tmpdir_stagger_phase_ values are optional and give you the ability
        # to limit how many plots are running at the start of the phase cycle.
        #
        # For instance, if you want to only ever have 4 plots running at a time in
        # the first phase, set tmpdir_stagger_phase_major to 2,
        # tmpdir_stagger_phase_minor to 1, and tmpdir_stagger_phase_limit to 4.
        # This means that Plotman will monitor the window BEFORE 2:1 and ensure
        # that plot-triggering is limited to a maximum of 4. The plot-limited
        # window always starts at the very start (0:0).

How does that feel to everyone?

    # This means that Plotman will monitor the window BEFORE 2:1 and ensure
    # that plot-triggering is limited to a maximum of 4.

Wonder if this is redundant given the opening sentence?  Perhaps another example to add would be:
# Another example, if you only want 2 plots to run in the first part of the plotting process, set
# tmpdir_stagger_phase_major to 3, tmpdir_stagger_phase_minor to 4, and
# tmpdir_stagger_phase_limit to 2.

Also just to point out how confused I was, I currently had tmpdir_stagger_phase_major: 20, tmpdir_stagger_phase_minor: 10  configured.  😆
I looked into the code and saw it uses milestone variable name, I think if things are renamed down the road, using language like that would help indicate these options are defining a point in time and not a max plot count.

tmpdir_stagger_milestone_major = tmpdir_stagger_phase_major etc?

@graemes another example would be useful! although how did you end up with 3:4 being "the first part of the plotting process" ? or maybe those numbers aren't what you would include?

Meant to say "first half"

Yeah cool!

is there any legitimate reason to tune the phase_minor here?
Yes, there absolutely is. I have 3:4 configured there. I can safely start a new plot when i reached that phase without running out of temp space

Yes, there absolutely is. I have 3:4 configured there. I can safely start a new plot when i reached that phase without running out of temp space

what changes between 3:3 and 3:4? Less temp space?

Out of curiosity I wonder if what you are describing is better served by the use of the second temp drive option that chia offers
I’m not opposed to the minor phase config as everyone might have their own reasons but I think it would be useful if the documentation (config comments, readme, or wiki) could paints picture of how the minor phase config could be useful to be something other than 1. Perhaps you can share your scenario to help others.

From start to end of phase 3 there is a difference of around 60GB. When i start a new plot after table pair 4 i have enough free space to start phase 1 again without risk of running out of space. There is enough time for the first plot to finish phase 3 and finish the first 2-3 tables in phase 1. Also beeing able to configure it that granulary makes it possible for me to time my RAM and CPU resources. This would all not work if we could not configure minor phase for staggering.
And no, another temp drive is no viable solution just because it does not exist and would defy my intention to use the hardware most efficient (Which is the fundamental idea about chia). I frankly do not care if you get this or not. This setting is then simply of no relevance for you. Also do i belive, the current help text explains exactly what this does. It should not, nor does it need to, explain why you might want to use this.

@speedmann  Ok cool thank you for your explanation.

From start to end of phase 3 there is a difference of around 60GB.

My understanding is that you would want to set 4:1 then, not 3:4.  As the phase limit window is defined as a lt not a lte.  If you have 3:4 then plotman, as I understand it, is monitoring a window of 0:0-3:3 inclusive.   Is that your understanding too?

I frankly do not care if you get this or not. This setting is then simply of no relevance for you.

Incorrect: I need this setting too.  It is just that I nearly gave up on plotman because I couldn't understand what these settings did or how it would help.  I've seen the same sentiment multiple times from other people, I don't think I'm the only one.

The current config file is largely derived from how I had my system
configured some number of months ago, plus a few complications requested by
others.  There are probably some options for simplification of the overall
system, but also, there's a lot of easy headroom in having improved
documentation and a simpler default config that doesn't include a bunch of
complicated extra stuff.
On Thu, Apr 29, 2021 at 6:05 PM BasilHorowt @.***>
wrote:

@speedmann https://github.com/speedmann Ok cool thank you for your
explanation.
From start to end of phase 3 there is a difference of around 60GB.
My understanding is that you would want to set 4:1 then, not 3:4. As the
phase limit window is defined as a lt not a lte. If you have 3:4 then
plotman, as I understand it, is monitoring a window of 0:0-3:3 inclusive.
Is that your understanding too?
I frankly do not care if you get this or not. This setting is then simply
of no relevance for you.
Incorrect: I need this setting too. It is just that I nearly gave up on
plotman because I couldn't understand what these settings did or how it
would help. I've seen the same sentiment multiple times from other people,
I don't think I'm the only one.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/ericaltendorf/plotman/issues/151#issuecomment-829727160,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ARPZIFQRBLOJ4EGDVZMH2DTTLH67FANCNFSM43M322BQ
.

Hi,guys! I have 2T SSD as tempdir,12-core-cpu and 64G RAM, how can i config the tmpdir_stagger_phase_major tmpdir_stagger_phase_minor and limit to maximize the profermance and job safety?
Thank you very much.

Thanks for your interest.  This forum is really for reporting issues with
Plotman.  For general tuning of your plotting setup, please discuss on
#plotting-hardware and #plotman on Keybase.  Thanks for understanding.
On Thu, Apr 29, 2021 at 6:12 PM huangwuping @.***>
wrote:

Hi,guys! I have 2T SSD as tempdir,12-core-cpu and 64G RAM, how can i
config the tmpdir_stagger_phase_major tmpdir_stagger_phase_minor and limit
to maximize the profermance and job safety?
Thank you very much.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ericaltendorf/plotman/issues/151#issuecomment-829729452,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ARPZIFWRFJHBCX3QXTTQPXTTLH7XLANCNFSM43M322BQ
.

Can someone explain me what table pair means?
This is a term I have never stumbled upon in months of working with Chia.

I'm not sure what that term means.  Where did you see it?
On Sun, May 2, 2021, 13:04 flotti455 @.***> wrote:

Can someone explain me what table pair means?
This is a term I have never stumbled upon in months of working with Chia.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ericaltendorf/plotman/issues/151#issuecomment-830863995,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ARPZIFSRMETDH7563G7XB3DTLWV5DANCNFSM43M322BQ
.

Perhaps a better term might be _phase pair _ (or at least that's how I think of it).  From https://github.com/Chia-Network/chia-blockchain/wiki/Beginners-Guide - There are 4 phases that does operations in 7 tables
A table (phase) pair is Phase (major) and Step (minor).

FYI - the table with the % complete is quite useful for guesstimating plot scheduling but keep in mind the 1st phase only is multi-threaded while the remainder are single threaded atm.

# Temporary Drive Plotter Phase Settings

# Because Chia Phase 1 utilizes more memory and cpu (thread) resources
# AND is the only multi-threaded stage (after phase 1, the jobs are single threaded)
# These phase settings can help fine tune your chia plotting process
# beyond setting a stagger or limit on number of concurrent jobs

# There are 3 parameters that will help you achieve this. But before learning those,
# take a look at the table below, to learn all the phases and steps of the chia plotting process

# Chia Phase:Step Table

# Phase	Step	% Progress
# 1	1	1%
# 1	2	6%
# 1	3	12%
# 1	4	20%
# 1	5	28%
# 1	6	36%
# 1	7	42%

# 2	1	43%
# 2	2	48%
# 2	3	51%
# 2	4	55%
# 2	5	58%
# 2	6	61%

# 3	1	66%
# 3	2	73%
# 3	3	79%
# 3	4	85%
# 3	5	92%
# 3	6	98%

# 4	1	100%

# Example 1 - Detailed Explanation

# The tmpdir_stagger_phase_major and tmpdir_stagger_phase_minor correspond
# to the phase and step the plotting process is currently in
# For e.g. it could be 2:1 (2 is the phase or major, 1 is the step or minor)
tmpdir_stagger_phase_major = 2		# phase 2
tmpdir_stagger_phase_minor = 1		# step 1

# The tmpdir_stagger_phase_limit corresponds to the number of plot jobs that can be
# concurrently run before reaching phase 2, step 1 or 2:1
# If this is set to 2, then there will always be 2 plot jobs
# concurrently ran during phase 1. Or specifically during (1:1, 1:2, 1:3, 1:4, 1:5, 1:6, 1:7)
tmpdir_stagger_phase_limit = 2		# limit of 2 plot jobs

# Example 2
# You may decide you want to allow 3 conconcurrent plot jobs to run during phase 1
# Increase the tmpdir_stagger_phase_limit to 3
# tmpdir_stagger_phase_major = 2	# phase 2
# tmpdir_stagger_phase_minor = 1	# step 1
# tmpdir_stagger_phase_limit = 3	# limit of 3 plot jobs

# Example 3
# You may decide you want to relax the restriction and expand the phase:step to 3:1
# And Increase the limit on jobs to 4
# tmpdir_stagger_phase_major = 3	# phase 3
# tmpdir_stagger_phase_minor = 1	# step 1
# tmpdir_stagger_phase_limit = 4	# limit of 4 plot jobs

@thomann061 I think this would be a most excellent addition to the wiki :-)

You can see under https://www.hpool.com/help/bulletin from yesterday it says
Dear hpool users
Due to the rapid growth of computing power in Chia's whole network, the average block income is reduced. At present, the mining income of less than 2T capacity is too low to be calculated, so there is no income distribution. With the increase of the capacity of the whole network, the capacity requirement of the initial revenue will continue to rise. Ask users to increase the access capacity, otherwise they will not be able to get normal revenue.

If I change these values when plotman has started plotting, will it reload from the config file every time it's about to start a job?

(NOTE: I am currently using an old AMD FX-4170 quad core based computer to learn chia farming while I save for a decent setup)
So, I have 5 spinning drives installed in this PC, ( I don't have SSD's right now) I have plotman.yaml setup with all of the drives acting as temps and destinations to spread the IO around. I only want to have a max of 2 plots in phase 1:X so each gets 2 "cores". I can have up to 4 in the other phases because of the single threaded nature of those phases, so how would I setup the scheduling section for this scenario?
Based on what thomann061 said above.....
The tmpdir_stagger_phase_limit corresponds to the number of plot jobs that can be
concurrently run before reaching phase 2, step 1 or 2:1
If this is set to 2, then there will always be 2 plot jobs
concurrently ran during phase 1. Or specifically during (1:1, 1:2, 1:3, 1:4, 1:5, 1:6, 1:7)
This is not what happens on my setup. It always seems to go based on the global_stagger_m: which I had at 60 minutes. I land up with 4 plots running in phase 1. From what I understand, I should have one that plots until 2:1 and then a new plot starts. When that one reaches 2:1 another new one is triggered and so on. Is that not correct?

@thomann061 your edit is immensely more clear.  Thank you. I definitely did not understand this when I first read started. This helps a lot.

@BasilHorowt thanks for raising this, while folks like me just kept blindly fiddling without understanding!


pypi install requires python>=3.7

Snakemake version
5.27.*
Describe the bug
Cannot install version 5.27 on python<3.7.
Minimal example
$ python -V
Python 3.5.9
$ pip install 'snakemake==5.27.*'
Collecting snakemake==5.27.*
  Using cached snakemake-5.27.3.tar.gz (246 kB)
    ERROR: Command errored out with exit status 1:
     command: /home/michael/.pyenv/versions/3.5.9/envs/tmp_snakemake/bin/python3.5 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-g840_80j/snakemake/setup.py'"'"'; __file__='"'"'/tmp/pip-install-g840_80j/snakemake/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-gn_89tny
         cwd: /tmp/pip-install-g840_80j/snakemake/
    Complete output (2 lines):
    At least Python 3.7 is required.

    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Add a minimal example for reproducing the bug.
Additional context
I can see from https://github.com/snakemake/snakemake/blob/a813ea96fcd546c63654f06ce67e429104e5ca5c/setup.py#L14-L16 that this seems to be deliberate.
That is fine of course, but I have a very strong feeling that this should have been a major version bump (sorry, I know I have complained about this in the past also).
Additionally, pypi lists the package as being compatible with Python version 3.5 or above (and so does the front page of the github repo - but this badge is getting that info from pypi).

Hi Michael,
this was actually not intentional, it should still be python 3.5. During development, I thought at some point that we need to raise the minimum version to 3.7, but then I found a nice solution to avoid that. However, I forgot to fix the constraints again before releasing. I will release another patch release tomorrow, which will fix the version constraint.


Agent installer

Build the installer:

Run the installer:

Installer prompts for input:

Installer completes successfully:

Now you have a running windows service:

Example of sending an HL7 message from HAPI HL7:

coverage: 94.251% (-0.04%) from 94.286% when pulling 53b44b7fe2be4762af669edc3ce7fb843903ffb3 on cody-agent-installer into 17bf7b25ea0b6ca497cfd9ff26ef44b81e336aeb on main.


Question: Unicode support

I see that you mention that nix doesn’t have Unicode support, but as I understand it, it could just pass those bytes through to wherever. What kind of issues will be caused by Unicode characters within the parsed files?
If this is not the case, perhaps it is a valid use case and it can be raised here:
https://github.com/NixOS/nix/issues/770

The issue is that the regexes don't match on unicode characters - at least not the more specific ones like matchCharacter (https://github.com/talyz/fromElisp/blob/master/default.nix#L53) which should be able to match single unicode characters, but isn't. Unicode should be fine in places where the match is generic enough that all bytes of a character are matched, though.

I see, because my config do have unicode chars in a few places. As I understood it you couldn't use this if any unciode chars were present. I wonder if I could still use this, but I guess I just have to try it.

Okay, so I did some tests and I got it to work, so it seems it is working. The only issue is when you use unicode characters together with the char specifier ?. E.g. ?λ. When I either put the numeric representation of the char instead it worked, or wrapped the char in a string.
But perhaps it is possible to fix the parser to work with ?X unicode chars, or would that make things too tricky?

I don't think it's possible to make it work, no. The relevant part of the regex matches "any character but ], [, \ (, or )" and then looks for a delimiter that is not part of the token. Strings and comments work fine since they match everything until they hit a delimiter -  " for strings and \n for comments.

Okay, makes sense, I guess that is fine enough trade-off. Thank you for the explanation!


design: Split Deployments secure by default

Issue #881 covers an initial phase of work to secure communication between Contour and Envoy, allowing for a more secure split deployment. This issue covers the third phase of a three-phase plan to make secure split deployments the default for Contour.
The ultimate outcome of the plan is:

Split deployments the default for Contour
Secure communication between Contour and Envoy via secured gRPC.
Authentication for Envoy such that Contour can ensure that only its Envoy can see secret information (like Ingress or IngressRoute certificates), without Envoy needing to access the Kubernetes apiserver.

This design should cover how to:

Change the recommended deployments to use a split style with mutual TLS for gRPC
Flip Contour and Envoy to expect mutual TLS config and not work if you don't supply it, with a --run-insecure-style argument to disable this behaviour.
Update the single-deployment or single-daemonset style examples with the argument to disable the increased security.

TODO
New flag
I will add a new flag to contour serve, (proposal: --insecure), that makes clear that running without the TLS gRPC config is not ideal.
Logic change
I will change the contour serve logic to complain if you:

don't supply the tls options and
don't supply --insecure.

This will be in the tlsconfig() function in cmd/contour/serve.go
This will be a breaking change, and will need to be managed in the upgrade process. However, doing this change this way means that people won't be able to end up insecure by accident, they will either need to make their deployment secure, or add --insecure.
New deployment types
I will update the deployments with three new examples, and deprecate the current ones.
The three new examples will be:

split: A split configuration, with Contour running as a deployment with 3 replicas and Envoy running as a daemonset with a Service of type: LoadBalancer. This will use the new process to generate TLS certs for secured gRPC.
split-hostnet: A split configuration with Contour running as a Deployment and Envoy running as a DaemonSet with host networking. This will allow the use of other load balancers.
single: A single-pod daemonset configuration, similar to the current default, using a LoadBalancer Service.

The deprecated examples will be removed in our beta1 version.
Docs updates
README.md needs an update to recommend using the split deployment as the default.
docs/deploy-options.md needs a substantial update, to use the split deployment as the default.
examples/ needs a README.md to describe what the examples are for, with a summary of positives and negatives.

I've removed the 'design' tag from this as I'll use this to cover the implementation.

Still to be done here: Tidy up the example deployments, and then update all the documentation.


Is there a way to deploy twilio-video using my own custom domain?

When I deploy, the domain name is set to video-app-XXXX-dev.twil.io
Is there a way to still deploy to twilio serverless, and use my own domain?
For example, host the app with live.deephire.com?
I saw this in the twilio docs, but it didn't seem relevant to what I am looking for.

Hey everyone,
This issue is not currently on our roadmap and we have decided to pursue other priorities. I will reopen if anything changes.


Outgoing integration bots

Are outgoing bots possible? If so, could someone point me to any example code?
Thank you.

@phalexo yes, see for example https://zulip.com/api/outgoing-webhooks.


[bug] Cannot use 'import.meta' outside a module

Describe the bug
I'm using latest expo with tamgui and receive an error running expo for web.
Uncaught SyntaxError: Cannot use 'import.meta' outside a module
it's coming from this line:
var isRSC = process.env.ENABLE_RSC ? import.meta.env ? import.meta.env.SSR : false : false;
To Reproduce
create fresh expo app and follow tamagui installation instruction from official docs, then run expo with TAMAGUI_TARGET=web expo start --web
System Info (please complete the following information):
npx envinfo --system --npmPackages --binaries --browsers

  System:
    OS: macOS 13.2
    CPU: (8) arm64 Apple M1 Pro
    Memory: 71.08 MB / 16.00 GB
    Shell: 5.8.1 - /bin/zsh
  Binaries:
    Node: 18.13.0 - ~/.asdf/installs/nodejs/18.13.0/bin/node
    Yarn: 1.22.19 - ~/.asdf/installs/nodejs/18.13.0/.npm/bin/yarn
    npm: 9.4.0 - ~/.asdf/plugins/nodejs/shims/npm
    Watchman: 2<IP_ADDRESS> - /opt/homebrew/bin/watchman
  Browsers:
    Chrome: 109.0.5414.119
    Safari: 16.3
  npmPackages:
    @babel/core: ^7.20.12 => 7.20.12
    @babel/plugin-syntax-import-meta: ^7.10.4 => 7.10.4
    @lingui/cli: ^3.17.0 => 3.17.0
    @lingui/macro: ^3.17.0 => 3.17.0
    @lingui/react: ^3.17.0 => 3.17.0
    @react-native-community/eslint-config: ^3.2.0 => 3.2.0
    @tamagui/animations-react-native: ^1.2.5 => 1.2.5
    @tamagui/babel-plugin: ^1.2.5 => 1.2.5
    @tamagui/config-base: ^1.2.5 => 1.2.5
    @tamagui/react-native-media-driver: ^1.2.5 => 1.2.5
    @tamagui/shorthands: ^1.2.5 => 1.2.5
    @tamagui/theme-base: ^1.2.5 => 1.2.5
    @types/react: ~18.0.24 => 18.0.27
    @types/react-native: ~0.70.6 => 0.70.10
    @typescript-eslint/eslint-plugin: ^5.50.0 => 5.50.0
    @typescript-eslint/parser: ^5.50.0 => 5.50.0
    babel-plugin-macros: ^3.1.0 => 3.1.0
    babel-plugin-transform-inline-environment-variables: ^0.4.4 => 0.4.4
    eslint: ^8.33.0 => 8.33.0
    expo: ~47.0.12 => 47.0.13
    expo-constants: ~14.0.2 => 14.0.2
    expo-dev-client: ~2.0.1 => 2.0.1
    expo-linking: ~3.3.0 => 3.3.0
    expo-router: ^1.0.0-rc5 => 1.0.0-rc5
    expo-splash-screen: ~0.17.5 => 0.17.5
    expo-status-bar: ~1.4.2 => 1.4.2
    prettier: ^2.8.3 => 2.8.3
    react: 18.1.0 => 18.1.0
    react-dom: 18.1.0 => 18.1.0
    react-native: 0.70.5 => 0.70.5
    react-native-mmkv: ^2.6.1 => 2.6.1
    react-native-safe-area-context: 4.4.1 => 4.4.1
    react-native-screens: ~3.18.0 => 3.18.2
    react-native-web: ~0.18.9 => 0.18.12
    tamagui: ^1.2.5 => 1.2.5
    typescript: ^4.9.5 => 4.9.5

Haven't seen this yet so seems like a regression on Expo side, can you open there first saying it regressed and when? The syntax is valid and behind a conditional that should fail.

I'm getting the same issue. Were you able to resolve this @dimonnwc3?

@JohnKing93 no, I'm not using this library anymore

I'm experiencing this exact issue trying out tamagui. Bare expo router template, followed the installation guide.
process.env.TAMAGUI_TARGET = "native"; (or web) doesn't seem to make a difference (at least while developing).

Getting the same issues. Expo 48 + freshly installed tamagui.

I’ll remove this for now, Jesus Christ es modules are a huge pain to support alongside React Native.
This is just a warning though right? I saw it yesterday.

What do you mean by "remove". In my case, I saw the error in the browser console (chrome). It wasn't a warning.

The latest version no longer references import.meta at all. There should be no error or warning.

@yepes what stack? Next or vite?

@yepes what stack? Next or vite?

Bare Expo (plus react native web)

I will try to recreate it tomorrow. Will come back with the result


UI Plugin: AWS os image - allow user to select

Currently the os image is hard-coded in awsDeployment.tsx (line 70). Instead, we need to allow the user to select it. The list should be relatively small since we take it from our own configuration

closing, duplicate


Virtee - Firmware Wrapping

Wrap SEV Status codes to error returns from IOCTLS in the Virtee Library

Closed by #228 (re-open if I'm wrong)


Support complex filter before merge procedure

Describe This Problem
A filter procedure according to the query predicates will be applied to the record batch stream from sst before feeding the batches to the merge iterator. However, the filter only supports a very simple form -- anded binary expression, so it doesn't work if the query predicate is complex, e.g. where (hostname = '<IP_ADDRESS>' or hostname = '<IP_ADDRESS>') and timestamp between 'xxxx' and 'xxxx'.
Proposal
The crucial point here is how to make the filter procedure can support complex predicate expressions, and basically there are two approaches to this target:

Utilize datafusion;
Implement the filter logic manually;

And I vote for the first approach, but we have to figure out how to utilize datafusion to implement the filter logic.
Additional Context
No response

Hi, @ShiKaiWi.
Seems we pushdown only some filters to table scan and reserve filter operator now, which is like prewhere optimization.
Recently datafusion also added a similar feature(row-filter), like prewhere.
I am not sure it is reasonable to pushdown all filters, but I think rewrite the filter logic by utilizing datafusion is ok.
To utilize datafusion, we can do:

Directly use create_physical_expr to transfer from Expr to PhysicalExpr.
Follow FilterExecStream in datafusion to implement filter logic.

Refences:
clickhouse: https://clickhouse.com/docs/en/sql-reference/statements/select/prewhere/
datafusion(row-filter): https://github.com/apache/arrow-datafusion/issues/3360

It will be appreciated if you volunteer to hrlp.

@ygf11 I have updated the code location about the filtering procedure, and I hopes it will help:
https://github.com/CeresDB/ceresdb/blob/43a84ba3c2ddcee69906e70322060b6dc4e91ddc/analytic_engine/src/row_iter/record_batch_stream.rs#L137

I have updated the code location about the filtering procedure, and I hopes it will help.

Thanks for reminding， it helps a lot.


[BUG] Unsatisfiable error in stateful testing if all operations have inbound links

Describe the bug
Schemathesis stateful testing raises hypothesis.errors.Unsatisfiable when all operations have inbound links. In this case, all rules of the underlying state machine require some input from existing bundles, that are empty and can't be filled because they require some input to be generated. Recursion! :)
To Reproduce

Run the following test with pytest:

SCHEMA = ...  # see below
schema = schemathesis.from_dict(SCHEMA, base_url="http://<IP_ADDRESS>:8081")

TestCase = schema.as_state_machine().TestCase

See

E   hypothesis.errors.Unsatisfiable: Unable to satisfy assumptions of hypothesis run_state_machine.

API schema:
{
    "openapi": "3.0.2",
    "info": {"title": "Test", "description": "Test", "version": "0.1.0"},
    "paths": {
        "/count": {
            "post": {
                "operationId": "counter",
                "requestBody": {
                    "content": {
                        "application/json": {
                            "schema": {
                                "type": "integer",
                                "minimum": 0,
                                "maximum": 5
                            }
                        }
                    },
                    "required": true
                },
                "responses": {
                    "200": {
                        "description": "OK",
                        "links": {
                            "CountMore": {
                                "operationId": "counter",
                                "requestBody": "$response.body#/value"
                            }
                        }
                    }
                }
            }
        }
    },
}

Expected behavior
Schemathesis won't fail with such error and will generate data from the operation definition

That's why I created #822 and completely forgot about the reason behind it

Also, I need to redesign the test case because it is too specific for the current behavior (will expand later)


Extract a flutter application using Odoo_rpc package

After i extract an application using this package , i can't sign in in the application that doesn't using debug mood , it's only work in debug mood .
When i use it in a debug mood , i get this error :
ClientException with SocketException : Failed host lookup :' my Url' ( OS Error: No address associated with hostname , error =7),
Please give me some clues to the solution

No address associated with hostname

Hi!
Which url address are you using to connect to Odoo instance? Looks like it has wrong hostname.

Hi,
it wasn't , because it work perfectly in debug mood , but not in the extract app
the url : https://cnsa-main-10458567.dev.odoo.com
do you have a clue


Localization

Can we allow a user to choose a preferred language?

It looks like django provides tools and a community of translators for localization: https://docs.djangoproject.com/en/dev/internals/contributing/localizing/

It looks like django provides tools and a community of translators for localization: https://docs.djangoproject.com/en/dev/internals/contributing/localizing/


Update to cef_binary_126.2.18+g3647d39+chromium-126.0.6478.183

I wasn't sure where/how to update the osx binaries.

I wasn't sure where/how to update the osx binaries.

Theres these 2 other repos:
https://github.com/OutSystems/cef.redist.win
https://github.com/OutSystems/cef.redist.osx

"Open Dev Tools" is broken for Avalonia/WPF demos, any ideas?


Google Stackdriver output

Is your feature request related to a problem? Please describe.
We use stackdriver as an output for fluentd for a manual deploy currently but would like to move to logging-operator.
Describe the solution you'd like
I'd like to use the fluent-plugin-google-cloud in the same manner as the other backend based plugins like ElasticSearch.
Describe alternatives you've considered
I have considered a GCS logging bucket and a function to ship the logs from there into the StackDriver API but it is a less than ideal option.
Additional context
This was previously raised in #40 but considered not required unless someone was willing to raise it as a separate request later.

Ay,
Is StackDriver output available ? We want to move from our old logging workflow to Banzai Cloud Logging Operator but StackDriver is mandatory for us.

This becomes a requirement for us too. Can we think of rescheduling this task?

@aslafy-z okay, I'll check it.

Any updates? Would love if this became available

@tarokkk would you be able to provide an update at all please? Just so we can assess to wait or use a different tool. Thanks

The problem, why this has not yet been implemented is the complexity it brings into the already complex and big fluentd image.
One idea I have at the moment is to create a base image that includes the base components only without any third party outputs, so that it is small and faster to build. Users of the the operator can than depend on this base image and add their combination of output plugins on top of that, which would work for their specific use case.
The only thing that bothers me in this case is that we cannot easily decide from the logging operator if an output is supported by the image or not, so if someone sees the google stackdriver output configuration could naively assume, that it is supported in the base image and they would be surprised, that they have to build their own.
Any thoughts?

Hi @pepov,
THanks for taking a look. II'm pretty sure its not possible but a bare base image with no outputs would be quickets, then at runtime install additiona outputs that the end user needs.?

@ekhaydarov we don't have a method to install plugins at runtime, if someone needs more outputs they have to build a custom image or they will be able to run multiple aggregators (through multiple logging resources) and use different images on each.

There is another option which some of the core maintainers would be open to explore, which is implementing an output for the syslogng based aggregator.
There are two possible solutions in that case as well: migrate to the syslog-ng based aggregator or use a separate syslog-ng logging without a fluentbit agent and relay logs designated in stackdriver into that aggregator.
Let me know if any of the above would work for you.

With the introduction of the new multi-stage image build it is now possible to have an image that supports Google Stackdriver only without having to pull the ruby dependency hell into our full image, which is already quite large. Given this, we can finally implement the required API in the Logging Operator as well. In case someone would need an image that contains not just the Stackdriver output they will have the option to base off of our stackdriver image quite easily.

anyone still interested in this?


Add Xamarin support

[ ] Find right abstraction library
[ ] Create assembly for hosting it
[ ] Create sample app (not in the library project)

Hi,
I recently started to write a lego boost xamarin library  when I found this awesome project. So I forked it and implemented the Xamarin Bluetooth implementation (based on Plugin.BLE) to be able to use the core lib within xamarin.
I would like to pull reques, but I have some questions. You recently updated the project to .NET 5 but this will be incompatible with Xamarin (Forms). I reworked it already in my feature branch (use latest c# version, targetFramework netstandard2.1).
But maybe a better solution would be to have two solution files with one Windows and one Xamarin projects?
Shall I open the pull request to discuss this within the pr itself?

Damn. What a unfortunate moment. A months earlier and I would have delayed the whole new tech wave till after .NET 6.
I think the best option is:

we create a maintenance release on the 3.x version which will support Xamarin.Forms 5.
we stabilize the whole feature there
with .NET 6 in November we merge the stuff in the v4+ at that time

Is that acceptable to you? It would lock you out of the mainline for 8+ months. Which is endless and super short at the same time. The other alternatives are duplicate implementions (like suggested but I do not like it) or some magic on the project files and #ifdefs.
I want your contribution!! We just need to find the right way of doing it.
Create the pull so I can have a look. that does not harm!


It's possible to ignore disk quota on btrfs storage backend

If you are setting disk quota for container on btrfs storage backend it's works only for current subvolume.
But if user creates another subvolume inside container, new subvolume does not inherit parent's limits.

Is there another way to control disk quota for btrfs storage backend?

So hmm, that's a bit upsetting indeed. I just went through a whole bunch of btrfs quota and qgroup documentation and there doesn't appear to be a way to tell btrfs to inherit from the parent subvolume's qgroup when the user creates a new subvolume.
That seems like a rather major flaw in the way quotas are handled, but maybe I'm missing something...

@brauner is that something you could take a quick look at when you have some time? We're certainly not going to go add bits to btrfs in the kernel, but I'd like to make sure I didn't overlook a way of doing this as it seems such an obvious use case...

Only snapshots automatically inherit quotas from their parent subvolumes. I'm looking into doing this for subvolumes.

@brauner ok, that confirms what I've seen.. That's pretty annoying then since as long as the user is allowed to create a subvolume from inside the container, they can therefore bypass the quota...

Well, the only way is to create a parent qgroup and assign both sub-qgroups as children of the parent qgroup in order to share a common limit. But this is terribly fiddly and according to my local testing this is not working smoothly at all.

@brauner right, but there's nothing which does that automatically at the kernel level and we can't have LXD constantly scanning for new subvolumes to try and redo their qgroup setup.
Not to mention that there would be nothing preventing the user from simply removing the qgroup from the subvolume they just created...

@stgraber, yup, yup, and yup.

@brauner can you maybe ping the btrfs upstream mailing-list about how to deal with such a setup?
I'd think it'd be the exact same problem if a system admin was to setup per-user subvolumes in /home to apply user quotas. Anyone of those users would be able to bypass the quotas by running "subvolume create" somewhere in their home directory and dump a bunch of big files in there.
If they say there's no way to do it at all, then we should update doc/storage-backends.md to reflect that and close this issue.

I'll ping them.

Sent a mail and CCed you (@stgraber).

@brauner thanks for that. So sounds like the conclusion is that btrfs quotas while hierarchical aren't automatically tied to the parent, letting whoever is allowed to create a new subvolume (everyone) to bypass their quota.
Can you send a pull request which updates storage.md to mention that?

Will do.

@stgraber @brauner btw, https://linuxcontainers.org/lxd/try-it uses btrfs and allows to anyone bypass quota using this method.

That depends on whether @stgraber is actually using btrfs quota on the btrfs {filesystem,subvolume} that the parent LXD daemon of the daemon your interacting with runs on. :)

@brauner and @stgraber Was there any response from the btrfs team?

@michacassola, not a very helpful one: https://marc.info/?l=linux-btrfs&m=149288068515307&w=2
The solution that they suggest only applies to snapshots taken by LXD itself not to snapshots taken by users inside the container. Short answer: it's broken and it looks like it'll stay this way.

I could look at this and see how difficult it would be to fix btrfs quotas.

@michacassola that wouldn't prevent a user from directly hitting the system with btrfs ioctls through some alternative binary. I suppose you could use a fancy seccomp filter to try and prevent the specific ioctls but that will also cause any tool that's used to being able to perform those on a btrfs filesystem to then fail miserably (Docker would likely be one of those).

When we have the seccomp userspace trap we can rewrite the corresponding argument in the btrfs ioctl() to always pass the option to create quota-based snapshots. :)

@stgraber and @brauner Thanks for explaining but since @stgraber s comment things started to go over my head. I hope you figure it out though!


Bug: 73e7f5f93b65 break complete_line/complete_bline

RTFM Checklist

[X] I have searched exisiting issues / discussions
[X] I have read the Wiki including the Advanced section
[X] I have read man fzf / I am well versed in shell fzf

Operating system
linux
Shell
bash
Neovim version (nvim --version)
NVIM v0.11.0-dev-1405+g99aa6bfd98
Fzf version (fzf --version)
0.56.3 (add1aec6)
Output of :lua print(os.getenv('FZF_DEFAULT_OPTS'))
nil
Is the problem reproducible with mini.sh?

[X] My issue is reproducible with mini.sh
[ ] My issue IS NOT reproducible with mini.sh
[ ] I have not tested with mini.sh (not relevant, requires LSP, Windows, etc)

Fzf-lua configuration
require('fzf-lua').setup({
})

Describe the bug / steps to reproduce
73e7f5f93b65 break complete_line/complete_bline

What does break mean? Not complete or does the interface not open?

https://github.com/ibhagwan/fzf-lua/commit/a4f1644efaa0e3be884096711fcd5f7bf90c494c
FYI, it's the refactor of lines/blines that broke this https://github.com/ibhagwan/fzf-lua/commit/7836b37b1d7f4786b8c8804bac29beb3926a56a7 as the formatting of the line changed, should work as expected now.


drop consumers if their out buffer fills up

right now if these buffers fill up we just start dropping events, and then when the buffers drain we just resume sending events. this seems quite wrong.

hrm.... this isnt quite right, it doesnt close out the connections and will result in a hanging subscriber

This should work now

Actually we should probably emit a ConsumerTooSlow Error Event as well before closing


minor image renaming

Coverage remained the same at 74.797% when pulling 3918f8df2d48e6c4d8f8e095b143f656a581794e on kthSim:master into 91f3cf9d4b7de52d56949c7abd6ed58e098a3497 on CS2103-AY1819S2-W17-2:master.


Add VM instancetype and preference label to vmi_phase_count metric

What this PR does / why we need it:
Recently Instancetypes and Preferences were added to simplify and reuse VM configurations. In this PR we will collect the Instancetypes and Preferences used in the vmi_phase_count.
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #
Special notes for your reviewer:
Release note:

Add VM instancetype and preference label to vmi_phase_count metric

jira-ticket: https://issues.redhat.com/browse/CNV-22079

/cc @lyarwood

/cc @sradco

/area instancetype

/retest

/approve

/cherry-pick release-0.59


Unused span shouldn't cover 'function' keyword

function area(shape) {
    throw shape;
}

'function' gets highlighted gray in my editor

We either should highlight the whole line, or only the identifier.


Not be able to compile scss file 'typeerror arguments to path.join must be strings'

Hi,
Please Help Me, I'm Trying to compile my scss by using gulp-ruby-sass
Its give me following errors

this is my gulpfile.js code

gulp.task('styles', function() {
gulp.src('scss/new-style.scss')
.pipe(sass())
.pipe(gulp.dest('css/compile'));
});

please help me??????????????????????????????????

Please read the readme for examples of how to set up the task or search
closed issues on github for your error. You'll find what you need :)
Rob
On Tuesday, August 25, 2015, shahjehanali1<EMAIL_ADDRESS>wrote:

please help me??????????????????????????????????
—
Reply to this email directly or view it on GitHub
https://github.com/sindresorhus/gulp-ruby-sass/issues/248#issuecomment-134553398
.

--
Rob Wierzbowski
@robwierzbowski http://twitter.com/#!/robwierzbowski
http://github.com/robwierzbowski
http://robwierzbowski.com

Specifically note that gulp-ruby-sass is a source adapter and is used in place of gulp.src.
https://github.com/sindresorhus/gulp-ruby-sass#usage
https://github.com/sindresorhus/gulp-ruby-sass/issues?utf8=✓&q=is%3Aissue+arguments+to+path.join+must+be+strings


Email OTP configurations not working properly

Describe the issue:
Refer - https://is.docs.wso2.com/en/latest/learn/configuring-email-otp/
Even when the user  disable the EMail OTP for Identity Server using following toml configuration, the OTP's are still sent sucesfully.
[authentication.authenticator.email_otp]
name ="EmailOTP"
enable=false

Moreover,

When a user go to the profile and try to disable the Email OTP through the user profile , it shows a text field.
When we provide "true" for that field, t converts to checkbox. And if we provide something else other than "true", it will save it as a string value.
How to reproduce:
Debug the issue in the toml configurations related to disabling the Email OTP
Fix the issue in UI , when disabling the email OTP through the user profile.
Expected behavior:
The email OTP should not send when disabled through toml configurations.
Better to have a checkbox at the first place , when disabling the Email OTP through the user profile
Environment information (Please complete the following information; remove  any unnecessary fields) :

Product Version: [e.g., IS 5.11.0]
OS: [Linux, ]
Database: [ H2]
Userstore: [ LDAP]

Optional Fields
Related issues:

Suggested labels:

This issue is being closed due to extended inactivity. Please feel free to reopen it if further attention is needed. Thank you for helping us keep the issue list relevant and focused!


Display as List

As a user I want to be able to display files as horizontal list items

@eunit99 is working on this


[BUG]: No internet after migrating PiVPN to new setup with PiHole installed (Solution included)

In raising this issue I confirm that

[X] I have read the documentation
[X] I have read and understood the PiVPN General Guidelines
[X] I have read and understood the PiVPN Troubleshooting Guidelines
[X] The issue I am reporting isn't a duplicate, see closed issues and open issues.
[X] I have searched for similar issues and solutions
[X] I can replicate the issue even after a clean OS installation

Describe the issue
No internet access occurs on client devices after following the migration guide for moving PiVPN, see documentation "PiVPN Docs: Migrating PiVPN & Wireguard", with a PiHole installation.
Requests are sent from the clients device to PiVPN where the DNS is resolved, as can be monitored in PiHole, however the answers are not returned to the client. This leads to no internet access however local network access is available.
Expected behavior
DNS requests should be returned when the client requests them. Essentially they are blocked by a Network Address translation (NAS) which does not pass the returned responses to the client.
This occurs due to how PiVPN installation on the new system almost always uses a different subnet mask compared to the subnet mask being imported from the migration. For example...
PiVPN-Alpha = New installation = Subnet <IP_ADDRESS>/24
PiVPN-Beta = Old installation being imported to maintain clients = Subnet <IP_ADDRESS>/24

PiVPN new installation, PiVPN-Alpha, is initiated with a subnet of <IP_ADDRESS>/24. This is written into the IP tables which can be viewed using sudo iptables -t nat -S as -A POSTROUTING -s <IP_ADDRESS>/24 -o eth0 -m comment --comment wireguard-nat-rule -j MASQUERADE
PiVPN-Alpha undergoes the migration where the server, clients and extras are imported from PiVPN-Beta. This is carried out following the "PiVPN Docs: Migrating PiVPN & Wireguard". However, PiVPN-Betas' subnet is not <IP_ADDRESS>/24, it is <IP_ADDRESS>/24. This leads to PiVPN-Beta subnet clients and previous working PiVPN-Alpha being unanswered on the new installation however clients and requests appearing and answered in PiHole.

Please describe the steps to replicate the issue
A previous PiVPN installation is required to replicate the issue.
PiVPN-Alpha = New installation = Subnet <IP_ADDRESS>/24
PiVPN-Beta = Old installation being imported to maintain clients = Subnet <IP_ADDRESS>/24

Backup the previous PiVPN installation, PiVPN-Beta, using command pivpn -bk
Transfer the PiVPN backup, "pivpnbackup.tgz", to the new installation.
Install PiHole on the new installation.
Install PiVPN using Wireguard on the new installation. PiVPN should detect a PiHole installation and apply relevant settings.
Follow the "PiVPN Docs: Migrating PiVPN & Wireguard".
Attempt to connect to VPN where local network is accessible but no internet is available. PiHole should present VPN client and DNS requests with response.

Have you taken any steps towards solving your issue?
PiVPN-Alpha = New installation = Subnet <IP_ADDRESS>/24
PiVPN-Beta = Old installation being imported to maintain clients = Subnet <IP_ADDRESS>/24
Firstly, check that this is the issue for your installation using sudo iptables -t nat -S and confirm that the subnet for your clients is not present in IP tables. If the migrated client subnet is present in the IP tables, this will not solve your issue. For example...
-A POSTROUTING -s <IP_ADDRESS>/24 -o eth0 -m comment --comment wireguard-nat-rule -j MASQUERADE
To solve the issue, add a new line to IP table for NAT with the previous subnet which was migrated. The previous subnet can be viewed on a client config file. For example...
sudo iptables -t nat -I POSTROUTING 1 -s <IP_ADDRESS>/24 -o eth0 -j MASQUERADE
Note it is important that the subnet is <IP_ADDRESS> whilst the client will have a number greater than 0. This is for the Wireguard server.
Screenshots
No response
Where did you run pivpn?
Raspberry Pi 5 using the latest Raspberry Pi OS Bookworm
Please provide your output from uname -a
Linux PiNAS 6.1.0-rpi4-rpi-2712 #1 SMP PREEMPT Debian 1:6.1.54-1+rpt2 (2023-10-05) aarch64 GNU/Linux
Details about Operative System
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
Installation
No response
Profile / Client creation
No response
Debug output
::: Generating Debug Output
::::            PiVPN debug              ::::
::::            Latest commit            ::::
Branch: master
Commit: f9ae9662ecc49896d09cdb38e177d2efa90e7119
Author: camdenAR
Date: Thu Oct 19 18:56:19 2023 -0500
Summary: fix(scripts): Match full client name when disabling
::::        Installation settings        ::::
PLAT=Debian
OSCN=bookworm
USING_UFW=0
pivpnforceipv6route=1
IPv4dev=eth0
IPv4addr=[REDACTED]
IPv4gw=[REDACTED]
install_user=[REDACTED]
install_home=[REDACTED]
VPN=wireguard
pivpnPORT=[REDACTED]
pivpnDNS1=[REDACTED - PiHole DNS]
pivpnDNS2=
pivpnHOST=REDACTED
INPUT_CHAIN_EDITED=1
FORWARD_CHAIN_EDITED=0
INPUT_CHAIN_EDITEDv6=
FORWARD_CHAIN_EDITEDv6=
pivpnPROTO=udp
pivpnMTU=1420
pivpnDEV=wg0
pivpnNET=[REDACTED]
subnetClass=24
pivpnenableipv6=0
ALLOWED_IPS="<IP_ADDRESS>/0, ::0/0"
UNATTUPG=1
INSTALLED_PACKAGES=(grepcidr bsdmainutils dhcpcd5 iptables-persistent wireguard-tools qrencode unattended-upgrades)
::::  Server configuration shown below   ::::
[Interface]
PrivateKey = [REDACTED]
Address = [REDACTED]
MTU = [REDACTED]
ListenPort = [REDACTED]
[REDACTED PEERS]
::::  Client configuration shown below   ::::
[Interface]
PrivateKey = [REDACTED]
Address = [REDACTED]
DNS = [REDACTED]
[Peer]
PublicKey = [REDACTED]
PresharedKey = [REDACTED]
Endpoint = [REDACTED]
AllowedIPs = <IP_ADDRESS>/0, ::0/0
::::    Recursive list of files in       ::::
::::    /etc/wireguard shown below       ::::
/etc/wireguard:
configs
keys
wg0.conf
/etc/wireguard/configs:
[REDACTED]
/etc/wireguard/keys:
[REDACTED]
::::            Self check               ::::
:: [OK] IP forwarding is enabled
:: [OK] Iptables MASQUERADE rule set
:: [OK] Iptables INPUT rule set
:: [OK] WireGuard is running
:: [OK] WireGuard is enabled
(it will automatically start on reboot)
:: [OK] WireGuard is listening on port 51820/udp
:::: Having trouble connecting? Take a look at the FAQ:
:::: https://docs.pivpn.io/faq
:::: WARNING: This script should have automatically masked sensitive       ::::
:::: information, however, still make sure that PrivateKey, PublicKey      ::::
:::: and PresharedKey are masked before reporting an issue. An example key ::::
:::: that you should NOT see in this log looks like this:                  ::::
:::: YIAoJVsdIeyvXfGGDDadHh6AxsMRymZTnnzZoAb9cxRe                          ::::
::::            Debug complete           ::::
:::
::: Debug output completed above.
::: Copy saved to /tmp/debug.log
:::

This will work until reboot (or at least the next time you run pivpn -d). If you want to change your subnet on the new install to the same as the old install, edit /etc/pivpn/wireguard/setupVars.conf and change the line pivpnNET=x.x.x.0 to your old subnet.
It would be nice if the install script asked what subnet you want to use during setup

Little tip needed here, likely that you'll want to remove old iptables rules for initial subnet settings.
To show all rules you can use command:
sudo iptables -t nat -L --line-numbers
You will find unneeded rule, remember it's number. To remove this rule you can use:
sudo iptables -t nat -D POSTROUTING rule_number
Next, you need to save your changes so they won't discarded after reboot. I don't know how this works in other distros, but in Ubuntu/Debian this is command:
sudo sh -c '/sbin/iptables-save > /etc/iptables/rules.v4'

Phew, thanks for reporting this, as well as for the instructions on how to solve the problem!
I was facing this exact problem after migrating my PiVPN instance to a new server.
My process for fixing the issue went more or less as follows, in case it might help someone (sorry for repetition):
Verify the mismatch between the subnets using a process similar to the original posting:
(SERVER) sudo iptables -t nat -L POSTROUTING
(CLIENT) sudo cat /etc/wireguard/[USERID].conf|grep Address
If there's a mismatch, check if adding a new POSTROUTING rule fixes it:
(SERVER) sudo iptables -t nat -I POSTROUTING 1 -s [ORIGINAL-SUBNET] -o eth0 -j MASQUERADE
Connect using the client and if everything works, continue making the changes persistent in the server.
(All of the following in SERVER)
Change your new PiVPN instance to use the old subnet:
sudo nano /etc/pivpn/wireguard/setupVars.conf # Edit pivpnNET=[ORIGINAL-SUBNET]
Backup your old iptables rules just in case:
sudo cp /etc/iptables/rules.v4 /etc/iptables/rules.v4.bk
Run PiVPN debug. This should notice the change in setupVars and create a new NAT rule:
pivpn -d
Delete the old NAT rule by first checking it's number and then removing it
sudo iptables -t nat -L --line-numbers
sudo iptables -t nat -D POSTROUTING [RULE-NUMBER]
(Confirm everything is as expected sudo iptables -t nat -L POSTROUTING)
Make NAT-rules persist reboots
sudo sh -c '/sbin/iptables-save > /etc/iptables/rules.v4' # on Debian/Ubuntu
Enjoy your working PiVPN 🎉 .
I'm not at all used to configuring these NAT-rules, and my only concern after having done the process is that my iptables rules in /etc/iptables/rules.v4 went from a very minimal looking one with 9 lines, to one with 48 lines and all sorts of entries (In my case, there's lot's of DOCKER-stuff that wasn't there before..). Maybe someone knows if this is something to be worried about?

Pre-archive closing, more information here


Minimal navigation

There has been enough times now where I'm in the wiki and want to go back to the main page or peek in the hallway that it'd be nice to have some sort of non-intrusive minimal navigation to be able to move around the various views of the webring.

We could make this footer standard for all projects.


Access to YJB Repos

Please supply all the requested information, so that your request can be processed as quickly as possible.
Which MoJ repository are you requesting access to (e.g. money-to-prisoners)?
yjaf-CMM
yjaf-Conversions
yjaf-Sentences
yjaf-View
yjaf-Returns
yjaf-RefData
yjaf-Case
What is the GitHub username of the person who needs access?
griffinjuknps
What is their full name?
Jeremy Griffin
What is their email address?
<EMAIL_ADDRESS>What company/organisation do they belong to (e.g. "MadeTech", "Department for Education")?
NPS (Northgate PS), formerly i2N, soon to be NEC Software Solutions.
What level of access should they have?
griffinjuknps - Admin
Why is this access being requested?
dev needs access to make app changes
Which MoJ team/person is responsible for this request?
Jeremy Griffin -<EMAIL_ADDRESS>- Devops for northgate

On hold pending availability of licences.


Add support for optional SQS arguments

FIFO queues are required to have a MessageGroupID, and a DeduplicationID if the queue is not set to have content-based deduplication. As we cannot provide our own arguments to SQS through the library, this makes it impossible to write to FIFO queues. This PR allows one to submit a dictionary of key-value pairs which will be sent as arguments, directly to the SQS library.

Hi John,
I'd be happy to accept this PR, but can you add tests to cover this new functionality?  And also make sure lint is passing.
Thanks!
Lloyd

Hello!
I added a test that runs the publish method with a dictionary of options - and ensures they're passed to sendMessage as expected.
I also ran the linter, which seemed to be happy with my code. The tests also seem to have 100% coverage.
Is that satisfactory?
Cheers,
John

Should be John - I'll test it out this evening, merge and publish the NPM package if everything looks good.

Hello!
I just wanted to touch base regarding the pull request. Would you mind letting me know how soon you think you would get to it?
Cheers,
John

I merged your changes (and updates some documentation) and bumped the package version to v0.4.0.  Thanks for the contribution!


Lines of chart draw over DataLabels

Sometimes Lines of chart draws over DataLabels, how i cat fix it ?
Possible draw DataLabels after draw all lines of chart ?

I move apexcharts-datalabels after line definition with developer console and fix it

Please fix SVG render

Thanks for reporting, the issue is fixed and will be released soon.


Dockerfile cleanup & fix deployment

cleanup dockerfile & use ENTRYPOINT
change mount paths to use node Temp storage for aks-periscope files

👍 Looks good, thanks.
I will give it a test from this fork tomorrow, also fyi: @JunSun17 - for name changes and if he can see any other impact.


Making model results display in terminal and does not create model

Laravel Version: 5.6.7
PHP Version: 7.1.12
Database Driver & Version:  MariaDB V10.1.29
Windows 10 Pro

Description:
Today I noticed a strange thing while creating a new model called: Con. Instead of creating a model, it displays the output of the command to the terminal, followed up by the message  Model created succesfully.
Ironically, the model is never created.
I asked other people with comparable set-ups, and they are having the same issue. I tried several terminals like: Git Bash and Windows Powershell.
Steps To Reproduce:

Set-up environment as listed above.
Run the command php artisan make:model Con.
The output of the model is displayed in your terminal rather and the model is never created.

I also have this issue while creating a Nova resource:

If Con is a reserved word while making a model, maybe a solution could be to give the user feedback in the terminal that this is not a valid name for the model?

Works for me in Windows 10?

Is Con a reserve word in one of your Windows scripts maybe?

Reading around a little - it looks like con is a Windows command to copy console output.
https://www.computerhope.com/jargon/c/copycon.htm
So it seems a limitation of some Windows systems and nothing to do with Laravel.
Interesting issue though :)

Thanks for you answer!
Indeed, it's an interesting issue. I was pulling my hair out over this haha :)


update GO:0102136

It seems EC have changed opinion on this activity, it is now under 1.14.14, see also https://www.rhea-db.org/rhea/27321
Could you please update definition reference, xrefs, and parentage of this entry? Also, in the reaction the electron acceptor changed to NADPH--hemoprotein reductase.
id: GO:0102136
name: 3-epi-6-deoxocathasterone C-23 hydroxylase activity
namespace: molecular_function
def: "Catalysis of the reaction: H+ + 3-epi-6-deoxocathasterone + NADPH + O2 <=>
 6-deoxotyphasterol + NADP + H2O." [EC:1.14.13.-, GOC:pz]
xref: MetaCyc:RXN-11531
is_a: GO:0016709 ! oxidoreductase activity, acting on paired donors, with incorp
oration or reduction of molecular oxygen, NAD(P)H as one donor, and incorporatio
n of one atom of oxygen

Also need to remove 3 digit EC from this leaf node anyways. May need RHEA request here.

What is the evidence for this reaction ? There are no pubmeds and only a link to a 3 digit EC.

I guess we can take from https://www.uniprot.org/uniprot/Q94IA6

PMID: 17138693 from MetaCyc info for CYP90D1
RHEA Ticket submitted:   [help #198719] i

Out of curiosity, what's a Rhea ticket?

@rwst, we use RHEA where possible to request a RHEA id to define the reaction. They specifiy specific chemicals as substrates .
I note however, now, from the UniProt ling for CYP90D1 (Q94IA6) that a RHEA has been assigned for the EC:<IP_ADDRESS>, which I can use for this so will now cancel the new rhea request.
THis is RHEA:27321

Yeah, I gave that rhea link in my original issue submission.

done closed


Adding stellarpay.org

This is a phishing site collecting secret keys

stellarpay.org
https://urlscan.io/result/8b5c56bb-ab67-475e-8b11-20931c70e57b/
Fake Stellar site phishing for secrets with POST /wp-json/contact-form-7/v1/contact-forms/7/feedback
Thanks for the PR!


Use strict-di

Annotate the directive to use strict dependency injection. Allows to use the directive in environments where strict-di is enforced

Fixed by https://github.com/hiebj/ng-focus-if/pull/15#issuecomment-157502064


Grant additional RBAC permissions to K8up

Since move to Operator SDK, some permissions have to be added that are
needed during runtime
Closes #213
Checklist
Remove items that do not apply. For completed items, change [ ] to [x].
-->

[x] Keep pull requests small so they can be easily reviewed.
[ ] Update the documentation.
[ ] Update tests.
[x] Link this PR to related issues.

I can not tell if those are the correct RBAC rules but the change is done using the correct tooling. When it comes to the usage of * instead of explicit list the verbs, I have my doubts. However, I can not argue why not to do so.

I can understand your reservations, I was also surprised by the need of it. However, apparently when K8up creates such runtime role, the SA running K8up needs the same permissions, otherwise kubernetes denies it for security reasons. Obviously, otherwise an SA could give itself more access than it has itself.
Maybe that can be revisited later or in K8up v2


[Bug]: In the Follow Us section show a scrolling bar.

Is there an existing issue for this?

[X] I have searched the existing issues

What happened?
In the Follow Us section show a scrolling section when hover the icons of instagram, facebook etc.
Add Screenshots
https://github.com/user-attachments/assets/831a7b02-c152-4866-be54-252e0f0801be
What browsers are you seeing the problem on?
No response
Record

[X] I agree to follow this project's Code of Conduct
[X] I'm a GSSOC contributor
[X] I want to work on this issue
[X] I'm willing to provide further clarification or assistance if needed.

done already


Correct ns of install-ws

In fact it is here https://github.com/fulcrologic/fulcro/blob/1d5f260a0c861d7156d00638a6ae3ee315d9de7a/src/main/com/fulcrologic/fulcro/inspect/inspect_ws.cljs#L70, not anywhere in https://github.com/fulcrologic/fulcro/blob/develop/src/main/com/fulcrologic/fulcro/inspect/inspect_client.cljc

OH, I forgot to update the readme...I see :p


Generation is running with single-token context

Hey, all. Thanks for the brilliant work. When I was modifying the codes, I found there seems to be some problem in generation:
generate/base.py: line 26
input_pos = torch.arange(0, T, device=device)

    # generate up to a fixed number of tokens
    for _ in range(max_returned_tokens - T):
        x = idx.index_select(0, input_pos).view(1, -1)

        # forward
        logits = model(x, input_pos)
        logits = logits[0, -1] / temperature

        # optionally crop the logits to only the top k options
        if top_k is not None:
            v, _ = torch.topk(logits, min(top_k, logits.size(-1)))
            logits = torch.where(logits < v[[-1]], -float("Inf"), logits)

        probs = torch.nn.functional.softmax(logits, dim=-1)
        idx_next = torch.multinomial(probs, num_samples=1).to(dtype=dtype)

        # advance
        input_pos = input_pos[-1:] + 1

        # concatenate the new generation
        idx = idx.index_copy(0, input_pos, idx_next)

        # if <eos> token is triggered, return the output (stop generation)
        if idx_next == eos_id:
            return idx[:input_pos]  # include the EOS token

    return idx

where
    # advance
    input_pos = input_pos[-1:] + 1

seems to set input_pos as a [1,1] input, with is unexpected for AR models.
I also referred to nanoGPT, and their implementation is:
            idx = torch.cat((idx, idx_next), dim=1)

where seems reasonable.

Hi! Why do you say it's a problem? inpus_pos can be a batch_size x seq_length tensor, which explains the two dimensions you see. The current generation code uses batch size 1 so after the prefill stage of generation, the shape is 1x1

Hi! Why do you say it's a problem? inpus_pos can be a batch_size x seq_length tensor, which explains the two dimensions you see. The current generation code uses batch size 1 so after the prefill stage of generation, the shape is 1x1

Hi! The focus here is that the second dimension is 1, instead of seq_len. Because whatever seq_len you set, since the second iteration, the dim turns into 1. I thought for autoregressive generation we iteratively add 1 to the previous seq_len to provide more context, instead of keeping it as one where the model only knows the previous ONE token. Is it a bug?

Hi @carmocca .
For your further reference, I present the case used in the code for validation, "Recommend a movie for me to watch during the weekend and explain the reason.", to see how exactly generation works.
In, generate/base.py: line 26, starting with for _ in range(max_returned_tokens - T):

first iter:
input_pos: [ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,       18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35,       36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52], which correctly selects the full prompt.
And idx_next = [13], which predicts the next token.
However, by input_pos = input_pos[-1:] + 1, input_pos turns into [53], which only points to the next token.
second iter:
x = idx.index_select(0, input_pos).view(1, -1) thus only selects [53] in the idx, therefore x = [[13]]. However, we expect x to be appended with [13], not set to [[13]]

The above results are obtained with pdb trace.

Hey @carmocca ,
Sorry to bother again. But do you have any comment on this? Should we continue use single context [1,1], or switch to [1, num_current_text for a full context during generation?

Hi! This is how text generation works the KV cache enabled. What you suggest is only necessary if you don't use it.
You can study test https://github.com/Lightning-AI/lit-gpt/blob/main/tests/test_model.py#L420-L454 which asserts that the logits are equal when doing token generation as you suggest compared to the kv cache way

Fully got it. Thanks a lot!


limiting matrix-based commutativity check

Summary
Fixes #10488.
The matrix-based commutativity check in CommutationChecker requires to construct 2^N x 2^N matrices, leading to memory problems when N is large.
I have added a new argument max_num_qubits to CommutationChecker::commute that allows to skip the commutativity check if the number of qubits of either operation exceeds this amount.
Note that simpler versions of the commutativity check not involving matrix multiplication continue to work as before (e.g., two quantum operations over disjoint sets of qubits commute regardless of the number of qubits).
Details and comments
This actually changes the behavior of commutation checking when the number of qubits is in the range from 4 to 12 (when the matrices could be constructed and multiplied). However personally I don't think we will be missing a lot of optimization opportunities. I am even thinking that we should maybe limit max_num_qubits to 2, what are your thoughts on this?
Implementation-wise, I was a bit unsure whether to make max_num_qubits to be a global constant, or to pass it as the argument to CommutationChecker::__init__, or to pass it as the argument to CommutationChecker::commute (this is what I have chosen at the moment). I can see a potential use-case of having the same instance of CommutationChecker, and then first doing some optimizations based on a lighter-weight version of the check, and then possibly more optimizations based on heavier-weight version of the check (the same instance allows to use the cached results from previous calls).

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

2 of 2 (100.0%)  changed or added relevant lines in 1 file are covered.
26 unchanged lines in 5 files lost coverage.
Overall coverage decreased (-0.02%) to 85.925%

Files with Coverage Reduction
New Missed Lines
%

qiskit/circuit/tools/pi_check.py
1
91.23%

qiskit/extensions/unitary.py
1
93.75%

qiskit/extensions/quantum_initializer/squ.py
2
80.0%

crates/qasm2/src/lex.rs
10
90.38%

crates/qasm2/src/parse.rs
12
97.13%

Totals

Change from base Build<PHONE_NUMBER>:
-0.02%

Covered Lines:
73041

Relevant Lines:
85006

💛  - Coveralls


Doesn't work on 4.2

Seems like newest updates broke the script

Yeah, I'm keeping tabs on Blender's new releases at all.
Not sure when I'll be able to get to fixing this(or how many people are even using it), it's been a while since I've last worked on it.


Use _shipping_address_2 at number suffix

At this time it is not possible to use address line 2 as a suffix.

This has been fixed in version [4.3.1] (https://github.com/myparcelnl/woocommerce/releases/tag/v4.3.1) of the MyParcel WooCommerce plugin.


Spike: Allow currently denied resources in environments repo but with review required from MP team

User Story
There are currently some resources that we prevent being created via the environments repository by restricting IAM permissions of the pipeline role.  Examples include, IAM users, VPC resources and EKS.  This is because we don't want users creating these resources unless we ok it.
The result of this is that we have users create these resources in the main MP repo, but this leads to infrastructure for users being split into 2 locations which is not great.
This spike is to determine if we could have all of the resources in the environments repo, but require a review from the MP team when certain resources are created.
CP do something similar with multiple namespace PRs or if policy documents are changed, if a PR contains multiple namespaces or policy changes, a github bot leaves a review requesting a change, which prevents a merge.  The review must be dismissed by an administrator to continue.
See examples here -
https://github.com/ministryofjustice/cloud-platform-environments/blob/main/.github/workflows/iam-role-policy-changes-check.yaml
https://github.com/ministryofjustice/github-actions/tree/main/iam-role-policy-changes-check
https://github.com/ministryofjustice/cloud-platform-environments/blob/main/.github/workflows/reject-multi-namespace-prs.yml
https://github.com/ministryofjustice/github-actions/tree/main/reject-multi-namespace-prs
There may be other ways to achieve this as well.
Value / Purpose
This would allow us to keep all user infrastructure in one location without compromising on security.
Useful Contacts
No response
Additional Information
No response
Proposal / Unknowns
Explore different options and try to get a PoC of the options working.
Definition of Done

[ ] Options explored
[x] PoC created
[ ] Demoed to team
[ ] Further issues created
[ ] Another team member has reviewed
[ ] Tests are green

So after working with Jacob last night I think
we need to take this workflow -  https://github.com/ministryofjustice/modernisation-platform/blob/1f9905d6b023077ec8720ab27d4d0b88b1508b42/.github/workflows/terraform-member-environment.yml#L40
and use it here https://github.com/ministryofjustice/modernisation-platform/actions/runs/6947227365/workflow#L169
So this will work https://github.com/ministryofjustice/modernisation-platform/actions/runs/6947227365


Passing correlation id to HttpClient

Is there any suggestion about how can I get and pass the current correlation id through HttpClient's from HttpClientFactory?
My objective is to propagate the current value to subsequent microservices.

Correlation id would be cached in HttpContext.Items as LogEventProperty and you can use it to get value.
var logEventProperty =  HttpContext.Items["CorrelationId"] as LogEventProperty;
var correlationId = logEventProperty.Properties["CorrelationId"].LiteralValue().ToString();


Cannot be used in fragment

picker.show(getSupportFragmentManager(), "CURRENCY_PICKER");
This line is giving me an error while using this in fragment

getSupportFragmentManager() is a function of FragmentActivity.  You need to call it on the parent Activity of the Fragment.
Many examples of using a Fragment store the Activity inside the Fragment as:
private Activity mActivity;

That would mean you need to call (note: this is using the new syntax as of v1.2.2):
picker.showDialog(mActivity.getSupportFragmentManager())

getSupportFragmentManager() is a function of FragmentActivity. You need to call it on the parent Activity of the Fragment.
Many examples of using a Fragment store the Activity inside the Fragment as:
private Activity mActivity;

That would mean you need to call (note: this is using the new syntax as of v1.2.2):
picker.showDialog(mActivity.getSupportFragmentManager())

still my error is not resolved

@Vishal-Likhitkar if you have a new problem, open a new issue, and actually describe the problem your are having including the text of the error you see. Commenting on a closed issue from 2.5 years ago without actually providing any description of your Penn won't help you.

Okay Thank you for reply.
I am attaching a separate issue.
On Sun, Jun 6, 2021, 8:58 AM MK @.***> wrote:

@Vishal-Likhitkar https://github.com/Vishal-Likhitkar if you have a new
problem, open a new issue, and actually describe the problem your are
having including the text of the error you see. Commenting on a closed
issue from 2.5 years ago without actually providing any description of your
Penn won't help you.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mukeshsolanki/country-picker-android/issues/62#issuecomment-855330828,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ASDYR42G5ALU6KCNAQEKTSLTRLTPRANCNFSM4FKXBGFQ
.


run time and loss

thank you for your code , what is the reason of that the loss is different at every time i restart the run time and run the train ?
can you help me please

The training process include random process, e.g. random data sequence order, random initialization etc.
If you want deterministic process, you need to fix random seed for python random module, numpy random module and tensorflow random seed.

thank you for your replay
i have read your paper and i found that the code converge at 4000 epochs but colab will be disconnected at 180 epochs . i want to use the final weight as initial . can you help me please ?


Add StableHLO dialect version for bytecode

This enables (at least) invalidating previously generated StableHLO bytecode serialization (not using VHLO) by bumping the version. The version is optionally set and if not set, not serialized - this would result in no changes to what is serialized by default today unless explicitly set.
This would enable a better error message than the existing where a loaded bytecode would fail and knowledge required as to why:
Before
<unknown>:0: error: 'stablehlo.broadcast_in_dim' op attribute 'broadcast_dimensions' failed to satisfy constraint: either a DenseI64ArrayAttr or a 1-dimensional I64ElementsAttr.
<unknown>:0: note: see current operation: %4 = "stablehlo.broadcast_in_dim"(%1) {broadcast_dimensions = array<i64>} : (tensor<f32>) -> tensor<500xf32>
<unknown>:0: note: in bytecode version 6 produced by: MLIR19.0.0git

Approximately after:
reading newer dialect version than supported

(Note this could also be refined, but at least point to version skew vs folks being concerned about generation error)
This does not represent a policy change nor an automatic test added.
Could also add to CHLO but wanted to discuss first.

Ping?


⚠ Samsung has degraded performance

In ebb9333, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 2688 ms

Resolved: Samsung performance has improved in 7a0ca88.


[TF 2.0] Nested Gradient Tape - unconnected graphs

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes
OS Platform and Distribution: MacOS 10.15.1
TensorFlow installed from binary (pip 19.3.1)
TensorFlow version: v2.0.0-rc2-26-g64c3d382ca 2.0.0
Python version: Python 3.6.5

Describe the current behavior
A copy of my model (model_copy) should be trained one step, then I need my meta_model to be trained with the loss of my model_copy. It seems, that the graphs are unconnected.
It only works, if I use the meta_model for the training step.
Describe the expected behavior
I would expect, that model_copy is known to both gradient tapes and can be used w/o using meta_model.
Code to reproduce the issue
import tensorflow as tf
import tensorflow.keras.backend as keras_backend
import tensorflow.keras as keras

class MetaModel(keras.Model):
    def __init__(self):
        super().__init__()
        self.hidden1 = keras.layers.Dense(5, input_shape=(1,))
        self.out = keras.layers.Dense(1)
    def forward(self, x):
        x = keras.activations.relu(self.hidden1(x))
        x = self.out(x)
        return x

def copy_model(model, x):
    copied_model = MetaModel()
    copied_model.forward(x)
    copied_model.set_weights(model.get_weights())
    return copied_model

def compute_loss(model, x, y):
    logits = model.forward(x)  # prediction of my model
    mse = keras_backend.mean(keras.losses.mean_squared_error(y, logits))  # compute loss between prediciton and label/truth
    return mse, logits

optimizer_outer = keras.optimizers.Adam()
alpha = 0.01
with tf.GradientTape() as g:
    # meta_model to learn in outer gradient tape
    meta_model = MetaModel()
    # inputs for training
    x = tf.constant(3.0, shape=(1, 1, 1))
    y = tf.constant(3.0, shape=(1, 1, 1))

    meta_model.forward(x)
    model_copy = copy_model(meta_model, x)
    with tf.GradientTape() as gg:
        loss, _ = compute_loss(model_copy, x, y)
        gradients = gg.gradient(loss, model_copy.trainable_variables)
        k = 0
        for layer in range(len(model_copy.layers)):
            """ If I use meta-model for updating, this works """
            # model_copy.layers[layer].kernel = tf.subtract(meta_model.layers[layer].kernel,
            #                                               tf.multiply(alpha, gradients[k]))
            # model_copy.layers[layer].bias = tf.subtract(meta_model.layers[layer].bias,
            #                                             tf.multiply(alpha, gradients[k + 1]))

            """ If I use model-copy for updating instead, gradients_meta always will be [None,None,...]"""
            model_copy.layers[layer].kernel = tf.subtract(model_copy.layers[layer].kernel,
                                                          tf.multiply(alpha, gradients[k]))
            model_copy.layers[layer].bias = tf.subtract(model_copy.layers[layer].bias,
                                                        tf.multiply(alpha, gradients[k + 1]))

            k += 2

    # calculate loss of model_copy
    test_loss, _ = compute_loss(model_copy, x, y)
    # build gradients for meta_model update
    gradients_meta = g.gradient(test_loss, meta_model.trainable_variables)
    """ gradients always None !?!!11 elf """
    optimizer_outer.apply_gradients(zip(gradients_meta, meta_model.trainable_variables))

Other info / logs
Is it intended to work as above? This would force me not to be able to use a different optimizer in the inner loop, as the networks need somehow to be connected.

@janbolle,
When trying to reproduce your issue, I encounter the error, ValueError: No gradients provided for any variable: ['dense_8/kernel:0', 'dense_8/bias:0', 'dense_9/kernel:0', 'dense_9/bias:0'].. Can you please help us reproduce the issue. Here is the Gist. Thanks!

@rmothukuru , thanks for your reply.
This is my problem. gradients_meta are always [None, None, ...].
So TF tells me that there are no gradients provided..

@janbolle,
So do you mean you are encountering same error as that of mine. Please confirm.

@rmothukuru , yes, same error on my side.

Chandan Kumar
On Nov 18, 2019 6:24 PM, "Jan Bollenbacher"<EMAIL_ADDRESS>wrote:
@rmothukuru https://github.com/rmothukuru , yes, same error on my side.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/34335?email_source=notifications&email_token=AMUWCJI3UKK2QVZRZWSF4ODQUKGA5A5CNFSM4JOEEYF2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEKKIUI#issuecomment-555000913,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AMUWCJJLEZ5UJYBX3FEMQZ3QUKGA5ANCNFSM4JOEEYFQ
.

Could reproduce the error with TF Version 2.0. Here is the Gist. Thanks!

Also what is very odd is if I print the weights of the layers before and after the training, they are available. But if I use the function model_copy.get_weights() it results in an empty array.
Following code:
        k = 0
        for layer in range(len(model_copy.layers)):
            # calculate adapted parameters w/ gradient descent
            # \theta_i' = \theta - \alpha * gradients
            print("pre: ", model_copy.layers[layer].kernel.shape, model_copy.layers[layer].kernel)
            model_copy.layers[layer].kernel = tf.subtract(model_copy.layers[layer].kernel,
                                                          tf.multiply(alpha, gradients[k]))
            model_copy.layers[layer].bias = tf.subtract(model_copy.layers[layer].bias,
                                                        tf.multiply(alpha, gradients[k + 1]))
            print("post: ", model_copy.layers[layer].kernel.shape, model_copy.layers[layer].kernel)
            k += 2
    print(model_copy.get_weights())  # results in empty array

@jvishnuvardhan do you need further information?
@rmothukuru did you connect the right person?

I was able to replicate the issue with Tf-nightly==2.2.0.dev20200318.
Please find the gist here. Thanks!

I was able to replicate the issue with Tf-nightly==2.3.0-dev20200612.Please, find the gist here.Thanks!.

I was able to replicate the issue with Tf-nightly==2.4.0-dev20200806, Please, find the gist here.

Has there been any updates with this issue? Running into a similar None gradients case when using nested tf.GradientTapes.

Looking for updates on this as well! I am able to get my U-Net model to do an inner update, but this issue shows its only possible to do one inner update.

Was able to replicate the issue in TF v2.5,please find the gist here..Thanks !

Was able to replicate the issue with TF 2.6.0-dev20210606,please find the gist here ..Thanks!

Hi There,
This is a stale issue. As you are using an older version of tensorflow, we are checking to see if you still need help on this issue. Please test the issue with the latest TensorFlow (TF2.7 and tf-nightly). If the issue still persists with the newer versions of TF, please feel free to open it in keras-team/keras repository by providing details about the issue and a standalone code to reproduce the issue. Thanks!
Please note that Keras development has moved to a separate Keras-team/keras repository to focus entirely on only Keras. Thanks!

Still an issue with TF 2.7. Gist here

Was there ever a resolution here? Curious why the issue was closed after the last poster confirmed it was still an issue in TF 2.7.


Outdated content

The "Restart services" section is outdated, the Service names have changed and the page should be updated as a result.
Current options are called:
"Finance and operations apps service"
"Data management workspace"
"Financial reporting service"
See
 here.
Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 42e51870-4f46-c1ae-4d85-2e74861c4314
Version Independent ID: bea7397e-9977-47d7-5fb1-57c73b9bf510
Content: Maintenance operations for deployments - Finance & Operations | Dynamics 365
Content Source: articles/fin-ops-core/dev-itpro/deployment/maintenanceoperationsguide-newinfrastructure.md
Service: dynamics-365-cross-app
Product: ****
Technology: ****
GitHub Login: @laneswenka
Microsoft Alias: laswenka

@miham-aib, thank you for calling this to our attention!  I just updated the content -- the updated content will be published later today.  Thanks again!