[feature request] Implement goal-parameterized algorithms (HER)

I'd like to implement Hindsight Experience Replay (HER). This can be based on a whatever goal-parameterized RL off-policy algorithm.
Goal-parameterized architectures: it requires a variable for the current goal and one for the current outcome. By outcome, I mean anything that is requires to compute the current outcome in the process of targeting the goal, e.g. the RL task is to reach a 3D target (the goal) with a robotic hand. The position of the target is the goal, the position of the hand is the outcome. The reward is a function of the distance between the two. Goal and outcome are usually subparts of the state space.
How Gym handles this: In Gym, there is a class called GoalEnv to deal with such environments.

The variable observation_space is replaced by another class that contains the true observation space observation_space.spaces['observation'], goal space (observation_space.spaces['desired_goal']) and outcome space(observation_space.spaces['achieved_goal']).
The observation returned first by env.step is now a dictionnary obs['observation'], obs['desired_goal'], obs['achieved_goal'].
The environment defines a reward function (compute_reward), that takes as argument the goal and the outcome to return the reward
It also contains a sample_goal function that simply samples a goal uniformly from the goal space.

Stable-baselines does not consider this so far.  The replay buffer, BasePolicy, BaseRLModels OffPolicyRLModels only consider observation, and are not made to include a notion of goal or outcome. Two solutions:

Adapt these default classes to allow the representation of goals and outcomes in option.
Concatenate the goal and outcome to the observation everywhere so that the previously mention classes don't see the difference. This requires to keep track of the indices of goals and outcomes in the full obs_goal_outcome vector. Ashley started to do this from what I understood. However, he did not take into account the GoalEnv class of Gym, which I think we should use as it's kind of neat, and it's used for the Robotic Fetch environment, which are kind of the only one generally used so far.

I think the second is more clear as it separates observation from goals and outcomes, but probably it would make the code less easy to follow, and would require more changes than the first option. So let's go for the first as Ashley started.
First thoughts on how it could be done.

we need (as Ashley started to do), a wrapper around the gym environment. GoalEnv are different from usual env because they return a dict in place of the former observation vector. This wrapper would unpack the observation in obs, goal, outcome from the GoalEnv.step. It would return a concatenation of all of those. Ashley considered that the goal was in the observation space, so that the concatenation was twice as long as the observation. This is generally not true. So we would need to keep as attribute the size of the goal and outcome spaces. It would keep the different spaces as attributes, keep the function to sample goals, and the reward function.

A multi-goal replay buffer to implement HER replay. It takes the observation from the buffer and redecompose it in obs, goal and outcome before performing replay.

I think it does not require too much work after what Ashley started to do. It would be a few modifications to integrate the GoalEnv of gym, as it is a standard way to use multi-goal environments. Then correct the assumption he made about the dimension of the goal.
If you're all ok, I will start in that direction and test them on the Fetch environments. In the baselines, their performance is achieved with 19 processes in parallel. They basically average the update of the 19 actors. I'll try first without parallelization.

Some questions to clarify my understanding:

goal and outcome are part of the observation? (Is that what you mean by "Goal and outcome are usually subparts of the state space." ?)
in a GoalEnv, does observation key contains both outcome and goal or do we need to concatenate them? (I think we need to concatenate them but I'm not sure)

So let's go for the first as Ashley started.

You meant second, no?

I think it does not require too much work after what Ashley started to do.

Good news then =)

, I will start in that direction and test them on the Fetch environments

For debugging and for your tests, Fetch env is ok. However, we will need either to adapt an existing env (e.g. Pendulum-v0) or create an artificial one (e.g. Identity env) in order to write unit tests for it (Mujoco requires a license and we don't have one for travis...).

I wanted to say: goal and outcome spaces are usually subspaces of the state space (but not always). Example: FetchReach (place gripper at 3D location). The observation is made of the position and velocities of all the joints, while the goal is only a 3D location, the outcome is the 3D position of the hand. In that case the outcome is a subpart of the state.

In GoalEnv, the observation return is a dictionary with three keys: observation, achieved_goal and desired_goal. What I'm doing now is wrapping the gym env to concatenate everything into one observation, and having some variable saying which indexes are what, so that HER can do its substitution. After that, I wrap in a DummyVecEnv.

(Yes I meant the second, as Ashley).
I guess we could modify easily Pendulum or MountainCarContinuous yes.

Last problem I encountered: DDPG considers the return to be the sum of all rewards. It's not absolutely general, sometimes it's only the last reward (in the Fetch env for instance). A solution can be to add a string parameter to the class return_func='last' or 'sum'.

Last problem I encountered: DDPG considers the return to be the sum of all rewards. It's not absolutely general, sometimes it's only the last reward

I thought return meant sum of discounted reward for one episode... In the Fetch env, I think it is a sparse reward (0 almost everywhere except when reaching the goal), so last reward is the same of the sum no?
Does it affect the training of the algorithm (the difference in formulation) or is it only for logging?

In Fetch, always -1 and 0 when the goal is touched. It's only for logging, the algorithm uses the transition-based rewards. In that case, saying that the sum is -49 for 50 steps for instance, does not indicate whether the goal was reached in the middle of the episode (in that case the episode is not solved), or at the end (episode solved).

I see... then for GoalEnv, it makes sense to have that feature (showing last return only).

The problem is that it's computed outside of the env, in the learn function of the algorithm. That would require to update all algorithm to allow that.

That would require to update all algorithm to allow that.

I would update only algorithms that can be used with HER (and therefore GoalEnv), I'm not sure if it makes sense for other type of env.

Is this still being worked on? This feature is something I'm quite interested in.

Update:
I implemented it but it's still bugged somehow. It runs but it does not learn anything on FetchReach. It's on a fork from stable baselines. I can give you access if you want to check it out.
In the meantime I also tried implementing it using sac's spinning up. It learns perfectly on FetchReach but nothing on FetchPush. I'm quite puzzled because I already tried to reproduce results on FetchPush last year using a TD3 base and it also worked perfectly on FetchReach and not on FetchPush.
Either I do something wrong, or there is some special trick in the OpenAI Baselines that I didn't catch. Their version uses 19 worked in parallel, each doing 2 rollouts, computing an update using a batch of 256 and summing the 19 updates (yes they sum, they don't average). I would say it's roughly equivalent to do 38 collection rollout, then to use a 19 times bigger batch size and 19 times bigger learning rate (the sum of 19 updates). I tried this also but it got even worse.
I don't have much time these days so it's on pause right now.

@ccolas don't hesitate to ping me if you need some help to double check some parts ;)

@ccolas @hill-a I'm taking over for this one.
I have read the paper again (and some implementations) and started to implement it (mostly from scratch).
There is one difference that was not mentioned that I also spotted: the original paper creates new transitions (by sampling goals) online after each rollout (not after each step) but this should be ok (in the sense we can overcome that limitation by using a custom replay buffer)
Here is my current plan (and current progress):

I created a FlippedBitEnv to iterate and debug quickly (both with continuous and discrete actions)
I plan to ensure that the environment inherits from GoalEnv
I started from scratch but in fact, now, my current plan really ressemble what @hill-a did ^^#
my current implementation requires the user to pass a sampling_strategy and a get_achieved_goal callable. The first argument refers to HER replay strategy, the second is a callable that convert an observation to a goal (so we can use env.compute_reward), I'm not sure if it is needed (still in early stage of development)

To overcome current limitation of stable-baselines (dict obs are not supported), I'll do something similar to @hill-a , using a wrapper over the environment.
Roadmap:

[x] implement dev env FlippedBitEnv
[ ] have a first limited version working on FlippedBitEnv , i.e. DQN with FINAL strategy
[ ] add support for DDPG and SAC and make it work for the FlippedBitEnv
[ ] implement more replay strategies
[ ] make them work on Highway env
[ ] polish code
[ ] documentation
[ ] add support for more type of env obs spaces  (Discrete, Binary, MultiDiscrete?)
[ ] add full support for all env obs spaces (e.g. where the observation and goal space are not of the same type)

Note: I consider the last point to be not a priority

@ccolas
I have a first working draft with DQN (tried with a flipped bit env with n_bits=30 and it worked =) ),
you can have a look at it even if it is not fully polished yet.
In short, I made those choices, which is mostly wrappers:

HER will be just a wrapper around a RL model (which makes sense)
instead of creating a new replay buffer class, I created a wrapper for HER replay buffer, that is were all the magic happens
i have a env wrapper that concatenate dict observation and can do the inverse operation

to sum it up, the implementation now looks pretty simple, it is just a wrapper around a model and an env (I still don't understand why the original baselines were so complicated).
My next step is to make it work with SAC ;)

Super cool, thanks !
Keep me updated about SAC. I found it was easy to make it work on FetchReach but impossible to make it work on FetchPush. If you don't have a mujoco license, I can run tests on my side when you're done.

@ccolas In my experiments, I found that SAC is quite hard to tune for problem with deceptive reward (I did not find good hyperparameters yet for MountainCar-v0 for instance), so this can be an issue when working with problem like FetchPush.
However, in my current experiments with HER + DDPG, I managed to get it work on harder problem where HER + SAC is failing (ex: continuous bit flipping env work for HER + DDPG when N_BITS=12).
I think I will open an issue on the original repo.
Also, the original baselines have several tricks (and I don't which one is useful or not) compared to the original HER paper:

they use 19 workers and average the parameters
they normalize the input, goal and obs (this can be done using VecNormalize(norm_reward=False))
they formulate sampling additional goal in a different manner: instead of adding artificial transition, they do it when sampling from the replay buffer

PS: SAC and DDPG are now supported on my dev branch ;) (just missing saving/loading for now)

@ccolas  interesting discussion on SAC with sparse rewards is happening there ;):
https://github.com/rail-berkeley/softlearning/issues/76

Very good work, thanks !
About the additional tricks used in the baseline, I can comment them based on my experience:

19 workers: they do not average the updates, but sum them. It kind of relates to the A2C way of doing things, having workers in parallel, each with their replay buffer, each doing an update. Changing this parameter has a great influence on performance (it's not equivalent to running with 1 worker for a longer time). I suspect it's part of the reason why it works better than other implementations on FetchPush.
I remove don't normalize obs and goals in my experiments and it works just as well. I guess it depends on the values of the obs and goals, in the Fetch environments they are small and centered so it doesn't impact much.
I think it's better to use counterfactual learning at sampling instead of at storing time. This way the size of the replay buffer does not depend on the amount of replay. You also might want to tune replay in particular ways. For example I replay more the tasks that have higher learning progress, and I need to have the control on replay whenever I want to do an update, thus at sampling time.

I'll try to run it on the mujoco envs to see what it gives !
Thanks for the thread on SAC and CMC, I used that environment in a previous project, it'll be interesting !

This way the size of the replay buffer does not depend on the amount of replay.

That's true, but I also feel it will be less clear in the implementation, no?
among the tricks I forgot, there is also a l2 loss on the actions. (and I have to check the DDPG implementation, to see what is different from the one in the baselines)
Linking related issue of highway envs: https://github.com/eleurent/highway-env/issues/15

Very good work, thanks !
About the additional tricks used in the baseline, I can comment them based on my experience:

19 workers: they do not average the updates, but sum them. It kind of relates to the A2C way of doing things, having workers in parallel, each with their replay buffer, each doing an update. Changing this parameter has a great influence on performance (it's not equivalent to running with 1 worker for a longer time). I suspect it's part of the reason why it works better than other implementations on FetchPush.
I remove don't normalize obs and goals in my experiments and it works just as well. I guess it depends on the values of the obs and goals, in the Fetch environments they are small and centered so it doesn't impact much.
I think it's better to use counterfactual learning at sampling instead of at storing time. This way the size of the replay buffer does not depend on the amount of replay. You also might want to tune replay in particular ways. For example I replay more the tasks that have higher learning progress, and I need to have the control on replay whenever I want to do an update, thus at sampling time.

I'll try to run it on the mujoco envs to see what it gives !
Thanks for the thread on SAC and CMC, I used that environment in a previous project, it'll be interesting !

@ccolas The current hyperparameters I'm using for the highway-env (for SAC and DDPG) and that works better than the default one:
SAC:
n_sampled_goal = 4

model = HER('MlpPolicy', env, SAC, n_sampled_goal=n_sampled_goal, goal_selection_strategy='future',
            verbose=1, tensorboard_log='/tmp/sac', buffer_size=int(n_sampled_goal * 1e6),
            tau=1-0.95, learning_rate=1e-3,
            gamma=0.95, batch_size=256)

DDPG:
n_actions = env.action_space.shape[0]
noise_std = 0.2
action_noise = NormalActionNoise(mean=np.zeros(n_actions), sigma=noise_std * np.ones(n_actions))

n_sampled_goal = 4

# to change the policy architecture:
# policy_kwargs=dict(layers=[256, 256])

model = HER('MlpPolicy', env, DDPG n_sampled_goal=n_sampled_goal, goal_selection_strategy='future',
            verbose=1, buffer_size=int(n_sampled_goal * 1e6),
            tau=1-0.95, actor_lr=1e-3, critic_lr=1e-3,
            gamma=0.95, batch_size=256, action_noise=action_noise, nb_rollout_steps=20)

Note: I did not change the policy architecture yet ([64, 64] for now instead of [256, 256, 256])
I also added the success rate to the logs ;)

Also related for the tricks: https://github.com/vitchyr/rlkit/pull/35

Update: working version with HER + DDPG on FetchPush. Still a bug with VecEnvs but should be ready to merge soon (see PR)

I saw that the algorithms can perform well in DDPG+HER and SCE+HER for Fetch Push Environment. How about Pick and Place? I saw the issue mentioned by fisherxue is still in the status of Open.

You can take a look at the trained agent (and hyperparameters) in the zoo ;)
https://github.com/araffin/rl-baselines-zoo/pull/53


get mumps-mpi onto the new compilers

Checklist

[x] Used a fork of the feedstock to propose changes
[x] Bumped the build number (if the version is unchanged)
[x] Reset the build number to 0 (if the version changed)
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

There appears to be something wrong with the 3.1.3 toolchain builds of openmpi, since they are failing on ptscotch itself as well. I can't tell if this is an openmpi problem, a build/link problem, a conda problem, or something else.
openmpi 4.0 is in PR now, so I'm inclined to mark this one build as broken and hope 4.0 fixes it.

Rebuild fixed it after blacklisting toolchain ompi 3.1.3. Still a mystery as to why it was broken.


Docs.MS Release Updates for azure-spring-cloud-starter-storage-queue

Update docs metadata and targeting for release of azure-spring-cloud-starter-storage-queue

Docs Build status updates of commit c70593a:
:warning: Validation status: warnings

File
Status
Preview URL
Details

docs-ref-services/latest/spring-cloud-starter-storage-queue-readme.md
:warning:Warning
View (azure-java-stable)
Details

docs-ref-services/latest/spring-boot-starter-active-directory-b2c-readme.md
:warning:Warning
View (azure-java-stable)
Details

docs-ref-services/latest/spring-cloud-starter-storage-queue-readme.md

Line 2, Column 1: [Warning-ms-prod-and-service] Only one of the following attributes can exist: 'ms.prod', 'ms.service'. Use ms.prod for on-premise products, or ms.service for cloud services.
Line 9, Column 16: [Warning-ms-prod-technology-invalid] Invalid value for 'ms.technology': 'azure' is not valid with 'ms.prod' value 'azure'.
Line 11, Column 13: [Warning-ms-service-subservice-invalid] Invalid value for 'ms.service': 'springcloud'.
Line 2, Column 1: [Suggestion-description-missing] Missing required attribute: 'description'.
Line 24, Column 1: [Suggestion-preserve-view-not-set] Did you mean to link to a specific version? If so, add &preserve-view=true to the URL: https://docs.microsoft.com/java/azure/jdk/?view=azure-java-stable

docs-ref-services/latest/spring-boot-starter-active-directory-b2c-readme.md

Line 2, Column 1: [Warning-ms-prod-and-service] Only one of the following attributes can exist: 'ms.prod', 'ms.service'. Use ms.prod for on-premise products, or ms.service for cloud services.
Line 9, Column 16: [Warning-ms-prod-technology-invalid] Invalid value for 'ms.technology': 'azure' is not valid with 'ms.prod' value 'azure'.
Line 11, Column 13: [Warning-ms-service-subservice-invalid] Invalid value for 'ms.service': 'springboot'.
Line 2, Column 1: [Suggestion-description-missing] Missing required attribute: 'description'.
Line 26, Column 1: [Suggestion-preserve-view-not-set] Did you mean to link to a specific version? If so, add &preserve-view=true to the URL: https://docs.microsoft.com/java/azure/jdk/?view=azure-java-stable

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel


'BUY' option for accounts with positive STX balances

Fiat onramp feature:
It would be nice to have a 'Buy' option for current existing user/account that has positive STX value on the account
Currently we only have this option for an account with 0 STX Or for a new account only.

@beguene Would you be able to include this for all accounts before the release, please?

@andresgalante @Eshwari007 This is fixed in the same PR https://github.com/hirosystems/stacks-wallet-web/pull/1981


Update Kitodo.Production and scripts for webhook manager version

Using current master state of Kitodo.Production
Adjust sql examples
Improve process dir script to use
Makefile improvements: building only specific services using SERVICES env, confirm make down

@bertsky Adjusted to the changed asynchronous behavior of OCR-D manager. Synchronous is now the default.

Wait! One bug did slip through:
https://github.com/slub/ocrd_kitodo/blob/b8a38856dc3fee1e5d45ce4ea1b1b6d834db077b/.env#L22-L23
Notice the wrong variable name for the pw?
Live this causes:
MongoServerError: Authentication failed.

Also fixed in https://github.com/slub/ocrd_kitodo/commit/c59b4cbd81f57b4de377b78ac641f977df2cd4c5.

@markusweigelt, short explanation: in trying to make this work, I noticed I do need to update the Controller (because our timeout was partially caused by glitches there, not SSH itself) and added waiting for ocrd-database (because that is what surfaced on the CI as well). So I basically had to put all on one card. Hope all this is sufficient now :crossed_fingers:


⚠ ClamAV has degraded performance

In c134bcd, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 199 ms

Resolved: ClamAV performance has improved in 5ff5ff5.


Add navmesh constraint support for movements

Resolves #163
Documentation on navmeshes: https://github.com/n5ro/aframe-extras/tree/master/src/pathfinding
Note: Because our cameras are offset @ y=1.6, navmeshes must be created with base 1.6 height.

Edit: As of 738e730 the 1.6 offset  should no longer apply to the mesh. Consider the mesh origined at 0,0,0.


Refs CVE-2022-34265 -- Properly escaped Extract() and Trunc() parameters.

These are the database API changes announced in the blog post.

buildbot, test on oracle.

buildbot, test on oracle.

Thanks y'all for reviews :1st_place_medal:

Did you mean to rename DatabaseOperations._convert_field_to_tz() to _convert_sql_to_tz() only for PostgreSQL (and not Oracle or MySQL)?

Did you mean to rename DatabaseOperations._convert_field_to_tz() to _convert_sql_to_tz() only for PostgreSQL (and not Oracle or MySQL)?

Not really, good catch :dart:, see #15832.


Handle IProjectInstancePropertiesProvider overrides to access implicit properties properly

@dotnet/project-system for review

@davkean addressed your concerns!


remove incorrect line on when to use Tiled Architecture

Summary
remove incorrect line on when to use Tiled Architecture
Why / Goal
Non-tiled approach is also available in Flink
Test Plan
Needs testing
Checklist

[x] Documentation update

Reviewers

@nikhilsimha mind reviewing? :)


pyenv: add back the logic to search for the pyenv command if not found

This reverts back to looking for pyenv in predefined directories if the pyenv command is not found, but discards the setting of PYENV_ROOT which should default to $HOME/.pyenv as per their documentation.
Such search will only be done if the pyenv command is not found, and we will only look for the homebrew prefix if all else fails. This way the load will take progressively longer as is needed, not taking so long for everybody as it did before.
This should satisfy users who complained that, after #6165, they can no longer use pyenv due to it not being found in their $PATH (see https://github.com/robbyrussell/oh-my-zsh/pull/6165#issuecomment-385840876, https://github.com/robbyrussell/oh-my-zsh/commit/8efcf2776b2e7e00c073b149f76c04715aa695d3).
Fixes #6810.
cc @cyphus @Kriechi @quietcoolwu @humitos @ysolis @brandonb927 @yuvalek

Looks like a reasonable change.


XTable 1st Release

Feature Request / Improvement
Copying the email sent to<EMAIL_ADDRESS>https://lists.apache.org/thread/kjwtg55rdcllvyol6sqszd6ozgto7z45
[DISCUSS] Targeting July Release for Apache XTable (Incubating)
Hi everyone,

I hope this message finds you well.

Based on the guidelines
<https://incubator.apache.org/guides/releasemanagement.html> mentioned for
Apache incubating projects - Podlings can use non-ASF releases as an
opportunity to find ASF policy violations and begin resolving them. I have
put up a release guide PR for the upcoming Apache XTable (Incubating)
release and propose running an interim non-ASF release to iron out issues.
https://github.com/apache/incubator-xtable/pull/434

After resolving issues from the non-ASF release, I propose we target an
ASF-compliant release for Apache XTable (Incubating) in July. Please share
your thoughts and any objections regarding this proposed timeline. Your
feedback is crucial to ensure we meet the necessary requirements and
address any potential issues before the release.

Looking forward to your input.

Best regards,
Vinish

Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

As per suggestions from @jcamachor, @zabetak and @vinothchandar - we will be adding doing an ASF release using the
"Work In Progress Disclaimer"

Pushed PR for DISCLAIMER-WIP  -> https://github.com/apache/incubator-xtable/pull/487

Staged source releases on dis.apache.org
https://dist.apache.org/repos/dist/dev/incubator/xtable/
Created PR for adding release profile and mvn deploy plugins -> https://github.com/apache/incubator-xtable/pull/493
Blocker: Not able to upload the artifacts to https://repository.apache.org
mvn deploy -Prelease -DskipTests  -DdeployArtifacts=true
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:3.1.1:deploy (default-deploy) on project xtable: Failed to deploy artifacts: Could not transfer artifact org.apache.xtable:xtable:pom:0.1.0-rc1 from/to apache.releases.https (https://repository.apache.org/service/local/staging/deploy/maven2): status code: 400, reason phrase: Bad Request (400) -> [Help 1]
I have checked my ~/.m2/settings.xml and ensured my username and password for the nexus repository is populated correctly.
<settings>
    <servers>
        <server>
            <id>apache.releases.https</id>
            <username>$USERNAME</username>
            <password>$PASSWORD</password>
        </server>
        <server>
            <id>apache.snapshots.https</id>
            <username>$USERNAME</username>
            <password>$PASSWORD</password>
        </server>
    </servers>
</settings>

Any help from people who have done releases for incubating projects is appreciated.

The artifacts have been uploaded to staging repository.
Repository Path: /org/apache/xtable/incubator-xtable-utilities/0.1.0-rc1/incubator-xtable-utilities-0.1.0-rc1-bundled.jar

Based on comments provided in rc-1, have fixed the artifact  names for the bundles that didn't have the incubator prefix.
Created 0.1.0-rc2 which can be accessed through https://repository.apache.org/content/repositories/orgapachextable-1001/
In your pom.xml, you need to make the following changes to access the rc2 bundles.
    <repositories>
        <repository>
            <id>apache-staging</id>
            <name>Apache Staging Repository for XTable</name>
            <url>https://repository.apache.org/content/repositories/orgapachextable-1000/</url>
        </repository>
    </repositories>
    ....
          <dependency>
            <groupId>org.apache.xtable</groupId>
            <artifactId>incubator-xtable-core</artifactId>
            <version>0.1.0-rc2</version>
        </dependency>

Based on feedback provided by @jcamachor, moving the "incubating" regex to the artifact version instead of the artifact name.
Created 0.1.0-rc3 which can be accessed through https://repository.apache.org/content/repositories/orgapachextable-1002/
In your pom.xml, you need to make the following changes to access the rc3 bundles.
    <repositories>
        <repository>
            <id>apache-staging</id>
            <name>Apache Staging Repository for XTable</name>
            <url>https://repository.apache.org/content/repositories/orgapachextable-1002/</url>
        </repository>
    </repositories>
    ....
          <dependency>
            <groupId>org.apache.xtable</groupId>
            <artifactId>xtable-core</artifactId>
            <version>0.1.0-incubating-rc3</version>
        </dependency>

Regarding this feedback for rc3, I have updated the release guide to reflect the right source distribution names without the rcX suffix.
https://dist.apache.org/repos/dist/dev/incubator/xtable/0.1.0-incubating-rcX/apache-xtable-0.1.0-incubating-src.tgz
https://dist.apache.org/repos/dist/release/incubator/xtable/0.1.0-incubating/apache-xtable-0.1.0-incubating-src.tgz

In terms of archive names and content the RC suffix should not appear.
Once the vote passes the release artifacts must be moved from the
staging area to the release area unaltered

@zabetak For the above feedback, the maven version needs to exclude the rcX suffix as well ?
https://github.com/apache/incubator-xtable/commit/58a327f096a47d27e91ee1efa17982d7c1a75372
Current
https://repository.apache.org/content/repositories/orgapachextable-1002/org/apache/xtable/xtable-api/0.1.0-incubating-rc3/xtable-api-0.1.0-incubating-rc3.jar
Recommend. Is this correct ?
https://repository.apache.org/content/repositories/orgapachextable-1002/org/apache/xtable/xtable-api/0.1.0-incubating/xtable-api-0.1.0-incubating.jar

@vinishjail97 Regarding the naming convention of the maven artifacts (xtable-api-0.1.0-incubating.jar vs xtable-api-0.1.0-incubating-rc3.jar) I would say that the RC suffix should not be present in the jars. Once the vote passes the artifacts are no longer release candidates so it doesn't make sense to have the suffix there.
For the same reasons I think we should remove the RC suffix from the maven staging repo URL. The staging repo structure should reflect the final state since that is the one released and it cannot be modified after.
Summarizing yes the URL below is the correct one:
https://repository.apache.org/content/repositories/orgapachextable-1002/org/apache/xtable/xtable-api/0.1.0-incubating/xtable-api-0.1.0-incubating.jar

mvn deploy failed for rc4 branch because of parent pom version upgrade, reverting it back from 33 to 31.
https://github.com/apache/incubator-xtable/pull/516
Address issues in the release guide based on rc3 feedback as well ->  https://github.com/apache/incubator-xtable/pull/434/commits

Uploaded rc4 artifacts for 0.1.0-incubating to staging repository.
https://repository.apache.org/content/repositories/orgapachextable-1004/
https://dist.apache.org/repos/dist/dev/incubator/xtable/0.1.0-incubating-rc4/
Release Notes for release-0.1.0-incubating-rc4
This is the first official apache release for Apache XTable, an incubating project under the Apache Software Foundation.Apache XTable™ (Incubating) is a cross-table converter for table formats that facilitates omni-directional interoperability across data processing systems and query engines. Currently, Apache XTable™ supports widely adopted open-source table formats such as Apache Hudi, Apache Iceberg, and Delta Lake.
Raw Release Notes

[minor] add no-transfer-progress and skip tests to release-packages workflow by @the-other-tim-brown in https://github.com/apache/incubator-xtable/pull/227
[minor] fix issue in test flakiness by @the-other-tim-brown in https://github.com/apache/incubator-xtable/pull/226
Mark source client as closeable and prevent leaks by @the-other-tim-brown in https://github.com/apache/incubator-xtable/pull/221
Update README.md to add example on how to provide ABFS path. by @lordozb in https://github.com/apache/incubator-xtable/pull/229
bumping axios version to 1.6.1 by @sagarlakshmipathy in https://github.com/apache/incubator-xtable/pull/224
Add check for iceberg when incremental sync is not safe by @vamshigv in https://github.com/apache/incubator-xtable/pull/228
Doc updates from delta source tests by @sagarlakshmipathy in https://github.com/apache/incubator-xtable/pull/211
Add partition filters Testing to all Formats by @vamshigv in https://github.com/apache/incubator-xtable/pull/218
Use iterator over table changes during incremental sync by @the-other-tim-brown in https://github.com/apache/incubator-xtable/pull/225
Class Memory Optimizations by @the-other-tim-brown in https://github.com/apache/incubator-xtable/pull/216
Optimize stream to list collection  by @the-other-tim-brown in https://github.com/apache/incubator-xtable/pull/232
add path utils for fetching relative path and unit tests by @the-other-tim-brown in https://github.com/apache/incubator-xtable/pull/231
Add Microsoft Fabric tutorial for OneTable by @ashvina in https://github.com/apache/incubator-xtable/pull/237
Docker presto extension by @sagarlakshmipathy in https://github.com/apache/incubator-xtable/pull/247
Docs update quickstart iceberg by @sagarlakshmipathy in https://github.com/apache/incubator-xtable/pull/239
add info about presto on docs by @sagarlakshmipathy in https://github.com/apache/incubator-xtable/pull/248
Improve sync utility cli parse error messages by @ashvina in https://github.com/apache/incubator-xtable/pull/255
Properly report errors from sync tool, fix logging by @the-other-tim-brown in https://github.com/apache/incubator-xtable/pull/259
add handling for incomplete commit in iceberg by @the-other-tim-brown in https://github.com/apache/incubator-xtable/pull/177
only set record key field in iceberg if it is a required field by @the-other-tim-brown in https://github.com/apache/incubator-xtable/pull/260
adding images to test thumbnails for blogs by @sagarlakshmipathy in https://github.com/apache/incubator-xtable/pull/266
[MINOR][DOC] Snowflake messaging images change by @sagarlakshmipathy in https://github.com/apache/incubator-xtable/pull/270
fix image res by @sagarlakshmipathy in https://github.com/apache/incubator-xtable/pull/272
remove duplicated dependency by @jackwener in https://github.com/apache/incubator-xtable/pull/281
Update README.md by @lmccay in https://github.com/apache/incubator-xtable/pull/283
Add blog section by @sagarlakshmipathy in https://github.com/apache/incubator-xtable/pull/280
fixed relative path, added images by @sagarlakshmipathy in https://github.com/apache/incubator-xtable/pull/288
added note for adding blogs by @sagarlakshmipathy in https://github.com/apache/incubator-xtable/pull/289
SimpleDateFormat setLenient(false) by @jackwener in https://github.com/apache/incubator-xtable/pull/284
Add New OneTable Blog by @dipankarmazumdar in https://github.com/apache/incubator-xtable/pull/291
Thumbnail_Blog by @dipankarmazumdar in https://github.com/apache/incubator-xtable/pull/292
[BLOGS] Added round of blogs until dec '23 by @sagarlakshmipathy in https://github.com/apache/incubator-xtable/pull/302
[DOCS][MINOR] Added blogs button in homepage by @sagarlakshmipathy in https://github.com/apache/incubator-xtable/pull/303
#300 - Change the Enum for TableFormat to String constants for future extens… by @lmccay in https://github.com/apache/incubator-xtable/pull/301
Introduce ServiceLoader use for TargetClients by @lmccay in https://github.com/apache/incubator-xtable/pull/305
Upgrade Iceberg version by @ashvina in https://github.com/apache/incubator-xtable/pull/310
Upgrade hadoop-aws version by @arifazmidd in https://github.com/apache/incubator-xtable/pull/314
adding in StarRocks connection information by @alberttwong in https://github.com/apache/incubator-xtable/pull/315
Update demo.ipynb by @alberttwong in https://github.com/apache/incubator-xtable/pull/318
Optimize object creation for new Delta snapshot by @ashvina in https://github.com/apache/incubator-xtable/pull/326
Update demo.ipynb - PerTableConfigImpl  by @danielscamacho in https://github.com/apache/incubator-xtable/pull/333
Avoid unnecessary Iceberg datafile to onedatafile conversions by @ashvina in https://github.com/apache/incubator-xtable/pull/330
Upgrade Delta Lake version:2.4 for deletion vector support by @ashvina in https://github.com/apache/incubator-xtable/pull/349
Fix data file extraction from Hudi restore metadata with log files by @yihua in https://github.com/apache/incubator-xtable/pull/332
Upgrade spotless-maven-plugin from 2.27.2 to 2.43.0 by @zabetak in https://github.com/apache/incubator-xtable/pull/371
Move github actions under .github folder by @jcamachor in https://github.com/apache/incubator-xtable/pull/373
Create .asf.yaml file by @jcamachor in https://github.com/apache/incubator-xtable/pull/372
Docs and Website XTable Rename by @kywe665 in https://github.com/apache/incubator-xtable/pull/352
Repackaging code under the apache namespace by @jcamachor in https://github.com/apache/incubator-xtable/pull/374
Add maven-build-cache extension to speedup builds and pin hadoop dependency version by @iemejia in https://github.com/apache/incubator-xtable/pull/380
Update github actions to their latest versions and add dependabot for automatic updates by @iemejia in https://github.com/apache/incubator-xtable/pull/381
Bump actions/checkout from 3 to 4 by @dependabot in https://github.com/apache/incubator-xtable/pull/383
Fix broken links after move to Apache and rename from onetable (#384) by @iemejia in https://github.com/apache/incubator-xtable/pull/384
Make build depend on Apache Maven Parent POM by @iemejia in https://github.com/apache/incubator-xtable/pull/379
Fix some typos in the api docstrings by @hussein-awala in https://github.com/apache/incubator-xtable/pull/387
Add NOTICE by @wuchunfu in https://github.com/apache/incubator-xtable/pull/389
Refactor, rename data file classes with One in prefix by @ashvina in https://github.com/apache/incubator-xtable/pull/399
Refactor: rename schema and snapshot classes with One in prefix by @ashvina in https://github.com/apache/incubator-xtable/pull/400
Refactor: rename classes to fix the prefix by @ashvina in https://github.com/apache/incubator-xtable/pull/402
Fix README, remove outdated aws credentials param by @ashvina in https://github.com/apache/incubator-xtable/pull/403
Rename and replace 'client' in classes and comments by @the-other-tim-brown in https://github.com/apache/incubator-xtable/pull/407
Remove duplicate dependencies by @wuchunfu in https://github.com/apache/incubator-xtable/pull/357
Add spotless exclusion check folder by @wuchunfu in https://github.com/apache/incubator-xtable/pull/410
Rename all instances of OneTable to xtable where possible by @the-other-tim-brown in https://github.com/apache/incubator-xtable/pull/408
[77] Remove schema catalog and schema versions by @the-other-tim-brown in https://github.com/apache/incubator-xtable/pull/420
[424] Move pull request template to .github folder by @daragu in https://github.com/apache/incubator-xtable/pull/423
[412] Move metadata to single json field by @the-other-tim-brown in https://github.com/apache/incubator-xtable/pull/421
[Docs] Fix wrong item number in fabric doc by @daragu in https://github.com/apache/incubator-xtable/pull/425
[Infra] Add Issue template for xtable project by @daragu in https://github.com/apache/incubator-xtable/pull/427
[Demo] Fix UnknownHostException: host.docker.internal in docker demo by @daragu in https://github.com/apache/incubator-xtable/pull/430
[Module] Optimize the module structure of the xtable by @daragu in https://github.com/apache/incubator-xtable/pull/439
Bump webpack-dev-middleware from 5.3.3 to 5.3.4 in /website by @dependabot in https://github.com/apache/incubator-xtable/pull/440
License header check by @wuchunfu in https://github.com/apache/incubator-xtable/pull/364
Revert "License header check" by @the-other-tim-brown in https://github.com/apache/incubator-xtable/pull/445
[433] Exclude transitive hadoop version from Hudi by @the-other-tim-brown in https://github.com/apache/incubator-xtable/pull/441
[MINOR] Fix missing xtable prefix to utilities-bundled.jar in docs by @daragu in https://github.com/apache/incubator-xtable/pull/442
[446] Upgrade or remove dependencies with known vulnerabilities by @the-other-tim-brown in https://github.com/apache/incubator-xtable/pull/447
Bump peaceiris/actions-gh-pages from 3 to 4 by @dependabot in https://github.com/apache/incubator-xtable/pull/418
[Demo] Make start_demo script easier to call by @daragu in https://github.com/apache/incubator-xtable/pull/437
[Infra] Introduce apache rat plugin to xtable project by @daragu in https://github.com/apache/incubator-xtable/pull/450
[hotfix] Add xtable prefix to hudi-support jar in demo jupyter node by @daragu in https://github.com/apache/incubator-xtable/pull/448
[Infra] Support GitHub Action build the xtable project by @daragu in https://github.com/apache/incubator-xtable/pull/462
[DOCS] adding changes related to oss unity catalog to the docs by @sagarlakshmipathy in https://github.com/apache/incubator-xtable/pull/471
Remove azure ci by @daragu in https://github.com/apache/incubator-xtable/pull/477
Remove azure ci (addendum) by @jcamachor in https://github.com/apache/incubator-xtable/pull/479
Update pull request template by @daragu in https://github.com/apache/incubator-xtable/pull/478
Bump braces from 3.0.2 to 3.0.3 in /website by @dependabot in https://github.com/apache/incubator-xtable/pull/483
[481] Handle null stats in Delta AddFile by @the-other-tim-brown in https://github.com/apache/incubator-xtable/pull/482
[486] Add DISCLAIMER-WIP for first release by @vinishjail97 in https://github.com/apache/incubator-xtable/pull/487
Fix typo in IdTracking by @daragu in https://github.com/apache/incubator-xtable/pull/489
Remove redundant modifier for interface members by @daragu in https://github.com/apache/incubator-xtable/pull/492
[486] Change artifact name to incubator-xtable and Add deploy plugin to mvn by @vinishjail97 in https://github.com/apache/incubator-xtable/pull/498
[486] Revert artifact name with incubator prefix by @vinishjail97 in https://github.com/apache/incubator-xtable/pull/505
added docker file (#1) by @guitcastro in https://github.com/apache/incubator-xtable/pull/506
docs: fix typos and tidying by @atccirium in https://github.com/apache/incubator-xtable/pull/508
Upgrade parent pom version from 31 to 33 by @zabetak in https://github.com/apache/incubator-xtable/pull/510
Incorrect product name and distribution years in NOTICE file inside jars (#509) by @zabetak in https://github.com/apache/incubator-xtable/pull/512
Add 4 New Community Blogs to the Apache XTable Blogs section by @dipankarmazumdar in https://github.com/apache/incubator-xtable/pull/513
New Community Blogs to be added to the Apache XTable Blogs section by @dipankarmazumdar in https://github.com/apache/incubator-xtable/pull/514
Error when generating Javadoc due to old JDK7 compiler version by @zabetak in https://github.com/apache/incubator-xtable/pull/517
[486] Add release guide for XTable by @vinishjail97 in https://github.com/apache/incubator-xtable/pull/434

Full Changelog: https://github.com/apache/incubator-xtable/compare/v0.1.0-beta1...release-0.1.0-incubating-rc4


UI enhancement

closes #850

:heavy_check_mark: Checklist

[ ] Added tests for new functionality and regression tests for bug fixes
[x] Added changeset (run yarn changeset in the root)
[x] Screenshots of before and after attached (for UI changes)
[ ] Added or updated documentation (if applicable)

changes

Fixes bug in PullRequestsTableView component where table layout did not readjust properly when table header text length exceeded.

screenshot before

screenshot after

We'll need one more rebase, the build pipeline was broken and the fix should be out now in the main branch

Thank you, let's merge it!


Exercise 1.5

Spelling mistake.
of is written as fo

Fixed.
Thanks for reading with scrutiny and raising this issue


Bump Go to 1.6rc1

Signed-off-by: Tibor Vass<EMAIL_ADDRESS>Mi RC es su RC & vice versa :heart:
Basically, if we ask and expect people to test our RCs, the minimum we can do to return the favor is to test others'.

Shouldn't we also update the RPM/deb build scripts? https://github.com/docker/docker/search?utf8=✓&q=GO_VERSION

@thaJeztah fixed

LGTM :smile:

LGTM

Basically, if we ask and expect people to test our RCs, the minimum we can do to return the favor is to test others'.

:heart_eyes_cat:

yayyy!

I was curious what impact this had on build times, so I tried it out:
Go 1.5:
real    0m39.729s
user    1m31.600s
sys     0m26.280s

Go 1.6-rc1
real	0m37.315s
user	1m23.230s
sys	0m23.540s

ITS BETTERRRRRRRRRR

Oh, my GitHub email comment didn't ever come through: As long as we don't release against the release candidate, I'm all for it!

There was also some IRC discussion: https://botbot.me/freenode/docker-maintainers/msg/58875061/
Do you think we could get some PR CI jobs setup for testing against Go 1.5 like we had for Go 1.4 once upon a time?  cc @jfrazelle

Can someone ping me right as this is about to merge as I need to take the Windows CI (both Windows to Linux and Windows TP4) offline to update the version of go on those servers too.
12:09:05 ---------------------------------------------------------------------------
12:09:05 WARN: CI should be using go version 1.6rc1, but it is using 1.5.3
12:09:05
12:09:05        This CI server needs updating. Please ping #docker-dev or
12:09:05        #docker-maintainers.
12:09:05 ---------------------------------------------------------------------------
12:09:05

Looks like there's a change to url.Parse where it's actually doing some syntax checks and breaking our test (because we get a different error).

LOL
On Fri, Jan 29, 2016 at 1:42 PM, Brian Goff<EMAIL_ADDRESS>wrote:

Looks like there's a change to url.Parse where it's actually doing some
syntax checks and breaking our test (because we get a different error).
—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/pull/19797#issuecomment-176980641.

--
Jessie Frazelle
4096R / D4C4 DD60 0D66 F65A 8EFC  511E 18F3 685C 0022 BFF3
pgp.mit.edu http://pgp.mit.edu/pks/lookup?op=get&search=0x18F3685C0022BFF3

@duglin we're setting up CI for Go1.5 and Go1.4 :) So your concern should be addressed.

@tiborvass excellent! thanks!

@tiborvass actually, isn't that backwards?  Shouldn't all of the normal work we do be on what we're going to ship - meaning 1.5?  And then add a CI for 1.6.rc1?  Otherwise if we ship before 1.6 goes all then all of our ad-hoc/dev testing will have been on the wrong version.

@duglin the idea was to ship it in experimental.

it's an RC from the Go team, it's much more stable than our master branch :)

@tiborvass ship which in experimental? 1.6rc1?

@duglin yeah

Since we cannot get consensus on this, I'm closing it. Will add Go 1.6rc1 CI.


adverbs of the quote operator are not documented

See:
http://doc.perl6.org/language/quoting
q:h https://github.com/perl6/roast/blob/ab74762da156fa6fda5dd39281f7ffcc361d5a88/S02-literals/quoting.t#L428
q:f https://github.com/perl6/roast/blob/ab74762da156fa6fda5dd39281f7ffcc361d5a88/S02-literals/quoting.t#L437
q:c https://github.com/perl6/roast/blob/ab74762da156fa6fda5dd39281f7ffcc361d5a88/S02-literals/quoting.t#L449
q:a https://github.com/perl6/roast/blob/ab74762da156fa6fda5dd39281f7ffcc361d5a88/S02-literals/quoting.t#L456
q:s https://github.com/perl6/roast/blob/ab74762da156fa6fda5dd39281f7ffcc361d5a88/S02-literals/quoting.t#L463
and shorthands https://github.com/perl6/roast/blob/ab74762da156fa6fda5dd39281f7ffcc361d5a88/S02-literals/quoting.t#L482

Sorry This ISSUE is same as #376


HMS-3230 - v1: Add module_hotfixes flag

Allow passing module_hotfixes flag to both Custom and Payload repositories. This will allow passing the flag to composer, where it will disable modularity filtering on given repository.
Refs HMS-3230
Needs https://github.com/osbuild/osbuild-composer/pull/3831 to go in first

Composer is in and I've added this to the integration test on AWS, so we are even testing this.
Looks like it's passing, @croissanne care taking a look? :)
Would be awesome to have it in stage already :)


open-cluster-management/backlog#10367 add task tem

I hope this one works @qiujian16 . Let me know if there is something else that I need to do, or if you have any updates. Thank you!

/approve
/lgtm
@cadawson yes, it works

Thank you very much! I am glad to hear that it worked.


Add command to validate an NLP Tool

Parameters

The address of the NLP Tool host, e.g. http://localhost:8080
The type of NLP Tool, e.g. date annotator

Suggested command
nlp-cli tool validate --host http://localhost:8080 --tool-type date-annotator

Motivation

Validate tool as part of the CI/CD of the repo of a tool using the dockerized NLP Sandbox client
Enable developer to easily validate their tool locally
Enable the validation of NLP Tools before evaluating them

Validation SOP
Assuming that the NLP Tool is running.
General

/api/v1/ui

[ ] GET must return status code 200

/api/v1/tool

[ ] GET must return status code 200
[ ] Validate the JSON object received against schema

/api/v1/health

[ ] GET must return status code 200
[ ] Validate the JSON object received against schema

Data annotator
TBA

Validate tool as part of the CI/CD of the repo of a tool using the dockerized NLP Sandbox client

Not sure it's actually possible to start a dockerized service and send queries to validate it..


Repeating scrollbar dark scheme queries

Hello, why there are more than one exactly same media queries for dark scrollbar?

@umitseyhan75 please assign me this issue. I want to start working on it immediately.

can you assign this issue to me


Tampermonkey script for kolos 26 livery selector. Much easier then having to paste everytime in console

other are:)

You can find it in the releases section: https://github.com/kolos26/GEOFS-LiverySelector/releases/tag/v2.1.5


Python server deleting inactive plugin instances - Error: instance id 0 not found

Hi,
I'm testing the python the py-hello plugin but I'm experiencing a weird behaviour. I attached the plugin to a testing service pointing to mockbin.org. The first time I call the service endpoint, the plugin runs correctly:
2021/05/05 09:26:47 [info] 25#0: *8 [python:27] INFO - [17:26:47] instance #0 of py_hello started, context: ngx.timer
2021/05/05 09:26:47 [debug] 25#0: *85 [kong] mp_rpc.lua:332 [py_hello] Hello plugin Called!!!!!

if I call the service again within 60 seconds, the plugin is correctly executed:
2021/05/05 09:27:22 [debug] 25#0: *85 [kong] mp_rpc.lua:332 [py_hello] Hello plugin Called!!!!!

Waiting more than 60 seconds and calling the service again causes a new plugin instance to be created and called:
2021/05/05 09:29:29 [info] 25#0: *8 [python:27] INFO - [17:29:29] instance #1 of py_hello started, context: ngx.timer
2021/05/05 09:29:29 [debug] 26#0: *952 [kong] mp_rpc.lua:332 [py_hello] Hello plugin Called!!!!!

So far so good, everything works as expected.
After the last call to the service, if I wait more than 60 seconds to call it again the plugin is not run and the log shows an error:
2021/05/05 09:31:40 [error] 25#0: *1641 [kong] mp_rpc.lua:354 [py_hello] instance id 0 not found

As a result, the response I receive from Kong does not contain the headers that the py-hello plugin is supposeed to add.
Looking at the python server implementation I noticed that: the PluginServer class, declared in server.py file, starts a thread running the _clear_expired_plugins function to delete plugins instances that are inactive for more than 60 seconds (according to the default expire_ttl parameter which is not set when PluginServer class is instanciated).
My understanding is that those plugin instances get deleted from PluginServer cache (the instances dict declared in the init function) but this information is not sent back to kong which has is how structure to maintain active plugin instances (running_instances present in kong/runloop/plugin_servers/init.lua file). Therefore, Kong tries to call the python plugin passing an instance id that it expects to find but that might not be present in the python PluginServer because of its inactivity, generating an error and preventing plugin execution.
This is what I understood using the error message I received and looking at the code but maybe the problem could be elsewhere (e.g. Kong should tell python server to create a new instance of the plugin each time).
I'm using the docker alpine distibution of Kong 2.4.0 and theses are the lines I added to kong.conf file to configure the plugin:
plugins=bundled,py_hello

pluginserver_names = python
pluginserver_python_socket = /usr/local/kong/python_pluginserver.sock
pluginserver_python_start_cmd = /usr/bin/kong-python-pluginserver --plugins-directory /usr/local/kong/custom-plugins/
pluginserver_python_query_cmd = /usr/bin/kong-python-pluginserver --plugins-directory /usr/local/kong/custom-plugins/ --dump-all-plugins

where /usr/local/kong/custom-plugins/ is a directory I create inside the container and where I place the py-hello.py file
Could you please give it a look?
Thanks!

@valeriobelcastro I think that's a incorrect behaviour, I will draft a PR to remove it.


Implement GPU feature in Julia v1.0

The GPU option is not working in Julia v1.0 due to dependencies (i.e. OpenCL.jl and CLFFT.jl) that are still being ported to the new version of the language.
Two issues must be addressed:

Optional dependencies should now be added with Requires.jl
Consider making use of ComputationalResources.jl to trigger the GPU code

Implementation with CUDA.jl added to master. Now we need to use ComputationalResources.jl to enable manual selection of hardware instead of automatic selection.

We are currently using CUDA.functional() to switch between different implementations. The goal is to replace it with PlatformAware.jl at some point.


Non-portable printf() in examples/dump-lines.cc

The printf() statement in examples/dump-lines.cc try to print a uint64_t value using %lx.  This do not work on all architectures. I worked around the issue by changing %lx to %llx and casting the value to long long, but suspect it isn't the best solution.  What is your view on this?
The Debian patch can be found in https://anonscm.debian.org/cgit/pkg-coz/libelfin.git/tree/debian/patches/0002-portable-uint64-print.patch .

The portable way is to include inttypes.h and use "%"PRIx64 as the printing format. More info: http://en.cppreference.com/w/c/types/integer

Just to be clear, the "do not work" statement mean the build fail when the default compiler flags are used in Debian.  https://buildd.debian.org/status/fetch.php?pkg=libelfin&arch=mipsel&ver=0.2-4&stamp=1479216536 show the kind of build failure we got on some autobuilders:
dump-lines.cc: In function 'void dump_line_table(const dwarf::line_table&)':
dump-lines.cc:16:55: error: format '%lx' expects argument of type 'long unsigned int', but argument 4 has type 'dwarf::taddr {aka long long unsigned int}' [-Werror=format=]
                                line.line, line.address);
                                                       ^
cc1plus: all warnings being treated as errors

As a bonus, I just cleaned up all of the places where I was lazy and cast arguments to printf to unsigned long so they now use the inttypes.h macros.


improvement/ declare fields before constructor

Current coverage is 45.40%

Merging #13 into master will not affect coverage as of 44e4b65

@@            master    #13   diff @@
=====================================
  Files           19     19
  Stmts          359    359
  Branches        11     11
  Methods          0      0
=====================================
  Hit            163    163
  Partial          6      6
  Missed         190    190

Review entire Coverage Diff as of 44e4b65

Powered by Codecov. Updated on successful CI builds.


Tutorial custom bios

Adds a tutorial for fuzzing a custom BIOS firmware image to the documentation and examples directory.

This depends on #40 and #39 so it should be merged after they are merged.


Deprecated Extensions in OpenShift 4.x

OpenShift is reporting the following alert in versions 4.x due to multiple resources using v1beta1 API extensions.
"A client in the cluster is using deprecated extensions/v1beta1 API that will be removed soon."

Hi there, thank you for letting us know.
May I ask which version of the Operator are you using? We removed our usages of the extensions/v1beta1 API around the Operator v0.4.0 I believe, unless we missed something.
Best regards,
Luis

We are using Argo CD to manage the deployment of the OneAgent Operator. We are using the yaml files in the deploy/openshift directory in this project to create the operator. The majority of the resources in those files are still using betav1 extensions. We are currently using the registry.connect.redhat.com/dynatrace/dynatrace-oneagent-operator:v0.5.4 image, but I see that the current version of the deployment file is pointing to snapshot, should we be pointing to a snapshot version?

Thanks. Don't think we have any "extensions/v1beta1" references anymore (i.e., PodSecurityPolicies are the only objects under this API we ever used IIRC) but we do have some other v1beta* objects. Regardless, will spin up a OCP4 environment and take a look.
As for the deployment manifests, please don't, that image is for our development builds. We've recently started to publish our release manifests as attachments for releases: https://github.com/Dynatrace/dynatrace-oneagent-operator/releases

Thanks for pointing me to the releases, we'll switch to using OpenShift.yaml. I took a quick look at that file and see a few betav1 references in there as well, not sure if those particular extensions are being deprecated yet, but I will deploy and test.
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding

Hello, apologies for the late reply. I've confirmed that we don't have any warnings for our latest manifests on OCP 4.3.
And as for the RBAC objects, indeed they are also deprecated although Kubernetes/OCP seem to handle them without any issues. Will try to update them regardless.
Hopefully this should be all, and thanks for bringing this up to our attention. If you have any issues with the current manifests please let us know.
Luis


Allow renaming of uploaded artifacts

I'm uploading artifacts that are nested a few layers deep in my maven project, and it would be much easier to deal with them in subsequent steps if I could rename them (essentially remove the directories) as I upload them. I see the buildkite agent supports this, but can't see any way to do it using the plugin. Can this please be added?

That's a great suggestion @keithbranton. Any idea on how you'd like the config syntax to be, based on your maven project?

Not sure - the yaml string for the upload could include some sort of delimiter and a destination. If nobody ever used spaces in filenames I'd vote for that, but maybe something like -> would be safer.
  plugins:
    ecr#v1.1.4:
      login: true
      account_ids: "880807510442"
      no-include-email: true
      region: us-east-2
<EMAIL_ADDRESS>      job: release
    artifacts#v1.2.0:
      upload:
      - rmq-shovel/target/rmq-shovel.jar -> rmq-shovel.jar
      - rmq-shovel/target/classes/build.properties -> build.properties
      - rmq-shovel/target/classes/config.yml -> config.yml
      - outbound-to-rabbitmq/target/outbound-to-rabbitmq.jar -> outbound-to-rabbitmq.jar

or perhaps it would be better to support objects - in this case it would be helpful to be able to support simple strings as well for what is possible the common case (and backward compatibility)
      upload:
      - src: rmq-shovel/target/rmq-shovel.jar
        target: rmq-shovel.jar
      - src: rmq-shovel/target/classes/build.properties
        target: build.properties
      - rmq-shovel/target/classes/config.yml

another possibility would be to include another optional section
      upload:
      - rmq-shovel/target/rmq-shovel.jar
      - rmq-shovel/target/classes/build.properties
      destination:
      - rmq-shovel.jar
      - build.properties

Looking at all these, I'd favor the second option - it seems like the most intentional - but maybe this will spark a better idea?

Thanks @keithbranton!
Just to be clear for future travellers, with the buildkite-agent upload command we have this example in our docs, if you want to strip the directory:
cd log
buildkite-agent artifact upload test.log

That's not possible with this plugin (yet).
I like your second suggestion too @keithbranton.
I wonder how the  would work with glob patterns also?
steps:
  - command: make
    plugins:
      artifacts#v1.2.0:
        upload:
          - src: rmq-shovel/target/classes/build.properties
            target: build.properties
          - src: **/*.jar
            target: .
  - wait
  - command: script/test-jar.sh
    plugins:
      artifacts#v1.2.0:
        download:
          - src: app.jar
            target: .

🤔
Related there's also buildkite/agent#752 and the super long-running buildkite/agent#91 and buildkite/agent#186, where @keithpitt has thrown together some other options.
But this plugin is a good place to experiment with a new syntax.

@toolmantim
These are both cool ideas, if they are easy to implement, and I'd probably use the upload globs if it was available. If you are aware of others who would like the uploading/renaming with globs, and being able to relocate downloads - then great, but I will think you are just as awesome if I can just get relocating individual upload files :)
if buildkite/agent#752 was implemented and there was a way to enable that option in the upload plugin - that would be a fine alternative for my use cases.

Closing this because it was fixed on PR #27


Policy revision reported incorrectly

When importing a policy, the policy is applied to the endpoint, correctly shown in the endpoint structure, the redirect is created (proxy stats show up) but the redirect in the datapath is not being put in place or not effective. The policy revision is still bumped which causes the policy to not be enforced even though it is reported to be enforcing.
Full log attached, a couple of suspicious errors:
level=error msg="error while receiving request from xDS stream" error="rpc error: code = 1 desc = context canceled" xdsStreamID=39
level=error msg="xDS stream context canceled" error="context canceled" xdsStreamID=37
level=error msg="error while receiving request from xDS stream" error="rpc error: code = 1 desc = context canceled" xdsStreamID=37
level=error msg="xDS stream context canceled" error="context canceled" xdsStreamID=38
level=error msg="error while receiving request from xDS stream" error="rpc error: code = 1 desc = context canceled" xdsStreamID=38
level=error msg="xDS stream context canceled" error="context canceled" xdsStreamID=39
level=warning msg="Envoy: Proxy crashed" error="signal: segmentation fault (core dumped)"

The following could be due to deletion of a k8s namespace
level=warning msg="Generating BPF for endpoint failed, keeping stale directory." containerID=4f39f8d734 endpointID=46367 file-path=/var/run/cilium/state/46367_next_fail policyRevision=192
level=warning msg="Regeneration of endpoint program failed" containerID=4f39f8d734 endpointID=46367 error="Skipping build due to invalid state: disconnected" policyRevision=192

cilium-no-l7-redirect.log

We need to understand why Envoy is crashing sometimes. We are now logging the Envoy logs into the Cilium debug logs. When you see that happen, please collect the debug logs.
But Envoy is always restarted immediately by Cilium when it crashed, and usually such a crash has no effect on policy regeneration. Envoy restarts, picks up the latest configuration, applies it, and ACKs it.
Same with the "xDS stream context canceled" error messages. They just indicate that Envoy has closed its xDS connection to Cilium. That obviously happened in this case because it crashed.

bad.log

bad2.log
monitor.log


fix: inline styled-components keyframes to css keyframes

@spawnia is it necessary to test this new version in the limes app? If yes, could you please do so with the alpha release available?


Don't check for [[Prototype]] equality

IMO Object.create(null) and {} are conceptually equal. i.e. both are empty objects. I've run into this issue a few times this week.
The first was when I was upgrading then-redis to use the latest redis, which recently switched to using Object.create(null) as the prototype for objects returned from HGETALL operations. Some of my tests were doing expect(result).toEqual({ some: 'stuff' }).
The second case was earlier today when I was trying to upgrade expect to use the latest query-string (see https://github.com/mjackson/history/pull/268). Like redis, they recently switched to returning an Object.create(null) from a parse of an empty query string.
Anyway, one of the main reasons I was attracted to using is-equal to power equality assertions in expect was because the project description says this library checks for conceptual equality, not exact. Conceptually, both Object.create(null) and {} are both empty objects. And in both of the practical instances I've mentioned above that's how Object.create(null) is being used.

I guess the other way I could do this would be to iterate over objects in toEqual and compare their values. Similar to how we discussed handling Set and Map in https://github.com/mjackson/expect/issues/93.

imo a normal object and a null object aren't conceptually equal tho - one is an empty object, one is an empty dict.

@ljharb That's fine. It's easy enough for me to get the kind of behavior that I want in expect by just wrapping is-equal to get the kind of loose equality I'm looking for.

Sounds good.
Another way to determine equality would be to compare new Map(Object.entries(x)) for both x, once you're comparing Maps in an unordered fashion.


EV3: The Calibrate Block is Not Considered As a Blocking Block

Describe the bug

For the EV3, the 'calibrate HT compass sensor' block is not considered as a blocking block even though it takes 40 seconds to complete.

To Reproduce
Steps to reproduce the behavior:

Switch to EV3.
Spawn the 'calibrate HT compass sensor' block.
See that it is not considered as a blocking block.

Expected behavior

A warning showing that it is a blocking block.

GIF

Device information

Type: Laptop
OS: Windows 10
Browser: Google Chrome

I would like to work on this. Is this a purely frontend bug?

Sure, block related changes need to be done over in our blockly repo and then built as described and copied over to the static resources of the server.
For first quick tests you can use your browser developer tools to edit the files and immediately see the changes.

I've figured out code changes. Where can I find the build instructions for the blockly repo?

It was previously documented in the upstream repo or the link, but as that changed a lot we should add it to our repo.
You can just run python build.py in the blockly directory while having the closure library on the same level as the blockly directory.

I tried that but build.py in running into an exception.

You should be able to try again, sometimes the build server is unreliable.

Should I directly make the required change in staticResources/blockly instead of building and copying, since doing so seems to be working locally?

Yes, it worked. Thank you.

Fix merged in https://github.com/OpenRoberta/blockly/commit/a7b5caffd53f9879e88d2f10f4dc630b6ebf592a. Is resolved once blocky is rebuilt.


docs: 📝  added content to readme

I have two questions;

Do we need more emoji? (yes or yes)
Who knows if we need those both flatpickr / react-flatpickr or if the latter is enough?

[ ] Add section about Icons


Remove CairoSerialize impl for String and &str. Use ByteArray explicitly

Which components does the task require to be changed? (think hard pls)
snforge
Description
Cairo does not support strings. We need to establish a one-to-one mapping and remove the CairoSerialize implementation for String and &str.
Proposed steps:

remove this implementation here
replace every occurrence in structs/enums implenting CairoSerialize  with ByteArray instead of String or &str.

Tip:

Conversion from String to ByteArray can be done as follows:

    let string: String = String::from("Hello world!");
    let byte_array: ByteArray = ByteArray::from(string.as_str());

Conversion from &str to ByteArray can be done as follows:

    let str: &str = "Hello world!";
    let byte_array: ByteArray = ByteArray::from(str);

Compiler can help you find places, where structs/enums implenting CairoSerialize use String or &str. After doing step 1. Run cargo lint and find places where errors occure

I am applying to this issue via OnlyDust platform.
My background and how it can be leveraged

Hey! I'm Gerson. 👋
Member of Dojo Coding. I have contributed before to other Rust projects like

kakarot-rpc
ef-tests
Madara
Starknet phone

Can I work on this issue?
Check my OnlyDust profile, I've contributed to many projects: Profile

How I plan on tackling this issue
Plan to Solve the Issue:

Study the codebase to understand the current structure and functionality.
Understand the problem that needs to be solved by thoroughly reading the issue details.
Leverage all provided resources within the issue to get a better grasp of the task at hand.
If I encounter roadblocks, I will ask questions ASAP in the Telegram group to move forward effectively.
I will open a PR and wait for ur feedback

I am applying to this issue via OnlyDust platform.
My background and how it can be leveraged
Hi, I am ICT student and for the last few months i was really intrested in Rust and Cario. I also work professionallyas software developer. I hope can help ypu solve this and start my journey in Cario world
How I plan on tackling this issue
First, I'd remove the CairoSerialize implementation for String and &str, then search the codebase for any structs or enums using these types and replace them with ByteArray. I'd use the provided methods to convert String and &str into ByteArray where needed. Running cargo lint would help me find any places I missed. After making the changes, I'd run tests to make sure everything works correctly, keeping an eye out for any issues that might pop up from the conversions.

I am applying to this issue via OnlyDust platform.
My background and how it can be leveraged
Rust experience and a regular Foundry user for Starknet smart contract testing. Regular "Cairo Book" and "Starknet By Examle" contributor. Creator and maintainer of Cairo learning track on Exercism. As a regular Foundry user, I'm very interested in making it as efficient and as clean as possible.
How I plan on tackling this issue

Update code as described in the issue description, it's clear enough.
Rerun all tests to make sure nothing broke.
Fix any broken tests.

I am applying to this issue via OnlyDust platform.
My background and how it can be leveraged
Good day, My name is Deon, I am a passionate Full-Stack Web3 developer. I have sufficient experience building web3 applications/smart contracts with rust as well as high-performance backends.
I have a solid understanding of both backend development and data management. I’m committed to writing clean, maintainable code and actively contributing to open-source projects.
How I plan on tackling this issue

Remove the Implementation: Firstly, I'll locate and delete the CairoSerialize implementation for String and &str from the codebase.

Update Structs/Enums: I'll then use the Rust compiler to identify all structs and enums that currently implement CairoSerialize with String or &str. I will then replace these types with ByteArray.

Implement Conversion: I'll ensure proper conversion from String and &str to ByteArray as specified, using the provided methods for both types.

Run Linting and Testing: I'll execute cargo lint to identify any remaining issues, making necessary adjustments to ensure all errors are resolved.

By following these steps, I will enhance the compatibility of our code with Cairo and improve overall data handling in the project.

Hi all! 🌟
Thank you so much for all the submissions! I'm curating this issue and would like to clarify that when I added this to the hackathon, I envisioned it more as an entry-level task. I want to provide a chance for new developers, so I'll be assigning this to @Nemezjusz.
Sorry, @Gerson2102! 😅 I know you were first, but I think this task might be too easy for you.
Thanks for understanding! 😊

@ksew1 got it mate, dont worry. Let me know if there is any issue for me please. Thanks!

@Nemezjusz any progress on this?

I couldn't find the time yesterday, but I'm on it now :)


fix warning, changes in Swift 3.1

fix warning, 'M_PI is deprecated' in Swift 3.1
fix warning, '@noescape is the default and deprecated' in Swift 3.1

Fix closure reference error and add new pull request. #7


KeyError: 'start_time' logger error 'start_time'

Can be reproduced with the colab
Not sure how that can happen given current code but there must be some edge case

I'm having this issue too across multiple environments

https://github.com/rom1504/clip-retrieval/blob/main/clip_retrieval/clip_inference/logger.py#L41 can happen if this branch runs before any call to the writer logger is done
Action item:

Add a check there to make sure stats is not an empty dict before saving
figure out in what case the write logger could get no call

https://github.com/rom1504/clip-retrieval/commit/c978a03ec54819a604c2d06b62db1f27ea71a217 probably caused by this commit

https://github.com/rom1504/clip-retrieval/blob/main/clip_retrieval/clip_inference/runner.py#L34 could happen if there is no shard to process for this runner

So probably we are generating empty partitions (cc @nousr )

Need to print calculate_partition_count output to try to figure out what's up
Could also add more checks in the code to get better errors

So probably we are generating empty partitions (cc @nousr )

anecdotal evidence:
i've checked and all the ones we generated for 2b-en were all ~1.89Gb.
is it possible that an empty partition could >0Gb?

No, i think in your case (laion2b on slurm) it's all working as expected,
but it seems in the small scale case local distributor there is something
off
Need more testing/printing to know what exactly,  the example notebook
reproduces the problem
On Tue, Nov 1, 2022, 00:44 zion @.***> wrote:

So probably we are generating empty partitions (cc @nousr
https://github.com/nousr )
anecdotal evidence:
i've checked and all the ones we generated for 2b-en were all ~1.89Gb.
is it possible that an empty partition could >0Gb?
—
Reply to this email directly, view it on GitHub
https://github.com/rom1504/clip-retrieval/issues/199#issuecomment-1297821396,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAR437WXT3RQRAYWNP7G4Y3WGBKVTANCNFSM6AAAAAARSGJEHQ
.
You are receiving this because you authored the thread.Message ID:
@.***>

Hello! I also encounter the error. I wondered if CPU/GPU type of env. could be a reason, but it's the same.  https://github.com/rom1504/clip-retrieval/blob/main/notebook/clip-client-query-api.ipynb
The number of samples has been estimated to be 3
Starting the worker
dataset is 12
Starting work on task 0
100%|████████████████████████████████████████| 354M/354M [00:01<00:00, 190MiB/s]
warming up with batch size 256 on cuda
/usr/local/lib/python3.7/dist-packages/clip_retrieval/load_clip.py:90: UserWarning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values.
To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). (Triggered internally at  ../aten/src/ATen/native/BinaryOps.cpp:594.)
  model.encode_image(image_tensor)
done warming up in 11.390223741531372s
/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py:566: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
  cpuset_checked))
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/clip_retrieval/clip_inference/logger.py", line 137, in reader
    start_time_no_initial_load = min(start_time_no_initial_load, v["start_time"])
KeyError: 'start_time'
logger error 'start_time

Yeah this is a known bug. Will be fixed soon.
You can use the previous version in the mean time.
On Wed, Nov 2, 2022, 19:19 Todor Arnaudov @.***> wrote:

Hello! I also encounter the error. I wondered if CPU/GPU type of env.
could be a reason, but it's the same.
https://github.com/rom1504/clip-retrieval/blob/main/notebook/clip-client-query-api.ipynb
The number of samples has been estimated to be 3
Starting the worker
dataset is 12
Starting work on task 0
100%|████████████████████████████████████████| 354M/354M [00:01<00:00, 190MiB/s]
warming up with batch size 256 on cuda
/usr/local/lib/python3.7/dist-packages/clip_retrieval/load_clip.py:90: UserWarning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values.
To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). (Triggered internally at  ../aten/src/ATen/native/BinaryOps.cpp:594.)
model.encode_image(image_tensor)
done warming up in 11.390223741531372s
/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py:566: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
cpuset_checked))
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/clip_retrieval/clip_inference/logger.py", line 137, in reader
start_time_no_initial_load = min(start_time_no_initial_load, v["start_time"])

KeyError: 'start_time'
logger error 'start_time
—
Reply to this email directly, view it on GitHub
https://github.com/rom1504/clip-retrieval/issues/199#issuecomment-1301042453,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAR437SHTIC6JTKUCPRZCNDWGKWERANCNFSM6AAAAAARSGJEHQ
.
You are receiving this because you authored the thread.Message ID:
@.***>

Fix released as 2.35.1

still happens to me :(

the bug was already fixed
you are probably encountering another issue causing this
we could improve the error though yes


Set a limit of characters on a message

When the message is too long with full of emojis, the application is crashing

#397 added a character limit on messages of 2000


Keda scaleType Job scales Jobs twice

When using Keda to scale a Job type, KEDA seems to scale the Job twice on first scale.  This happens when creating a ScaledObject when there is already a message existing in the queue.
Succeeding scaling works as expected.  It only happens on first time scaling.
Expected Behavior
Keda should not scale Jobs twice on first time scaling.
Actual Behavior
Keda scales the Jobs twice.
Steps to Reproduce the Problem

Put one message into a Queue.
Create a ScaledObject of scaleType = job
Watch Keda creates 2 jobs almost at the sametime.

Specifications

Version: from master branch
Platform: amd64,Ubuntu 18.04
Scaler(s): RabbitMQ scaler with scaleType: job

Sample ScaledObject
apiVersion: v1
kind: Secret
metadata:
  name: rabbitmq-consumer
data:
  RabbitMqHost: YW1xcDovL3VzZXI6UEFTU1dPUkRAMTAuMTUyLjE4My4xMDM6NTY3Mg==
---
apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
  name: rabbit-queue-scaledobject-jobs
  namespace: default
spec:
  scaleType: job
  pollingInterval: 10
  maxReplicaCount: 0
  jobTargetRef:
    activeDeadlineSeconds: 600
    ttlSecondsAfterFinished: 1
    backoffLimit: 6
    template:
      spec:
        containers:
        - name: pi
          image: perl
          command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(3000);"]
          env:
          - name: RabbitMqHost
            valueFrom:
              secretKeyRef:
                name: rabbitmq-consumer
                key: RabbitMqHost
  triggers:
  - type: rabbitmq
    metadata:
      queueName: hello
      host: RabbitMqHost
      queueLength  : '1'

I think the issue is at this line
https://github.com/kedacore/keda/blob/61d7959ec98f6d91e7d7232c3a0cc8d63a460f73/pkg/handler/scale_loop.go#L14
Would it be better to let the pollingInterval to pass before the scaling happens?
I tried removing the line above, and it seems to prevent it from doubling the scale.  But it waits for the pollingInterval to pass before it starts scaling up or down.

@balchua good catch, thanks.
We shouldn't remove the first h.handleScale() call, KEDA should be able to scale imidiately and not wait for the pollingInterval for the first scale and this change would effect deploymentType as well.
This could be better place where to fix this issue (since it is jobType related problem): https://github.com/kedacore/keda/blob/10c92ff1348b20df14921b156f611b0243924f89/pkg/handler/scale_loop.go#L50

@zroubalik i thought i will be able to prevent this when i prevent the number of Jobs to scale beyond the maxScale in the source scale_jobs.go.
But seems to me that it is happening almost instantaneously.  Is this even possible that it is triggered 2x at almost the same time?

@balchua I saw your PR for preventing jobs from exceeding maxScale earlier and combining your solution with preventing more jobs to be created than there are messages seems to fix this scaling issue for me.
This might not be the cause of the problem though...

It seems I spoke too soon! Might have been some funny race condition but when I tried it again, the scaling triggered twice.
I did fetch the logs this time, using the keda:master image:
{"level":"info","ts":1578071613.7561276,"logger":"cmd","msg":"Go Version: go1.13.3"}
{"level":"info","ts":1578071613.7562637,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1578071613.7563248,"logger":"cmd","msg":"Version of operator-sdk: v0.11.0"}
{"level":"info","ts":1578071613.7563548,"logger":"cmd","msg":"Keda Commit: 61d7959ec98f6d91e7d7232c3a0cc8d63a460f73"}
{"level":"info","ts":1578071613.756532,"logger":"leader","msg":"Trying to become the leader."}
{"level":"info","ts":1578071616.3683953,"logger":"leader","msg":"No pre-existing lock was found."}
{"level":"info","ts":1578071616.372165,"logger":"leader","msg":"Became the leader."}
{"level":"info","ts":1578071618.9748654,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":"<IP_ADDRESS>:8383"}
{"level":"info","ts":1578071618.975012,"logger":"cmd","msg":"Registering Components."}
{"level":"info","ts":1578071618.9752345,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"scaledobject-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1578071621.5783374,"logger":"cmd","msg":"Could not generate and serve custom resource metrics","error":"discovering resource information failed for ScaledObject in keda.k8s.io/v1alpha1: unable to retrieve the complete list of server APIs: external.metrics.k8s.io/v1beta1: the server is currently unable to handle the request"}
{"level":"info","ts":1578071624.2016356,"logger":"metrics","msg":"Metrics Service object created","Service.Name":"keda-operator-metrics","Service.Namespace":"keda"}
{"level":"info","ts":1578071626.8036015,"logger":"cmd","msg":"Could not create ServiceMonitor object","error":"no ServiceMonitor registered with the API"}
{"level":"info","ts":1578071626.803761,"logger":"cmd","msg":"Install prometheus-operator in your cluster to create ServiceMonitor objects","error":"no ServiceMonitor registered with the API"}
{"level":"info","ts":1578071626.8037968,"logger":"cmd","msg":"Starting the Cmd."}
{"level":"info","ts":1578071626.8040416,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1578071626.9041996,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"scaledobject-controller"}
{"level":"info","ts":1578071627.0045247,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"scaledobject-controller","worker count":1}
{"level":"info","ts":1578071627.004942,"logger":"controller_scaledobject","msg":"Reconciling ScaledObject","Request.Namespace":"keda-sample","Request.Name":"consumer-scaler"}
{"level":"info","ts":1578071627.004993,"logger":"controller_scaledobject","msg":"Adding Finalizer for the ScaledObject","Request.Namespace":"keda-sample","Request.Name":"consumer-scaler"}
{"level":"info","ts":1578071627.015848,"logger":"controller_scaledobject","msg":"Detecting ScaleType from ScaledObject","Request.Namespace":"keda-sample","Request.Name":"consumer-scaler"}
{"level":"info","ts":1578071627.015923,"logger":"controller_scaledobject","msg":"Detected ScaleType = Job","Request.Namespace":"keda-sample","Request.Name":"consumer-scaler"}
{"level":"info","ts":1578071627.1162179,"logger":"controller_scaledobject","msg":"Deleting jobs owned by the previous version of the ScaledObject","Request.Namespace":"keda-sample","Request.Name":"consumer-scaler","Number of jobs to delete":8}
{"level":"info","ts":1578071627.1163352,"logger":"controller_scaledobject","msg":"Reconciling ScaledObject","Request.Namespace":"keda-sample","Request.Name":"consumer-scaler"}
{"level":"info","ts":1578071627.1163745,"logger":"controller_scaledobject","msg":"Detecting ScaleType from ScaledObject","Request.Namespace":"keda-sample","Request.Name":"consumer-scaler"}
{"level":"info","ts":1578071627.11638,"logger":"controller_scaledobject","msg":"Detected ScaleType = Job","Request.Namespace":"keda-sample","Request.Name":"consumer-scaler"}
{"level":"info","ts":1578071627.116399,"logger":"controller_scaledobject","msg":"Deleting jobs owned by the previous version of the ScaledObject","Request.Namespace":"keda-sample","Request.Name":"consumer-scaler","Number of jobs to delete":8}
{"level":"info","ts":1578071627.1260946,"logger":"scalehandler","msg":"Scalers count","ScaledObject.Namespace":"keda-sample","ScaledObject.Name":"consumer-scaler","ScaledObject.ScaleType":"job","Count":1}
{"level":"info","ts":1578071627.1261015,"logger":"scalehandler","msg":"Scalers count","ScaledObject.Namespace":"keda-sample","ScaledObject.Name":"consumer-scaler","ScaledObject.ScaleType":"job","Count":1}
{"level":"info","ts":1578071628.2392795,"logger":"scalehandler","msg":"Active trigger","ScaledObject.Namespace":"keda-sample","ScaledObject.Name":"consumer-scaler","ScaledObject.ScaleType":"job","Scaler":{},"isTriggerActive":true}
{"level":"info","ts":1578071628.2394001,"logger":"scalehandler","msg":"Scaler max value","ScaledObject.Namespace":"keda-sample","ScaledObject.Name":"consumer-scaler","ScaledObject.ScaleType":"job","Scaler":{},"MaxValue":5}
{"level":"info","ts":1578071628.2424963,"logger":"scalehandler","msg":"Active trigger","ScaledObject.Namespace":"keda-sample","ScaledObject.Name":"consumer-scaler","ScaledObject.ScaleType":"job","Scaler":{},"isTriggerActive":true}
{"level":"info","ts":1578071628.2425148,"logger":"scalehandler","msg":"Scaler max value","ScaledObject.Namespace":"keda-sample","ScaledObject.Name":"consumer-scaler","ScaledObject.ScaleType":"job","Scaler":{},"MaxValue":5}
{"level":"info","ts":1578071628.247716,"logger":"scalehandler","msg":"QueueLength Metric value","ScaledObject.Namespace":"keda-sample","ScaledObject.Name":"consumer-scaler","ScaledObject.ScaleType":"job","Scaler":{},"queueLength":1}
{"level":"info","ts":1578071628.2478073,"logger":"scalehandler","msg":"Scaler is active","ScaledObject.Namespace":"keda-sample","ScaledObject.Name":"consumer-scaler","ScaledObject.ScaleType":"job","Scaler":{}}
{"level":"info","ts":1578071628.2492247,"logger":"scalehandler","msg":"QueueLength Metric value","ScaledObject.Namespace":"keda-sample","ScaledObject.Name":"consumer-scaler","ScaledObject.ScaleType":"job","Scaler":{},"queueLength":1}
{"level":"info","ts":1578071628.2492595,"logger":"scalehandler","msg":"Scaler is active","ScaledObject.Namespace":"keda-sample","ScaledObject.Name":"consumer-scaler","ScaledObject.ScaleType":"job","Scaler":{}}
{"level":"info","ts":1578071628.2533662,"logger":"scalehandler","msg":"Creating jobs","ScaledObject.Namespace":"keda-sample","ScaledObject.Name":"consumer-scaler","ScaledObject.ScaleType":"job","Number of jobs":1}
{"level":"error","ts":1578071628.2555327,"logger":"scalehandler","msg":"Error updating scaledObject status","ScaledObject.Namespace":"keda-sample","ScaledObject.Name":"consumer-scaler","ScaledObject.ScaleType":"job","error":"Operation cannot be fulfilled on scaledobjects.keda.k8s.io \"consumer-scaler\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/kedacore/keda/pkg/handler.(*ScaleHandler).updateScaledObjectStatus\n\tkeda/pkg/handler/scale_handler.go:47\ngithub.com/kedacore/keda/pkg/handler.(*ScaleHandler).scaleJobs\n\tkeda/pkg/handler/scale_jobs.go:22\ngithub.com/kedacore/keda/pkg/handler.(*ScaleHandler).handleScaleJob\n\tkeda/pkg/handler/scale_loop.go:99\ngithub.com/kedacore/keda/pkg/handler.(*ScaleHandler).handleScale\n\tkeda/pkg/handler/scale_loop.go:42\ngithub.com/kedacore/keda/pkg/handler.(*ScaleHandler).HandleScaleLoop\n\tkeda/pkg/handler/scale_loop.go:14"}
{"level":"info","ts":1578071628.2556405,"logger":"scalehandler","msg":"Creating jobs","ScaledObject.Namespace":"keda-sample","ScaledObject.Name":"consumer-scaler","ScaledObject.ScaleType":"job","Number of jobs":1}
{"level":"info","ts":1578071628.2740905,"logger":"scalehandler","msg":"Created jobs","ScaledObject.Namespace":"keda-sample","ScaledObject.Name":"consumer-scaler","ScaledObject.ScaleType":"job","Number of jobs":1}
{"level":"info","ts":1578071628.2803454,"logger":"scalehandler","msg":"Created jobs","ScaledObject.Namespace":"keda-sample","ScaledObject.Name":"consumer-scaler","ScaledObject.ScaleType":"job","Number of jobs":1}

The same problem also happens with the keda:1.0.0 image.
I added some logging and noticed that startScaleLoop is called twice from the reconcileJobType method, I'm not sure if this might be part of the problem?

I also saw an error Operation cannot be fulfilled.....  don't know if this is benign.

@balchua @lee0c may be able to help. I tried to add @stgricci but somehow cant seem to.

@anirudhgarg I can take a look. Not sure why it's not assignable to me.

I tested it on v2 logic.  Using the new scaling logic, It doesn't create two jobs against one message. I tested your code with v2 configuration.  However, this yaml has an issue. It  doesn't any message. Just execute some perl then finished. That means, once polling interval has finished, it is going to execute one more job. (since it doesn't consume the queue).  A job should consume the queue (at least one message).
apiVersion: v1
kind: Secret
metadata:
  name: rabbitmq-consumer
data:
  RabbitMqHost: YW1xcDovL3VzZXI6UEFTU1dPUkRAMTAuMTUyLjE4My4xMDM6NTY3Mg==
  LocalHost: YW1xcDovL3VzZXI6UEFTU1dPUkRAMTI3LjAuMC4xOjU2NzI=
---
apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
  name: rabbit-queue-scaledobject-jobs
  namespace: default
spec:
  pollingInterval: 10
  jobTargetRef:
    activeDeadlineSeconds: 600
    ttlSecondsAfterFinished: 1
    backoffLimit: 6
    template:
      spec:
        containers:
        - name: pi
          image: perl
          command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(3000);"]
          envFrom:
            - secretRef:
                name: rabbitmq-consumer
  triggers:
  - type: rabbitmq
    metadata:
      queueName: hello
      host: RabbitMqHost
      localhost: LocalHost
      queueLength  : '1'

New Scale Logic you can find on this PR.
https://github.com/kedacore/keda/pull/951
This PR is enable us to test rabbit mq easily.
https://github.com/kedacore/keda/pull/953
I'l also provide samples for updating the rabbit mq sample with scaled job. If you want to test the behavior.
https://github.com/kedacore/sample-go-rabbitmq/pull/15

@TsuyoshiUshio so can we close this one then?


Show Python environments from the VS Code server

Version: 1.75.0-insider (user setup)
Commit: 1379f0375440df946adc62f7afb8375dd45d5adf
Date: 2023-01-17T05:21:07.217Z
Electron: 19.1.9
Chromium: 102.0.5005.194
Node.js: 16.14.2
V8: <IP_ADDRESS>-electron.0
OS: Windows_NT x64 10.0.22621
Sandboxed: Yes
When connected to a VS Code server, I expect to see all the Python environments from the tunneled server (like remote SSH-ing). Instead, I see:

Steps to repro:

Start a code tunnel (code-insiders tunnel)
Open the URL in the browser and open an ipynb file
Click on the kernel picker. I would expect to be able to see and select my kernels on the the code server I'm connected. Since I didn't, my workaround was to start up a new Jupyter server from the integrated terminal (i.e., jupyter notebook --no-browser in the terminal and then copy the Jupyter server URL by going to the kernel picker -> Existing Jupyter Server).

cc\ @rebornix

Please provide the logs as mentioned in the issue template.

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

XXX

@minsa110 can you double check the version of the Python extension that you're using? I was able to reproduce this but the problem is that for some reason VS Code Server is installing the stable version of the Python extension (v2022.20.2) which is not compatible with VS Code insiders. When I installed the pre-release version, it worked. I don't know why VS Code server would be installing the stable release though, I thought by default it would pick up the pre-release one?

@luabud you're right--this does work when I switch over from release (v2023.2.0) to pre-release (v2023.3.10371010)!
@karrtikr sorry about that. Here is the log, though this is no longer an issue since installing the pre-release works for me.

```
Experiment 'pythonaa' is active
LSP Notebooks experiment is enabled
LSP Notebooks interactive window support is enabled
> conda info --json
> ~\AppData\Local\Microsoft\WindowsApps\python3.10.exe -I ~\.vscode-server-insiders\extensions\ms-python.python-2023.2.0\pythonFiles\get_output_via_markers.py ~\.vscode-server-insiders\extensions\ms-python.python-2023.2.0\pythonFiles\interpreterInfo.py
> "~\OneDrive - Microsoft\Documents\Test Project 3\.venv\Scripts\python.exe" -I ~\.vscode-server-insiders\extensions\ms-python.python-2023.2.0\pythonFiles\get_output_via_markers.py ~\.vscode-server-insiders\extensions\ms-python.python-2023.2.0\pythonFiles\interpreterInfo.py
> "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64__qbz5n2kfra8p0\python3.10.exe" -I ~\.vscode-server-insiders\extensions\ms-python.python-2023.2.0\pythonFiles\get_output_via_markers.py ~\.vscode-server-insiders\extensions\ms-python.python-2023.2.0\pythonFiles\interpreterInfo.py
Python interpreter path: .\.venv\Scripts\python.exe
> "~/OneDrive - Microsoft/Documents/Test Project 3/.venv/Scripts/activate.bat" && echo 'e8b39361-0157-4923-80e1-22d70d46dee6' && python ~/.vscode-server-insiders/extensions/ms-python.python-2023.2.0/pythonFiles/printEnvVariables.py
Starting Pylance language server.
Jupyter provided interpreter path override: c:\Users\somin\OneDrive - Microsoft\Documents\Test Project 3\.venv\Scripts\python.exe
```


avoid ChainNotify & update to newer interopnet

Fixes #359
Set unique tmp dir per test package since now they run in parallel and that was causing some problems.

Also, a new version of interopnet was released again:

Fixes #323: Good news, I don't have to maintain textileio/lotus-client anymore; archived it.
Update to new bootstrap peers from new interopnet network.
Switched to new lotus-devnet from new interopnet version. Involved doing: https://github.com/textileio/lotus-devnet/pull/11

Will merge to unblock Marcus. Still open to reviews.