No secondary seed and essence drops for customized crops.

Describe the bug
I defined a new crop Insanium seeds whose recipes and drops are both Insanium essence in Mystical Agradditions.
When harvesting, the crop always drops one Insanium essence and one Insanium seeds only. No extra essences and seeds.
To Reproduce
Steps to reproduce the behavior:
Create a new file called insanium.json and move it to the folder mysticalcustomization/crops.
File contents are as follows:
{
    "name": "Insanium",
    "type": "resource",
    "tier": "mysticalagradditions:6",
    "ingredient": {
        "item": "mysticalagradditions:insanium_essence"
    },
    "color": "7400b8",
    "textures": {
        "flower": "mysticalagriculture:block/flower_dust",
        "essence": "mysticalagriculture:block/essence_dust"
    },
    "essence": "mysticalagradditions:insanium_essence",
    "baseSecondaryChance": 0.1,
    "respectsEffectiveFarmland": true
}

Open a world in Minecraft.
Get the seed mysticalagriculture:insanium_seeds (crop id: mysticalcustomization:insanium) and plant it on the Insanium farmland (from Mystical Agradditions mod).
Harvest it by left-clicking the crop.
Expected behavior
Each harvest should have a 20% chance of extra drops on Insanium farmland.
Screenshots / Scripts / Logs
Please add your logs and scripts (if applicable).
Versions (please complete the following information):

Minecraft: 1.19.2
Forge: 43.3.13
Cucumber: 6.0.10
Mystical Agriculture: 6.0.14
Mystical Agradditions: 6.0.4
Mystical Customization: 4.0.1
Singleplayer or Multiplayer: Singleplayer

So the mysticalagradditions:6 tier explicitly disables secondary seed drops. If you edit it and set secondarySeedDrop to true it should work.

It works. Thank you :)


Bug Report: cannot install Node-RED nodes

Expected Behavior
It should be possible to install additional Node-RED nodes.
Current Behavior
The installation takes forever.
Relevant log output
No response
Steps to Reproduce

Install the stack
Try to install additional Node-RED nodes via the UI

Context (Environment)
Various reported customer issues
Possible Solution
Remove the init containers, which install various nodes. Because there are so many nodes installed by default it causes the installation to take forever (caused by npm). even the manual installation using npm takes forever (npm install xxx).
Version
v0.9.x
Anything else?
No response

Should we take out the default flows too ?
Cause they won't install correctly without those plugins ?


Declare InQuery Variables

Add the ability to declare variables for the query especially when the variable is used more than once.

Easier for debugging -> informative variable @date instead of  @p1, @p2, ....
Enhance query abstraction, inner queries can assume that some variable are declared without the need to pass variable in C# code.

db.Query("Balances")
.WithVar("date", DateTime.UtcNow)
.WhereRaw("[ValueDate] > @date")
.WhereRaw("[ValueDate] < @date")

Simple implementation：
public partial class Query : BaseQuery<Query>
{
    public Dictionary<string, object> VarMap { get; set; } = new Dictionary<string, object>();
    public Query WithVar(string parameterName, object parameterValue)
    {
        VarMap.Add(parameterName, parameterValue);
        return this;
    }
}

public partial class Compiler
{
    public virtual SqlResult Compile(Query query)
    {
        var ctx = CompileRaw(query);

        ctx = PrepareResult(ctx);

        foreach (var item in query.VarMap)
        {
            ctx.NamedBindings.Add(item.Key, item.Value);
        }
        return ctx;
    }
}

It takes a lot of work to be fully usable.

@ahmad-moussawi  Please take a look when you have a chance.

@ahmad-moussawi this could work with MysQL, SQLServer, PostgreSQL.
I already have the code to build just the declaration part for MySQL, SQL Server, and SQLite but the challenge would be the Where conditions. Let's review how can we generate the SQLite and PostgreSQL.
Base on this answer https://stackoverflow.com/questions/7739444/declare-variable-in-sqlite-and-use-it
if we want to use this feature for sqlite we will have to write the code like in the listing 1.1
Listing 1.1
CREATE TEMP TABLE Variables (Name TEXT PRIMARY KEY, Value TEXT);
INSERT OR REPLACE INTO Variables VALUES ('var1', 'account 1');

SELECT *
FROM accounts WHERE name =  (SELECT  Value FROM Variables where Name = 'var1')

There are multiple functions in sqlkata that are responsible to build the part of the where clause depending on what type of Where clause you use and as you see this will require more time to build it for SQLite.
For postgresSQL there is a way to build declarations, I didn't test it but base on this answer  we can use something like in the listing 1.2
WITH myconstants (var1, var2) as (
   values (5, 'foo')
)
SELECT *
FROM somewhere, myconstants
WHERE something = var1
   OR something_else = var2;

For this task, as you saw previously like in SQLite will require some time because I have to modify the code relates to the WhereConditions clause

Hi @ahmad-moussawi
In PR #184 you turned into this code
DECLARE @date DATETIME = ?
SELECT * FROM [Table] WHERE [Date] = @date;

but why you want to declare the inline var in raw sql and get very platform specific code?
I hope, that we need only replace 'unnamed' @p... params to named in binding collection,
so code
new Query("Table")
  .WithVar("@date", DateTime.UtcNow)
  .Where("Date", "@date")

should be compiled on every platform to
SELECT * FROM [Table] WHERE [Date] = @date;

with bindings collection is [{@date, 2019-05-08 12:10:00}]

@mota57 thank you for the informative research, as you've mentioned Oracle, SQLite and Postgres seem that they don't support a direct way of variable declaration, and the provided solutions would add unwanted complexity.
so sticking to the previous approach you were working on is the best for now (same example provided by @for7raid).

Ok as @for7raid I think might replace @px and I did that for a simple query. You can see the results in the picture, but I will have to test each function at Compiler.Condition.cs to make sure it works for every case.
var query = db.Query("Products")
              .WithVar("@name", "Chai")
              .Where("ProductName", "@name");

@for7raid one question if we avoid using the declaration how I am supposed to implement this with the method "WhereRaw"?

@mota57 your last picture is not I mean
My opinion that WithVar function should only add variable to bindings collection, nothing more. Then your can use this var in any part of the query, even in raw clause. No slq "declaration" needed.
var query = db.Query("Products")
              .WithVar("@name", "Chai")
              .WithVar("@date", DateTime.Now)
              .Where("ProductName", "@name")
              .WhereRaw("@date between [SaleStart] and [SaleEnd]");

goes to
select * from [Product] where [ProductName] = @name and @date between [SaleStart] and [SaleEnd]

and compile returns binsings
[{@name, "Chai"}, {@date, 2019-05-09}]

@for7raid thanks for the information I did some changes based on what you said, I add the variables to the binding property,  I removed the declaration part,   I tested what I have with methods like WhereIn, WhereRaw, Where, OrWhere, WhereIn and with nested conditions. So, it works but I have to make a couple more changes and complete the test cases in order to upload a new branch.

@mota57 amazing, if you need any help/assistance let us know.

@ahmad-moussawi ready

@mota57 thanks for the update, appreciate it, I will review it soon and get back to you

Any news about this ?

MyCode:

@ahmad-moussawi In the result.ToString() method, there is still @variable, maybe it can be better.

My Code:
var q1 = new Query("Posts").WhereRaw("Id=@Id").Define("Id", 1);

So this was never merged? I see no traces of WithVar() anywhere in the api. Was there ever another way introduced?


(feat) add fetch command to download charts for debug

Should the temporary directory for charts download be $HOME/.helmfile/cache for commands like lint?

the temporary directory for charts download be $HOME/.helmfile/cache for commands like lint?

Yes, that sounds like a great idea! I also guess doing so would probably allow us to let Helmfile reuse already prepared charts if requested.

the temporary directory for charts download be $HOME/.helmfile/cache for commands like lint?

Yes, that sounds like a great idea! I also guess doing so would probably allow us to let Helmfile reuse already prepared charts un subsequent command if requested.

Yes. I will try to address this proposal in another PR.


Accessibility: input does not have an associated label

Describe the bug
Axe accessibility testing reports the error "Form elements must have labels". While there is a label the for attrible has a differant ID to that of the input.
To Reproduce
Steps to reproduce the behavior:

Follow repo storybook example but without placeholder prop (as that prop stops Axe with another warning)

Expected behavior
The input should have a valid label for accessibility WCAG2A, and other standards
Additional context
The issue appears to be the result of the id of the input being set on input.tsx line 14. A change to     const id = props.id appears to resolve the issue, however I have not checked if there are adverse effects from that change.

Hello !
Happy to open a PR ;)


Opt out of autogenerating manifest files.

I think it would be a good idea to have the choice of not generating manifest files every time you run operator-sdk build <image> . Maybe there could be a flag like, --auto-generate-manifest-file=true/false. I would like to update the default values that are being updated each time I run build.

Closing this in favor of #84 and moving the discussion over there.


Where is the canonical hook for an off-thread task to be running just after the WSGI app has started?

For example, let's say a user has a callable which they want to run off-thread, but only after settings.configure has run for a Django app.
It's not clear how to do this without either monkey-patching the WSGI object to place to defer the signal handler or having to subclass both the Deploy and Service objects.
Surely there's a smarter way?

I ignored arguably the most obvious way: to create a Service and append it to the services list of the HendrixDeploy object.
Is this the consensus view?  If so, I'll close this wontfix.


Move host endpoints docs

Moving Host Endpoints in Security tab to Reference/resources/hostendpoint
Part of reorg of Security tab for Q2 goals. Approved by Alex.

Deploy preview for calico ready!
Built with commit 6756a424c6f808ab98cc730ec714a4849b6f5b17
https://deploy-preview-2763--calico.netlify.com


Fix for broken tweetGIF function

With this commit Xcode can build the project
Post tweet function which is used by tweetGIF function has different parameter identifier and type for media  and Data.

Thanks for the PR, I've fixed the issue in 457c44b4c492f2a59019ec4f3bcc39b2e6cd3106 as this is simply a bug from renamed arguments and not using the wrong argument. This PR will now be closed.

Cool, I think it would be better to add CI integration to prevent small bugs for the future.


[extractor/mediaset] Better DRM/Paywall message + cleanups

Description of your pull request and other information

assumed that when no formats are found is because the content is behind paywall and DRM
removed _extract_from_webpage as no longer needed
cleaned up unnecessary tests and fixed/updated not working ones
f-strings

reported various times by @Heros99
https://github.com/yt-dlp/yt-dlp/issues/4125#issuecomment-1322060176
https://github.com/yt-dlp/yt-dlp/pull/1463#issuecomment-1326027800
https://github.com/yt-dlp/yt-dlp/pull/2158#issuecomment-1271146511
Template

Before submitting a pull request make sure you have:

[x] At least skimmed through contributing guidelines including yt-dlp coding conventions
[x] Searched the bugtracker for similar pull requests
[x] Checked the code with flake8 and ran relevant tests

In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under Unlicense. Check one of the following options:

[x] I am the original author of this code and I am willing to release it under Unlicense

What is the purpose of your pull request?

[x] Fix or improvement to an extractor (Make sure to add/update tests)

thank you very much... i'll try it now

removed _extract_from_webpage as no longer needed

Are you sure? The generic extractor calls this to detect & extract embeds.


[QUESTION] About the upgrade

Hello, can you provide some updates about 1.2.0?
And 1.2.1 are there any improvements we plan to make

of course, 1.1.0 can upgrade to 1.2.0

and 1.2.1 release note:
https://github.com/apache/incubator-dolphinscheduler/releases/tag/1.2.1

for now I suggest use 1.2.0


tests: update dedidated mgr node all_daemons

5b29144 change the mgr node to a dedicated node instead of the first
monitor node.
But the change didn't update the switch-to-containers inventory which
cause this playbook to fail.
Also update the ubuntu inventory to have the same configuration.
Signed-off-by: Dimitri Savineau<EMAIL_ADDRESS>

jenkins test luminous-centos-non_container-switch_to_containers


Promise.race behavior

📖 Documentation
I asked some days before about Promise.allSettled, but I have the same question for Promise.race.
Let's say I have an array of
declare const myTEs: TE.TaskEither<E,A>[];

How would one implement a Promise.race? It should return TE.TaskEither<E,A>.
I think this example could go on this page: https://gcanti.github.io/fp-ts/recipes/async.html

I would go with:
import { fold } from 'fp-ts/lib/Monoid'
import { getRaceMonoid } from 'fp-ts/lib/Task'

fold(getRaceMonoid<Either<E, A>>())(myTEs)

We could add to the docs a table similar to:

Promise
Task
TaskEither

resolve to success
Promise.resolve(value)
T.task.of(value)
TE.taskEither.of(value) or TE.right(value)

resolve to failure
Promise.reject(value)
N/A
TE.left(value)

transform the result of a task with the function f
promise.then(f)
T.task.map(task, f)
T.taskEither.map(taskEither, f)

perform a task depending on the result of a previous one
promise.then(r => getPromise(r))
T.task.chain(task, r => getTask(r))
T.taskEither.chain(taskEither, r => getTaskEither(r))

execute an array of tasks in parallel
Promise.all(promises)
A.array.sequence(T.task)(tasks)
A.array.sequence(TE.taskEither)(taskEithers)

execute an array of tasks in parallel, collecting all failures and successes
Promise.allSettled(promises)
N/A
A.array.sequence(T.task)(taskEithers)

execute an array of tasks and succeed/fail with a single value as soon as one of the tasks succeeds/fails
Promise.race(promises)
fold(T.getRaceMonoid())(tasks)
fold(T.getRaceMonoid())(taskEithers)

How would one implement a Promise.raceToSuccess behavior (I just made up that name)? Basically, given n Promises, resolve when the first of them resolves, or reject if all promises reject.

@amaurymartiny precisely what would be the signature?

Same as Promise.race:
declare function f<E,A>(tes: TE.TaskEither<E,A>[]): TE.TaskEither<E,A>

What's that E in the return type?

or rejected with an AggregateError holding the rejection reasons if all of the given promises are rejected

Shouldn't be Array<E>?

My mistake, you're right, TE.TaskEither<E[],A> as the return type

What if tes is an empty array? The proposal doesn't seem to specify the expected behavior.
AFAIK there are two possible behaviors:

the return task never resolves (in this case the return type could be TaskEither<NonEmptyArray<E>, A>)
the return task resolves with an empty Array<E>

Another option is to change the whole signature to:
declare function f<E, A>(tes: NonEmptyArray<TE.TaskEither<E, A>>): TE.TaskEither<NonEmptyArray<E>, A>

See https://github.com/tc39/proposal-promise-any/issues/17.
It seems that it should be: "the return task resolves with an empty Array"

Another option is to change the whole signature to:

I'd stick as close as possible to the proposal, for new people transitioning to fp-ts.

ok, so you could wrap a polyfill
import * as E from 'fp-ts/lib/Either'
import * as TE from 'fp-ts/lib/TaskEither'

declare global {
  interface PromiseConstructor {
    any<A>(promises: Array<Promise<A>>): Promise<A>
  }
}

function toPromise<E, A>(te: TE.TaskEither<E, A>): Promise<A> {
  return te().then(E.fold<E, A, Promise<A>>(Promise.reject, Promise.resolve))
}

export function any<E, A>(tes: Array<TE.TaskEither<E, A>>): TE.TaskEither<Array<E>, A> {
  return () => Promise.any(tes.map(toPromise)).then(E.right, E.left)
}

I would have imagined there existed a pure fp-ts way of reaching this behavior, without using a Promise.any polyfill.
All promise methods in giogonzo's table can be done with the above code and the correct polyfill, the interesting part (for me, as a fp learner) is to only use functional programming structures.

to only use functional programming structures

Not sure what you mean, all promise methods in giogonzo's table are actually done with native implementations and polyfill, consider that under the hood:

T.task.of uses Promise.resolve
T.task.map uses Promise#then
T.task.ap uses Promise.all
T.task.chain uses Promise#then
T.getRaceMonoid uses Promise.race


Killed

Hi!
Thanks for the nice example. First of all, it works for me with small PNG images.
However! I'm trying to test out this example with larger files. I've bumped up the upload file size limit to 14 GB, and I'd like to do a test upload with that.
However, upon transferring about 1 GB of data, server drops down with message "Killed". I'm thinking it's getting killed either because it cannot calculate file size, or due to timeout, or both. Or another reason could be not enough RAM on the host, as I don't think it writes the file directly to the disk.
How can I prolong the waiting time and also write file directly to the disk, so it will accept large files like 15 GB or 100 GB?
I'll try to look into upstream documentation but maybe you could help me here.
Thanks
Den

Hey Den
Hmm...interesting. The issue seems to be upstream, so maybe creating a ticket like this on https://github.com/seanmonstar/warp would make the most sense, since I never looked into this in any more detail than shown in this example.
Best,
Mario


[build] Update babel to 7.0.0, add isNativeFunction helper

Fixes #398
Confirmed working with web-padawan/polymer3-webpack-starter@4775c2d

This change is

(CI didnt like it though :cry: )

Looks like the failing test has to be updated or removed, will look into it tomorrow.

So, yarn install gave me warnings in the past when it installed polymer-build.
yarn install v1.9.4
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
warning "polymer-cli > polymer-build ><EMAIL_ADDRESS>has incorrect peer dependency "<EMAIL_ADDRESS>[4/4] 📃  Building fresh packages...
success Saved lockfile.
✨  Done in 96.92s.

Looks like an issue on @babel side. I haven't tested this feature branch, yet.

@aomarks I have updated the goldens but there seem to be still failures likely not related to this PR.
Feel free to modify this PR or merge into separate branch in case you want to confirm everything is ok.

So, babel 7.0.0 is released and this PR needs another update.
Should I rebase, nuke and update goldens once #664 is merged? @usergenic @aomarks WDYT?

We don't have to wait for #664 -- this PR is awesome, btw THANK you for it.  Lets update to babel 7 nuke, update goldens and I'll merge as soon as verify its not breaking anything.

@web-padawan thanks again for this contribution.

@usergenic executed npm run nuke && npm run update-goldens
Wondering if my local setup is incorrect. I have seen polyserve saying it updated goldens too, but only 3 fixtures for CLI have been marked in git as changed after the script terminated.

@usergenic executed npm run nuke && npm run update-goldens
There is one failure in Travis unrelated to this PR:
@polymer/esm-amd-loader: chrome 62                Tests failed: {"value":{"stacktrace":"org.openqa.selenium.SessionNotCreatedException: session not created exception: Chrome version must be >= 66.0.3359.0\n  (Driver info: chromedriver=2.40.565383

Other that that, I believe this should be ready to merge.
Note this PR only targets polymer-build where exact version matters, I haven't tried updating other packages (some of those still use Babel 6).

Thanks, @web-padawan looking at this now.

@usergenic it will fail for external contributions because of this line: https://github.com/Polymer/tools/blob/bf6b44904e1b6034eb70dd63e4cffa4a9e03b87d/packages/esm-amd-loader/test/run-wct.sh#L6
Chrome 62 is the latest version installed in Travis by default when using config:
apt:
  sources:
    - google-chrome

I'm just not sure where to tweak wct-local config to get those tests passing.

Note it is not supposed to be started at all, actually: https://github.com/Polymer/tools/blob/bf6b44904e1b6034eb70dd63e4cffa4a9e03b87d/.travis.yml#L67

So, the esm-amd-loaders tests don't honor the SKIP_LOCAL_BROWSERS=true (that's only for WCT's own integration tests.)
However the bit about the npx wct --plugin local failing for external contributions, that's worth checking out.  Perhaps the way to solve this is to merge the PR into a branch and run the tests and then I can merge to master.  Updating PR one moment...


Fix [#3272] Add escape character missing to LITERAL_REGEX

Add the following escape characters to LITERAL_REGEX.

\h hex number class
\H not hex number class
\p Unicode property
\P not Unicode property
\R Linebreak
\X Unicode extended grapheme cluster
\S not white space

Before submitting the PR make sure the following are checked:

[x] Wrote good commit messages.
[x] Commit message starts with [Fix #issue-number] (if the related issue exists).
[x] Used the same coding conventions as the rest of the project.
[x] Feature branch is up-to-date with master (if not - rebase it).
[x] Squashed related commits together.
[x] Added tests.
[x] Added an entry to the Changelog if the new code introduces user-observable changes. See changelog entry format.
[x] All tests are passing.
[x] The new code doesn't generate RuboCop offenses.
[x] The PR relates to only one subject with a clear title
and description in grammatically correct, complete sentences.

`

The commit message should start with [Fix #3272]. Also - rebase against the current master.

Done

The build is affected by https://github.com/bbatsov/rubocop/issues/3278

The build now passes, but this has to be rebased again.

rebased


Replace prototype legacy with new code

We are using prototype_legacy_helper[1] in order to be able to use some Rails 2 only things in Rails 3. We need to update our code so as to not use any functions listed in [1] anymore. In theory it should work to replace code with code in the legacy helper.
[1] https://github.com/rails/prototype_legacy_helper/blob/master/lib/prototype_legacy_helper.rb

I'll take a spin at this.

Splitting the link_to_remote work up:
I am taking assignments, graders, results, rubrics. Danesh will take the rest.


should Site have an email?

submitted by resource specialist: "Airs:Agency has a separate notation for email, but I do not see one under airs:Site?"

Wouldn't it be SiteService and not Site that needs an email, since the Site is not related to a specific service; it's just a location where the service is.


MonoBERT and MonoT5 defaults

Follow up to #83, we have:
from pygaggle.rerank.base import Query, Text
from pygaggle.rerank.transformer import MonoT5

model_name = 'castorini/monot5-base-msmarco'
tokenizer_name = 't5-base'
reranker =  MonoT5(model_name, tokenizer_name)

I think the model_name and tokenizer_name should have defaults? So we can boil down to:
from pygaggle.rerank.base import Query, Text
from pygaggle.rerank.transformer import MonoT5

reranker =  MonoT5()

Also, the regression scripts should be modified to use this new abstraction?

@lintool it already defaults to 'castorini/monot5-base-msmarco' and 't5-base'! I included it in the example because I thought it'd be helpful to show the user that  they can easily change model/tokenizer using pretrained stuff from huggingface.
as for regression scripts (if you mean the evaluation scripts), there shouldn't be any changes necessary because monoBERT/T5 also accepts instances of PretrainedModel or PretrainedTokenizer

I see. Perhaps change to a comment?
Something like:
reranker =  MonoT5()
# Initializes with defaults, model_name = 'castorini/monot5-base-msmarco', tokenizer_name = 't5-base'
# Same as: MonoT5('castorini/monot5-base-msmarco', 't5-base')

README docs should be as simple as possible IMO.
re: regression scripts, e.g., https://github.com/castorini/pygaggle/blob/master/pygaggle/run/evaluate_passage_ranker.py#L80
Wouldn't it make sense to use this new abstraction so we don't have multiple code paths (that might subtly diverge down the road...)? And same with MonoBERT?

agree with changing it to a comment! will make a PR for it
as for the regression scripts, I did think of changing those, but wasn't sure how because of PassageRankingEvaluationOptions, since values like options.device, options.from_tf, options.batch_size don't fit in nicely in the current abstraction.
I don't particularly want to add more parameters to MonoBERT/MonoT5's __init__ method for these options (except maybe batch_size) as it'd be imo an anti-pattern


Significantly speedup protorev tests.

It no longer sets up all pools in every test.
Instead tests that need pool setups have to call a separate function
Locally takes my laptop from 27 seconds to 12 seconds, we will see what happens on CI. (Was 65s)

Summary by CodeRabbit

Tests

Enhanced testing infrastructure for more accurate simulation of pool setups.

Refactor

Improved setup procedures for testing, focusing on more detailed pool configurations.

Chores

Streamlined test setup processes to better reflect real-world scenarios.

Old: 65s in CI
New:
ok  	github.com/osmosis-labs/osmosis/v24/x/protorev/keeper	19.530s


Fix #2417, #2701

Fix #2417: AF in data
Fix #2701: EnsureInitialized must be called on the UI thread

👍


NIFI-9800: Unwrap SQLException in PutDatabaseRecord when table does not exist

This PR simply unwraps the ProcessException thrown when getting the table information in PutDatabaseRecord, so the original SQLException can be processed accordingly. No behavior should change, it just shows up differently in the logging. I didn't add a unit test as the change is supposed to not cause a regression, and the existing tests still pass.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[x] Has your PR been rebased against the latest commit within the target branch (typically main)?

[x] Is your initial contribution a single, squashed commit? Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not squash or use --force when pushing to allow for clean monitoring of changes.

For code changes:

[ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[ ] Have you written or updated unit tests to verify your changes?
[ ] Have you verified that the full build is successful on JDK 8?
[x] Have you verified that the full build is successful on JDK 11?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible.

reviewing...


Hindi localization of content/en/partners/

We are restarting the effort for Hindi localization of Docs.
Issue ->  #13957
Slack thread -> https://kubernetes.slack.com/archives/C1J0BPD2M/p1628742429109500
/language hi
/kind feature

lgtm
Good work @anubha-v-ardhan !!


⚠ Bofrost.lu has degraded performance

In 64d823b, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 3464 ms

Resolved: Bofrost.lu performance has improved in 61e397d.


osm2pgsql: 1.4.0 -> 1.4.1

Motivation for this change
upstream released 1.4.1
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[x] Tested compilation of all pkgs that depend on this change using nix run nixpkgs.nixpkgs-review -c nixpkgs-review wip
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

This is a semi-automatic executed nixpkgs-review with nixpkgs-review-checks extension. It is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch).
If you have any questions or problems please reach out to SuperSandro2000 on IRC.
Result of nixpkgs-review pr 111819 run on x86_64-darwin 1

1 package built:

osm2pgsql


ECC-1118| migration to hmrc-mongo, updating dependencies WIP

Once we made the decision to replace the SessionCache with on that extends SessionCacheRepository from hmrc-mongo then the interface to the cache changes and this change becomes huge. I questioned the value of doing so at the time when we did it in subscription... but its done now, and at least our session caches are now consistent with each other and the rest of hmrc :)

checked that the hmrc-mongo SessionCache was being used

Once we made the decision to change the interface to the SessionCache by extending SessionCacheRepository from hmrc-mongo, this refactor became huge :) I questioned the value at the time when we did so in subscription... but its done now, and both our session caches are now consistent with each other and other projects that use hmrc-mongo.

Checked that the SessionCache now extends SessionCacheRepository from hmrc-mongo
Reviewed the ripple effect the change in the SessionCache interface has caused (implicit request params etc).
All compiles, and the tests are all passing locally.

LGTM 👍


--proxy

Is your feature request related to a problem? Please describe.
I live in russian, and sometimes "Roskomnadzor" blocks sites. Webtoon are blocked for no reason. How do i download chapters?
Describe the solution you would like
add "--proxy [optional proxy adress]", if there is no connection, WD tries to use some proxy adresses or using ur own adress.
Additional context
I hate living in russia cause' "Roskomnadzor" does more crazy.

I added the ability to use --proxy in https://github.com/Zehina/Webtoon-Downloader/commit/e965330c066044a6b6a4f849caf3de74c6388ed2


[backport] Fix typo in n_step_gru docs

Backport of #5006

Jenkins, test this please.

Jenkins CI test (for commit d5d068f35391aed7bb2e75867948943602fa2af9) succeeded without errors!


Added support for Widgets in carousel

Addresses #10

I am not a maintainer, but I don't like how this class is a copy of the other one...
isn't it possible to extract common pieces and inherit a base class?
this is a lot of duplication that will create a burden in the long term.

Yes  Mathieu, I agree. I just did not want to disturb the existing code and
hence tweaked it to add the new support. I will try to integrate it in the
same class so that we need to import only a single class to achieve the
goal. Thank you for getting back to me.
Regards,
Jinu
On Wed 26 Sep, 2018, 5:04 AM Mathieu Jobin<EMAIL_ADDRESS>wrote:

I am not a maintainer, but I don't like how this class is a copy of the
other one...
isn't it possible to extract common pieces and inherit a base class?
this is a lot of duplication that will create a burden in the long term.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/jlouage/flutter-carousel-pro/pull/16#issuecomment-424535919,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Ao_FWdMntCLirM_a_RxwTjvSr-BPS9JZks5uer2RgaJpZM4Wh7xG
.

Thanks Mathew.
On Mon 15 Oct, 2018, 2:42 PM Jlouage<EMAIL_ADDRESS>wrote:

Merged #16 https://github.com/jlouage/flutter-carousel-pro/pull/16 into
master.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/jlouage/flutter-carousel-pro/pull/16#event-1903496186,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Ao_FWfdaPzN-DdlPrSIhM932EX89TJCUks5ulFGQgaJpZM4Wh7xG
.


OrientDB. Connection leaks or just not closing connections

OrientDB Version, operating system, or hardware.

v2.0 SNAPSHOT[ ] - .18[ ] .17[ ] .16[ ] .15[ ] .14[ ] .13[ ] .12[ ] .11[ ] .10[ ] .9[ ] .8[ ] .7[ ] .6[ ] .5[ ] .4[ ] .3[ ] .2[ ] .1[ ] .0[ ]
v2.1 SNAPSHOT[x] - .19[x],.16[ ] .15[ ] .14[ ] .13[ ] .12[ ] .11[ ] .10[ ] .9[ ] .8[ ] .7[ ] .6[ ] .5[ ] .4[ ] .3[ ] .2[ ] .1[ ] .0[ ]
v2.2 SNAPSHOT[ ] - .rc1[ ] .beta2[ ] .beta1[ ]

Operating System

[x] Linux
[ ] MacOSX
[ ] Windows
[ ] Other Unix
[ ] Other, name?

We have a large multithreaded Java EE application running on Wildfly 8.
We are using OrientDB 2.1.19.
And we have some problems with the connection leaks. At some point orient server stops responding and all threads working with db stuck on retrieving new connection.
Configuration is following:
OGlobalConfiguration.CLIENT_CONNECT_POOL_WAIT_TIMEOUT.setValue(5000);
OGlobalConfiguration.CLIENT_DB_RELEASE_WAIT_TIMEOUT.setValue(5000);
OGlobalConfiguration.CLIENT_CHANNEL_MAX_POOL.setValue(1000);
OGlobalConfiguration.DB_POOL_MIN.setValue(100);
OGlobalConfiguration.DB_POOL_MAX.setValue(5000);
OGlobalConfiguration.STORAGE_LOCK_TIMEOUT.setValue(5000);
OGlobalConfiguration.DB_POOL_IDLE_TIMEOUT.setValue(5000);
OGlobalConfiguration.DB_POOL_IDLE_CHECK_DELAY.setValue(1000);

OPartitionedDatabasePool we're getting thru OPartitionedDatabasePoolFactory
poolFactory = new OPartitionedDatabasePoolFactory();
documentPool = poolFactory.get(orientDBPath, username, password);

Then depending on our needs we are using ODatabaseDocumentTx or OObjectDatabaseTx using
documentPool.acquire();

Acquired ODatabaseDocumentTx or OObjectDatabaseTx we are passing to so called executor, which executes Runnable or Callable.
public void run(ORunnable<DB> oRunnable) {
    // db is the private member of executor containing acquired db.
    try {
        //...
		oRunnable.run(db);
        //...
	} catch (Exception e) {

	} finally {
		if(!db.isClosed()) {
			db.close();
		}
	}
}

As you can see we are closing db in finally section, also we are fully detaching all documents recieved from db. But in some cases number of connections to DB is not dropping and after that we're getting following exception
    2016-07-28 23:10:45,957 FINE  [com.orientechnologies.orient.client.remote.ORemoteConnectionManager] (default task-46) Network connection pool is receiving a closed connection to reuse: discard it
2016-07-28 23:10:45,957 FINE  [com.orientechnologies.orient.client.remote.ORemoteConnectionManager] (default task-46) Cannot unlock connection lock: java.lang.IllegalMonitorStateException
 at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:155) [rt.jar:1.7.0_13]
 at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1260) [rt.jar:1.7.0_13]
  at java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:460) [rt.jar:1.7.0_13]
  at com.orientechnologies.common.concur.lock.OAdaptiveLock.unlock(OAdaptiveLock.java:123) [orientdb-core-2.1.19.jar:2.1.19]
  at com.orientechnologies.orient.enterprise.channel.binary.OChannelBinaryAsynchClient.unlock(OChannelBinaryAsynchClient.java:371) [orientdb-enterprise-2.1.19.jar:2.1.19]
  at com.orientechnologies.orient.client.remote.ORemoteConnectionManager.remove(ORemoteConnectionManager.java:128) [orientdb-client-2.1.19.jar:2.1.19]
  at com.orientechnologies.orient.client.remote.ORemoteConnectionManager.release(ORemoteConnectionManager.java:119) [orientdb-client-2.1.19.jar:2.1.19]
  at com.orientechnologies.orient.client.remote.OStorageRemote.endResponse(OStorageRemote.java:1643) [orientdb-client-2.1.19.jar:2.1.19]
  at com.orientechnologies.orient.client.remote.OStorageRemote.command(OStorageRemote.java:1240) [orientdb-client-2.1.19.jar:2.1.19]
  at com.orientechnologies.orient.client.remote.OStorageRemoteThread.command(OStorageRemoteThread.java:453) [orientdb-client-2.1.19.jar:2.1.19]
  at com.orientechnologies.orient.core.sql.query.OSQLQuery.run(OSQLQuery.java:72) [orientdb-core-2.1.19.jar:2.1.19]
  at com.orientechnologies.orient.core.sql.query.OSQLSynchQuery.run(OSQLSynchQuery.java:85) [orientdb-core-2.1.19.jar:2.1.19]
  at com.orientechnologies.orient.core.query.OQueryAbstract.execute(OQueryAbstract.java:33) [orientdb-core-2.1.19.jar:2.1.19]
  at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.query(ODatabaseDocumentTx.java:717) [orientdb-core-2.1.19.jar:2.1.19]

Here is the piece of code for the exception above
ODocument storedSession = orient.onDocuments().call(new ODocCallable<ODocument>() {
					@Override
					public ODocument call(ODatabaseDocumentTx db) {
						try {
							List<ODocument> list = db.query(new OSQLSynchQuery<ODocument>("select from " + CLAZZ + " where storedSession_sessionID = ?"), sessionId);

							if (list.isEmpty()){
								return null;
							}
							return documentUnpin(list.get(0));
						} catch (Exception e) {
							// PersistentSession may not be in DB on request (for example in workspaces)
						}
						return null;
					}
				});

Any suggestions how to improve the situation with connections?
Thanks.

hi @mikayeln,
checking it

hi @mikayeln,
Is the database acquire from the pool by the same thread that use it and release it ?
Regards

Hi @tglman,
Yes. The database acquired from the pool will be released by the same thread, which has received the database.

Under what conditions acquired connection will not be released?

hi @mikayeln,
actually the release should be done by the same thread that did the actquire:
ex:
public void run(ORunnable<DB> oRunnable) {
    // db is the private member of executor containing acquired db.
    DB db= pool.acquire();
    try {
        //...
        oRunnable.run(db);
        //...
    } catch (Exception e) {

    } finally {
        if(!db.isClosed()) {
            db.close();
        }
    }
}

otherwhise could be there some issues on releasing.

Hi @tglman ,
Thanks for the suggestion. We foolwed it, also we swithed orient version to 2.2.8.
And we faced another interesting situation.
At some point OClientConnectionManager kept to itself ~80000 (sometimes more) active sockets. And as a result all requests're rejected. So the question is why sockets are still open? SO timeout is set to 5000ms. storageKeepOpen = false. cleanDelay = 2000ms.

hi @mikayeln,
We did a few fix on this topic in the last few days, some are already included in 2.2.9 and other will be released in 2.2.10, i'm quite confident that with 2.2.10 this is going to be sorted out.
the 2.2.10 is going to be at most in ~2 weeks.
Regards

hi,
This was fixed and released, closing.
bye


Remove code not used in DMT (probably some things in the API and stuff related to tree node)

domain classes:

use of tree_node -> remove

test_tree_err
domain.py -> code in this file is not used anywhere else, remove
document_service ->  generate_tree() can be removed
genereate_code_with_plugin_use_case -> can be removed

schema.py  -> delete
schema is used by:

generate_system_blueprints.py -> code not used anywhere, delete
used in code_genrators...
bdd test

dto -  remove

home
DMT/code_generators - docuble check before remove.

repository
dmss/TemplateRepositoryForDMSS.py - remove

restful
request_object.py , only usedf in code generator.

utils

data_structure folder:

dictionary_test.py

only pretty_eq, print_pygments and find function are used by other code...
maybe some of the code in data_structure belongs to utils instead.
dictionary.py can be removed, along with dictionary_test.py
investigate some more...

create_entity_utils:

CreateEntityException and InvalidDefaultValue delete
generate_system_blueprints.py is unused

group_by.py not used - remove

should this be removed? (new api structure
features/blueprints endpoints
features/system endpoints
home/DMT/code generators


store.getMidnightMode doesn't get the new value

When I select the parameter MidnightMode in the settings panel and click on Apply, everything updates correctly.
But if I want to change later the value (without closing the app), the store.getMidnightMode variable does not change even though the value is updated with store.setMidnightMode.
See main.js:
    // MidnightMode
    const midnightMode = args[15]
    console.log(midnightMode + ' store.getMidnightMode = ' + store.getMidnightMode)
    if (midnightMode !== store.getMidnightMode) {
      changed = true
      console.log('Storage: the parameter midnightMode is different.')
      store.setMidnightMode(midnightMode)
    }

Here is an extract from the terminal for the line console.log(midnightMode + ' store.getMidnightMode = ' + store.getMidnightMode):

when I choose Mid Sunset to Fajr

Apply button clicked.
1 store.getMidnightMode = 0
Storage: the parameter midnightMode is different.
...

change of mind, I prefer the other parameter (Mid Sunset to Sunrise):

Apply button clicked.
0 store.getMidnightMode = 0

store.getMidnightMode stay to 0 even though I just updated to 1 the first time

Fixed this issue.
The problem was that all get parameters returned variables but should be functions.
Also replaced setTimeout with await on fetching data.

Hi! Thank you for your time! :)
I'm new to JavaScript and the syntax async/await is not something I understand completely..
But your answer and modifications are a great help.

Hi, actually this is my first contribution ever🙂 Hope it was helpful. found by label "good first issue".


Fix the problem with re-preparing already prepared statements [WIP]

Fixes #1268
Problem
The PrepareStatementCache.async was not thread safe.
Solution
The cache now stores Futures and the async version simply waits for a result. That is equivalent of what was going under the hood anyway. Check out AbstractSession.java#L79.
Checklist (TBD)
I am using this PR to trigger test suite.

[ ] Unit test all changes
[ ] Update README.md if applicable
[x] Add [WIP] to the pull request title if it's work in progress
[ ] Squash commits that aren't meaningful changes
[x] Run sbt scalariformFormat test:scalariformFormat to make sure that the source files are formatted

@getquill/maintainers

Should I write explicit changes trying out the concurrency issues?

Well, I'm almost sure I've seen an occurrence of the await at least once in
a test outside of the test folder. I'll check it when I get home.
Anyway, I'll do as you suggest. I was just worried about the solution not
being complete - but people will use both sync and async contexts together
rarely. Plus the underlying datastax driver is blocking the execution
anyway.
Thanks for the review.
On Mon, 4 Feb 2019 at 18:26, Flavio W. Brasil<EMAIL_ADDRESS>wrote:

@fwbrasil commented on this pull request.
In
quill-cassandra/src/main/scala/io/getquill/context/cassandra/PrepareStatementCache.scala
https://github.com/getquill/quill/pull/1315#discussion_r253564482:

@@ -8,34 +8,24 @@ import com.google.common.cache.CacheBuilder
import com.google.common.hash.Hashing

import scala.concurrent.{ ExecutionContext, Future }
-import scala.util.Success
class PrepareStatementCache(size: Long) {
Let's make the cache generic and have two versions of it: one sync and one
async.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/getquill/quill/pull/1315#pullrequestreview-199708582,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFBL8enZ9fpUQvyKneU3xAMlAKhJXJmzks5vKG1PgaJpZM4acilk
.

Any progress on this PR?

As far as I can remember, I'm waiting for a feedback - I changed the code
as requested.
I'll check it out when I get to my laptop. Maybe I should remove the WIP
flag, right?
On Mon, 4 Mar 2019 at 06:55, Dennis Mellert<EMAIL_ADDRESS>wrote:

Any progress on this PR?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/getquill/quill/pull/1315#issuecomment-469128096, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AFBL8Uz7UL_fqYiGU308RBsdsr_sIwTWks5vTLVngaJpZM4acilk
.

Rebased ...


Delete obsolete files from older versions in root folder

Questions
Answers

Description?
Delete obsolete files from root folder - footer.php, header.php, images.inc.php.

Type?
improvement

BC breaks?
no

Deprecations?
no

Fixed ticket?

Sponsor company
https://www.openservis.cz/

How to test?

@Hlavtox
People already have, for example, PS version 8.1.2 with these files. So, giving it to 8.0.0.sql it makes no sense.
If we put it to 9.0.0, we can be sure whis files will be deleted for all people.
Of course its possible to add to earlier version as well (8.1.3+)...
But i dont know if/when this PR will be merged/allowed.
So I targeted do develop branch...

Hi @ShaiMagal
Thank you for your PR, could you provide us an how to test or an issue that can give us some details on how we can validate your issue ?
Thank you

@AureRita
Create files footer.php, header.php, images.inc.php in root folder of PrestaShop.
Make update PS to version 9 (I don't know, if this is possible to do this yet?)
wdyt @Hlavtox ?

Hi @ShaiMagal
Thank you for your details, currently we can only upgrade to 8.1.x, when we want to upgrade to develop, we have some issue on it. When we want to upgrade from <IP_ADDRESS> to 8.1.2, we have the files 'images.inc.php' wich is deleted, but I think this was before your PR. The 'header.php' and 'footer.php' are still here from an upgrade from <IP_ADDRESS> to 8.1.2.
@PrestaShop/product-council WDYT, do we put this PR in blocked or anything else ?

Yeah, it pretty much aligns what we talked about with @ShaiMagal
in private. I think a proper solution would be to restore and improve pscleaner module. Dominik offered to maintain it. :-)
We could all of these things there.

@matks @Hlavtox ok, I am closing this PR.
I will continue with deleting this files "manually" with my script after every update , so for me it is not problem. I tried to help other people, not me. :)


NIFI-12592 Update curator-client to 5.6.0

Summary
NIFI-12592
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking

[X] Apache NiFi Jira issue created

Pull Request Tracking

[X] Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
[X] Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

[X] Pull Request based on current revision of the main branch
[X] Pull Request refers to a feature branch with one commit containing changes

Verification
Please indicate the verification steps performed prior to pull request creation.
Build

[ ] Build completed using mvn clean install -P contrib-check

[ ] JDK 21

Licensing

[ ] New dependencies are compatible with the Apache License 2.0 according to the License Policy
[ ] New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

[ ] Documentation formatting appears as expected in rendered files

@exceptionfactory thanks for the information. The additional version has been bumped up to 5.6.0


When adding or removing nodes, animations start at the root

Hi, I've seen this issue before. https://github.com/bumbeishvili/org-chart/issues/261
Sorry for posting the same question issue, I think I may have misunderstood.
chart.update(node)
It says that writing this code will fix it, but I'm not sure how.
https://stackblitz.com/edit/web-platform-lwyild?file=index.html
I want to use the site here as an example.
chart.removeNode('O-6069')
When I write this code,
or when I write it with addNode,
chart.addNode({
		      name: "test",
		      imageUrl: "",
		      area: "AP",
		      profileUrl: "",
		      office: "kilo",
		      tag: "something",
		      isLoggedUser: false,
		      positionName: "CTO",
		      id: 'O-6069_1',
		      parentId: 'O-6069',
          _expanded: true
		    });

how do I prevent the animation from starting on root?

I think there can be many ways to achieve the result you want.
The simplest way would be to retrieve the parent node of the node you are adding
(you can get all nodes by calling chart.getState() and then look into props and find property which stores all nodes, then filter them by id)
After you get that node, you can invoke
chart.addNode and then immediately invoke chart.update(foundParentNode).
This might work, but again, not sure , haven't tested

Ah, I get it now.
Thanks for the quick reply.

Sorry, it still animates from root.
I do chart.addNode and then immediately invoke chart.update(foundParentNode).
Can you let me know if this is the way to go?
const currNode = chart.getChartState().allNodes.find(obj => obj.id === nodeId);

chart.addNode
chart.update(currNode)

If it does not work, then no.
What I'd suggest now, is to update the data itself (push some dummy node to the data array)
And then invoke
chart.update(currNode)
So, you need to do this part manually
https://github.com/bumbeishvili/org-chart/blob/c8ebb0490ecfcf1424c16a8528a4a835d3efde53/src/d3-org-chart.js#L503
And then you need to override this part
https://github.com/bumbeishvili/org-chart/blob/c8ebb0490ecfcf1424c16a8528a4a835d3efde53/src/d3-org-chart.js#L1188
Hope this makes sense, if not, then I am out of options

Thanks for being so kind to let me know.
The code you gave me didn't work, but it was a great hint.
I gave the buttons on the node that I open when I draw the chart an id of class.
I don't know if my approach is correct, but this is how I did it.

call addNode without giving _expanded: true
click the node button with the specific node id

Interesting approach


how to: configure the concat field

From #6
*) Mosaic Chart with Labels
I think ConfigurationProperty is missing a concat option:
  "config": {
    "view": {
      "stroke": ""
    },
    "concat": {"spacing": 10},
    "axis": {
      "domain": false,
      "ticks": false,
      "labels": false,
      "grid": false
    }
  }

Yes you are correct. I have pushed concat configuration for the next release with coConcat as the configuration function and cocoSpacing and cocoColumns as the options.
e.g.
cfg =
    configure
        << configuration (coView [ vicoStroke Nothing ])
        << configuration
            (coAxis
                [ axcoDomain False
                , axcoTicks False
                , axcoLabels False
                , axcoGrid False
                ]
            )
        << configuration (coConcat [ cocoSpacing 10 ])


Defending Major Hold Speak to Fieldco objective not marked as completed.

this appeared i do not remember what hold it was but it will not go away.

@LABWAZHERE
Thanks for reporting this. This is the Legate Field CO for Markarth. Defense quests of major holds have this objective to meet with the Field CO. Unfortunately its not getting marked as completed.

same with the riften one for storm cloaks

Defense quests of major holds have this objective to meet with the Field CO. Unfortunately it’s not getting marked as completed.

I will also add that the battle of markarth just boots up without meeting with that general, it just started.

Everything reported here should be resolved now in v1.0.6a. Please try again.

Marking this as closed. I am unable to reproduce this now


update prow-deploy image to 0.0.7

Description of changes:

update prow-deploy image to 0.0.7
I have built and pushed the new image

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

cc: @JamesJHPark


Can't authenticate on Debian 9: WRONG_CURVE

I'm using lxd (installed from snap, version 4.2) and pylxd from pip (version 2.2.11) on Debian 9.12.
When I try to authenticate with client.authenticate() according to https://pylxd.readthedocs.io/en/latest/authentication.html, I get an error from ssl.
ssl.SSLError: [SSL: WRONG_CURVE] wrong curve (_ssl.c:720)
From what I can see, this is because of a change in openssl, but I'm not sure how to fix this.
OpenSSL version is 1.1.01-1~deb9u1

What's the python version on Debian 9?  (I think it's Py3.5?).  If so, you are probably hitting https://bugs.python.org/issue31809.  If it is py35, can you try with a more recent version?

Yes, Python version is 3.5. Can confirm that Python 3.6 works fine. However, IMO, it's not really an option to install Python3.6 on Debian 9. The package would have to be maintained manually, as no prebuilt version of 3.6 exists for Debian (afaik) as Debian 10 uses Py3.7.

I don't think you can fix this using Debian 9 / Python 3.5 packages as it appears to be an issue with Python35 and the version of SSL that ships with Debian 9 ... i.e. it's not an pylxd issue.  You might be able to build a custom OpenSSL that doesn't have the issue, or upgrade to Python > 3.5.

Yes, a fix is probably not possible in pylxd. However, a change in LXD (i.e., the addition of a non-elliptic curve cipher) should fix the issue, right? According to testssl, currently only ECDHE-ECDSA-AES* ciphers are enabled, which is odd, because according to the source in lxd/network.go, RSA ciphers should be supported. I'll file an issue in the other repo, thanks.


Can it be adapted to support models in OpenAI format？

Can it be adapted to support models in OpenAI format? Some third-party model providers like togetherai support the OpenAI format.

Done, thanks for your star
Can I customize prompts as well? That would be perfect, thank you!


Adding more_info into reportData schema

Description
Adding "more_info" into the report data schema, as it is the only field missing in the OpenAPI spec.
Type of change

Bug fix (non-breaking change which fixes an issue)
Refactor (refactoring code, removing useless files)

Testing steps
make openapi_check
Checklist

[ ] make before_commit passes
[x] updated documentation wherever necessary
[x] added or modified tests if necessary
[x] updated schemas and validators in insights-data-schemas in case of input/output change

Codecov Report

Merging #906 (bb82624) into master (317b97f) will increase coverage by 0.06%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #906      +/-   ##
==========================================
+ Coverage   59.92%   59.99%   +0.06%
==========================================
  Files          22       22
  Lines        3137     3137
==========================================
+ Hits         1880     1882       +2
+ Misses       1044     1042       -2
  Partials      213      213

Impacted Files
Coverage Δ

content/content.go
77.77% <0.00%> (+0.82%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 317b97f...bb82624. Read the comment docs.


how to reproduce the pitch, intonation and other nuances of a voice? I would like to find a tutorial or a guide (Intonation and Pitch in RVC or mastery of RVC Configuration)?

RVC ConfigurationI know how to reproduce a voice using a certain number of data, clean them and then run RVC. But I know nothing about the index file, or anythign related to: n_fft, hop_length, win_length, and n_mels, IVF, Flat ??
I would like to learn to not only reproduce a voice (vocal timbre) of someone, but also how to reproduce the pitch, intonation and other nuances of a voice.
Is that possible with RVC?
If so, where can I learn that please?

Sorry to break you up, but I have no knowledge either about those things.
Iirc, Index files are like external/additional voice data, the more data you have the more bigger index file size.
If you want to learn more, you could try asking the original RVC creator or you could try asking here
https://discord.gg/aihub

Sorry to break you up, but I have no knowledge either about those things.
Iirc, Index files are like external/additional voice data, the more dataset you have the more bigger index file size.
If you want to learn more, you could try asking the original RVC creator or you could try asking here https://discord.gg/aihub

Since you are the creator, can I ask you a direct question please?
(Also, ok that part, maybe I am imagining people using those stuff and me being missing stuff? Maybe only professionnal like the ones hired at elevenlabs know that stuff? idk, anyway I hoped. I asked gpt for some stuff but seemms too complicated hoped someone could explain)
Anyway, my question: what would YOU (@ArkanDash ) do to try to train the voice of Geralt of Rivia (example of his voice: https://www.youtube.com/watch?v=zxau03Row9U) in order to obtain a voice ALMOST EXACT as his?  I would not mind any long text, I can read as much as you can write lol (even if thats 2 pages heheh).
Finally, where do you guys use the pth file obtained, do you only use it in the same rvc software? Here is an important question:

Do you know ways to use the pth model outside and in other "voice techs" to introduce perhaps variations of all sorts? Please give me the best info You have (if you feel want to share what I am seeking via discord or whatever, it's okay just tell me whatever. Again the magical word "please".

Since you are the creator, can I ask you a direct question please?
(Also, ok that part, maybe I am imagining people using those stuff and me being missing stuff? Maybe only professionnal like the ones hired at elevenlabs know that stuff? idk, anyway I hoped. I asked gpt for some stuff but seemms too complicated hoped someone could explain)
Anyway, my question: what would YOU (@ArkanDash ) do to try to train the voice of Geralt of Rivia (example of his voice: https://www.youtube.com/watch?v=zxau03Row9U) in order to obtain a voice ALMOST EXACT as his?  I would not mind any long text, I can read as much as you can write lol (even if thats 2 pages heheh).
Finally, where do you guys use the pth file obtained, do you only use it in the same rvc software? Here is an important question:

Do you know ways to use the pth model outside and in other "voice techs" to introduce perhaps variations of all sorts? Please give me the best info You have (if you feel want to share what I am seeking via discord or whatever, it's okay just tell me whatever. Again the magical word "please".

First thing first, I'm not the original RVC creator, here is the original rvc project
https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI
If you want to train a voice, you would need the voice dataset, it could be getting the voice inside the game or recording the audio from a cutscenes. "How long of the data do I need?" At least 10 minutes or more. It kinda depends, sometimes more dataset have its advantage than lower dataset. You just need to test it for yourself. If your voice has some kind of background sound you could use UVR (search on github) before training.
.pth files are the model voice that has been generated by the rvc software. you can only use it on rvc software. Other software sometimes have .pth file but it's not compatible with rvc or other software. .pth files are generated using pytorch library in python.


Set version of cohttp-lwt-unix from 1.2.0 to 1.1.1

This change fix the issue #244 , the version of cohttp-lwt-unix must be the same as cohttp-lwt.

Thanks!


Added code to tutorial to prevent non-existent queue error

I'm a novice in this area, but the example of worker.js in the 2nd part of the javascript tutorial failed for me as the queue didn't exist when the worker.js file was launched. I added the necessary "assertQueue" call to create prior to attempting to consume.

@carled Please sign the Contributor License Agreement!
Click here to manually synchronize the status of this Pull Request.
See the FAQ for frequently asked questions.

@carled Thank you for signing the Contributor License Agreement!

This matches what the executable version does. I will review the formatting and merge manually. Thank you!


Mazerunner cell size setting

Hello. Where is the cell size of the maze adjusted in the Mazerunner code? We are working on a maze solving robot with a cell size of 200mm. Where do you use the size of the cell in the code, we would be very happy if you could help us with this. Thank you in advance.

OK - so that may have been easier than I expected.
I just pushed up a branch called cell-size that should have all the changes. config.h has the cell size as a constant FULL_CELL and I think I have found the places where that needs to be used in the code.
Since it did not take long, I may have missed some so I would be grateful if you could grab that branch and test it to confirm that it is good. If so, I will merge it into the main branch.

 I will inform you with the test results. Thank you very much for your interest.

I shall create a pull request for the changes. Perhaps you could
feedback through that?
 I will inform you with the test results. Thank you
very much for your interest.
—
Reply to this email directly, view it on GitHub
https://github.com/ukmars/ukmarsbot-mazerunner/issues/6#issuecomment-1099544852,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAFC3KWVOG5NIU6MH5SLI4LVFBVJ5ANCNFSM5TMH6EZQ.
You are receiving this because you commented.Message ID:
@.***>

--
Peter Harrison

Hello Mr. Peter. The change you made solved our problem and the code is now more functional. Thank you very much for your help.
Also, I have one more question for you.
forward.start(-60, 120, 0, 1000); I guess the value -60 in the line is for the robot to go back from the cell center before going to the target. Does the value calculation also occur by adding 12mm of wall thickness to the BACK_WALL_TO_CENTER constant? We are very grateful to you for helping us with our questions about the code.
int Mouse::search_to(unsigned char target)
{
flood_maze(target);
// wait_for_front_sensor();
delay(1000);
enable_sensors();
reset_drive_system();
enable_motor_controllers();
if (not handStart)
{
forward.start(-60, 120, 0, 1000);

that is great, i am glad that you are making progress.
That code is intended to make the robot back up to the wall behind it so that it starts from a known position in the cell.
Normally, a hand start would have the robot already placed there by the user. If the robot is auto-starting, it may have stopped a little out of place and so we cannot be certain of its position after turning around.
The distance it backs up is not critical - I don't really care about the wall thickness. It just needs to be more than the likely offset and slow enough that it does not push out a wall or bounce of awkwardly. Also, I hope that, if it is not correctly aligned, the extra motion will force it to straighten up against the wall. For this reason it is a goot idea to have the back end of the robot flat and wide enough for this self-alignment.

Hello Peter. Our work and development on UKmarsbot continues. I have a new question for you. IR sensors and phototransistors are used as sensors in all micromouse robots we examined. These sensors are highly affected by the lights in the competition environment. What is the use of TOF sensors such as vl53l0x instead of these sensors? The reading speed of these sensors is 20ms, but your Timer2 interrupt is 2ms. I think this temporal difference seems to be the biggest factor. We are curious about your experiences and opinions on this subject.

The TOF snsors are far too slow for normal running. They can be useful during search if placed at the front of the mouse to detect wals a long way ahead but I think that is quite advanced stuff. For now, I would stick with the simple IR sensors.


Feature: can set default CookieJar for Client

Description
This is useful for api clients where need to add a cookie stored in a database or somewhere else.
Example
new Client([RequestOptions::COOKIES => $cookieJar]);

I don't understand what this issue is about. Can you provide more info?

Closing as comment on the related PR here: https://github.com/guzzle/guzzle/pull/2317


Light up GEOS support in GDAL

This PR lights up GEOS support in GDAL.
Fixes #400

Is there any particular hold-up on this PR?

Thanks for the ping!
I had started testing it, but ran into issues for static builds.


Confusing log message when all tests are filtered out

Description
I noticed a confusing warning in the LUT log after restart LUT (see below). Essentially, after filtering out tests, none left to execute, which is expected because there's no code change since LUT was stopped. But this message makes it sounds like an error.  @Satya Madala Could you guys change this message at least for filter scenario (if there’s tests discovered but all filtered out).
[15:02:09.546 Info] [TestRunner 1] No test is available in C:\Users\gel\source\repos\XUnitTestProject1\.vs\NUnitCoreTests\lut\0\t\NUnitCoreTests\Debug\netcoreapp2.0\NUnitCoreTests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

Steps to reproduce
With latest dogfood build:
Open a test project -> Start LUT -> Stop LUT -> Start LUT again

@genlu  Is below message looks good?
PS C:\Users\samadala\src\vstest> .\artifacts\Debug\net451\win7-x64\vstest.console.exe .\test\TestAssets\SimpleTestProject\bin\Debug\net451\SimpleTestProject.dll /testcasefilter:"Name=abc"
Microsoft (R) Test Execution Command Line Tool Version 15.8.0-dev
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
No test available for testcase filter `Name=abc` in C:\Users\samadala\src\vstest\.\test\TestAssets\SimpleTestProject\bin\Debug\net451\SimpleTestProject.dll

What is the timeline your looking for this issue to get fixed?

The message above is fine. But for LUT, the filter can be very long (e.g. all tests in project). So can we not include the filter string itself in the message?
In terms of timeline, 15.8 preview 4 would be great. @ManishJayaswal

"No test matches the given testcase filter" may be better.
For LUT, in most cases, this would not be an error condition. It would just point to the fact that no tests are covering the line which was edited. Is there any way we can make this message "informational" so that it only shows up with verbose logging? Else this will keep showing as error when technically there is no error- the program is working as expected. If it only shows up in verbose logging then spending the filter text to the message may help with diagnostics so it may be ok.
For timeline - preview 4 of 15.8 is fine.

@ManishJayaswal We didn't get chance fix this for 15.8 preview 4. We will try to get this for preview 5.
@genlu we discuss with about message and log level when we are fixing it.

@smadala - we still have a week to fix this in preview 4.

@ManishJayaswal Good to know, I thought 19th June was last checking for M2 approval(Looks like schedule is changed).
@cltshivash @pvlakshm for triage this.

@ManishJayaswal I'm working on this.
@genlu We will show first 60 characters of testcase filter. Is that sounds good?
@ManishJayaswal We can't make this to info level. Because

The default console log level for dotnet test is minimal which won't print info messages.
If user try to run the tests and no tests were discovered then warning makes more sense.

Can we do something on LUT side for not to show obvious warning?

@smadala - we should show this error if no tests are found in the assembly. However if tests are found in the assembly but everything is getting filtered out then this should probably be an info, right?

@ManishJayaswal

we should show this error if no tests are found in the assembly.

I agree with you. Because

See the below message in our PR check,

01:17:23 No test is available in D:\j\workspace\Windows_NT_De---9fe03a42\test\DataCollectors\Microsoft.TestPlatform.Extensions.EventLogCollector.UnitTests\bin\Debug\net451\Microsoft.TestPlatform.Extensions.EventLogCollector.UnitTests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

I don't know since when the tests has not been running from test-assembly Microsoft.TestPlatform.Extensions.EventLogCollector.UnitTests.dll. Here we are silently printing warning(In some cases warning helps in build configurations they fails the build on warnings).

I have see VSTest Task in VSTS CI/CD, we try to run tests for product assemblies if it's name match the test-sources regex. This impact perf.

However we may not afford to do this as this change breaks lot of build pipelines once they update the VS/vstest.console.exe.

However if tests are found in the assembly but everything is getting filtered out then this should probably be an info, right?

No, No tests run on test command means there is something wrong we should report it.
Even we want to do this, Currently TestPlatform don't have knowledge that there is no tests because of filter/some other reason.

How much work would it be for TP to know if no tests were found in assembly or tests were found but all got filtered away. I suspect that initially all tests are getting discovered and then passed through the filter?

Can we do something on LUT side for not to show obvious warning?

Did you think about this?

I have provided enough info why the message level should be warning. Why do you want it to be info other than harmless and meaningful warning message on LUT restart.
Why the LUT sending run request if it expects no tests to run always?

The only things we can do on LUT side is to check every error and block this one which does not seem right. LUT needs to send the request because there may be new tests. Gen can confirm that. Gen is there anything LUT can do to not send requests in such cases?


STAC harvester to harvest from copernicus

copernicus uses the STAC catalogue conventions to advertise their data collections
create the harvester which fetches collection metadata
STAC catalogue is a common cataloguing approach in earth observation repositories such as Copernicus
The copernicus STAC catalogue is available at https://catalogue.dataspace.copernicus.eu/stac
STAC is organised in collections: S2GLC, TERRAAQUA, SENTINEL-5P
For each collection a range of items (usually grid datasets) is available. In theory collections can also be nested in other collections, for now we just assume a single level of collections.
See also soilwise-he/governance#12

we harvest from EEA, which includes copernicus


DOC: Add code examples to jax.scipy.fft.idct and jax.scipy.fft.idctn docs

It looks like some of the outputs are wrong in the docs - can you update? Thanks!

It looks like some of the outputs are wrong in the docs - can you update? Thanks!

Thanks! Updated the outputs in the examples.

Looks good! Last thing is to squash the changes. Thanks!

Thanks! Squashed into single commit.


Update page.tsx

:tada: This PR is included in version 1.3.2 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Crawling CC using the Webgraph

Hi,
I am interested in gathering data from common crawl using the webgraph as a metric for webpage similarity. So far, I have recreated parts of the webgraph and am using the following code to download WET files from the common crawl dataset:
from warcio import ArchiveIterator
import cdx_toolkit
import requests

def get_page_url(webURL):
	cdx = cdx_toolkit.CDXFetcher(source='cc')
	for obj in cdx.iter(webURL, limit=1):
		return wet_builder(obj['filename'])

def wet_builder(page_url):
	warc_url = f'https://commoncrawl.s3.amazonaws.com/{page_url}'
	wet_url = warc_url.replace('/warc/', '/wet/').replace('warc.gz', 'warc.wet.gz')
	r = requests.get(wet_url, stream=True)
	records = ArchiveIterator(r.raw)
	record = next(records)
	record = next(records)
	a = record.content_stream().read()
	retText= a.decode('utf-8')[:1000]
	r.close()
	return retText

However, this code runs slowly and requires each website to be passed in manually. Is there a faster (more native) way to crawl the webgraph and download each webpage's text content? I don't need to go through the entire webgraph, I am just looking to crawl through a subset of it.

This is hard to do efficiently because we have not published an index of the WET files. You're using the cdx index correctly; if there was a WET index, this code would run as fast as just the get_page_url() does now.

Ok! Understandable. Do you think this is the best I can do, or do you have any ideas on how I could speed up the code?

I would download the html from the WARCs, using the index to make it relatively efficient, and then do the text extraction yourself. We use Apache Tika to do that.

To generate the WARC links, I am using the following SQL: https://github.com/commoncrawl/cc-index-table/blob/main/src/sql/examples/cc-index/get-records-home-pages.sql, where instead of the alexa1M sites, I am filling it with the websites that I encounter in the webgraph. However, certain sites do not appear within the common crawl table for CC-MAIN-2023-50. For example: inese-poga.artistwebsites.com does not appear when I try directly searching for it as such:
SELECT *
FROM "ccindex"."ccindex" AS cc
WHERE cc.crawl = 'CC-MAIN-2023-50'
AND cc.url_host_registered_domain = 'inese-poga.artistwebsites.com';

Am I missing something?

For example, the website "com.westgateband" is found in the webgraph but I cannot find it within the common crawl table using the following SQL command:
SELECT *
FROM "ccindex"."ccindex" AS cc
WHERE cc.url_host_name = 'westgateband.com'
  AND cc.subset = 'warc';

The webgraph is usually constructed using 3 crawls, and, there can be nodes in it that we did not crawl. That means that we found a link to that web host, somewhere.

I see. Is there any easy way to figure out which webpages were not crawled without having to query athena with every webpage in the webgraph?

Sorry, no. I do have a plan to have a "host level database" that includes that information, but I have no idea when we'll implement it.

That is okay! I figured out how to get the needed WARC files. Is there a GitHub repo you recommend that converts the Warc files to extracted text?

We use Tika3, but there are many other choices.


Child reconciliation algorithm question

@localvoid, hello! :)
Thank you for taking the time to go into such detail about how kivi's diffing algorithm works in syncChildren. I just had a few questions of clarification on some of the cases that you've outlined in your comments.
Most cases make perfect sense to me, but I had a question about one case that wasn't mentioned.
So in review,
--> [ a d b c ] <--
--> [ a b d c ] <--

Here a and c are the same so we keep drilling into the list both ways. Makes sense. We increment first indices, and decrement last indices, on both lists.
--> [ d b ] <--
--> [ b d ] <--

Here we can flip-flop d and b.  Still makes sense. We do insertBefore trick and swap them.
--> [ e j ] <--
--> [ k r ] <--

Nothing matches, so we escape into special case, and do more complicated things. Still makes sense. Do special increasing subsequence and track if nodes have been moved, etc.
Now, this case... What case does this fall into?
    [ a b ] <--
--> [ b d ]

Is there a special way to handle this? Or does it fall into the above case (where nothing matches) ?
Thanks again :)

This is the edge case when it will use more DOM ops then it is necessary because of prefix/suffix optimization, I've added a note about this case in ivi sources: https://github.com/ivijs/ivi/blob/dd4f3f3c92e1c758f5e3df2e803604bc071ce9cd/src/vdom/implementation.ts#L1651-L1652
It should go through this code path:

Move node "b": https://github.com/localvoid/kivi/blob/26eae1043efb9cc4ba7e54c1c4ab30389a669149/lib/vnode.ts#L1572-L1589
Find all removed/inserted nodes and detect if someone were moved: https://github.com/localvoid/kivi/blob/26eae1043efb9cc4ba7e54c1c4ab30389a669149/lib/vnode.ts#L1628-L1664
Remove node "a": https://github.com/localvoid/kivi/blob/26eae1043efb9cc4ba7e54c1c4ab30389a669149/lib/vnode.ts#L1706-L1713
Insert node "d": https://github.com/localvoid/kivi/blob/26eae1043efb9cc4ba7e54c1c4ab30389a669149/lib/vnode.ts#L1738-L1748

It is possible to slightly optimize this use case when one DOM node is left after prefix/suffix phase, but I am not sure if it is worth it: https://github.com/ivijs/ivi/blob/dd4f3f3c92e1c758f5e3df2e803604bc071ce9cd/src/vdom/implementation.ts#L1767-L1768
ivi has a slightly modified version of this algo. It can handle use cases with implicit positional indexing, when nodes without explicit keys automatically assigned with implicit keys to support code patterns like this <div>{condition ? <ComponentA> : null}<ComponentA></div>. Most vdom libraries doesn't handle such use cases, and instead of removing/inserting first ComponentA when condition is changing, they will remove/insert the second one, and internal state of this components will be incorrect. Here is an example when vue2 loses internal state, because it doesn't support this edge case: https://jsfiddle.net/4ppdfyy8/


Fix IP validation test

:tada: This PR is included in version 1.8.3 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


stricter jsdoc type resolution causes weird behaviour for value-only declarations

/** @param {Image} image */
function process(image) {
}

Expected behavior:

Error "'Image' refers to a value, but is being used as a type here." and image: any, as TS does.
No error, and image: typeof Image, as we often do in JS.

I think I prefer (2), but after the stricter-jsdoc-type-resolution PR, I expected (1), which is also OK.
Actual behavior:
No error and image: any.

@sandersn Might #33575 be related?

@ahocevar Yes, although the scenario still doesn't work completely, so it still needs work.


Provide completion for referred keywords required using :as-alias

Before submitting a PR make sure the following things have been done:

[x] The commits are consistent with our contribution guidelines
[x] You've added tests to cover your change(s)
[x] All tests are passing
[x] The new code is not generating reflection warnings
[x] You've updated the changelog (if adding/changing user-visible functionality)

Thanks!

🍻!
Shall we cut a release, or you having something else on the radar?

Let's cut a release. I've been looking into another issue you've mentioned earlier, but it's a bit tricky and will require a bit more discussion.

Suitable has been cut!
cider-nrepl not (should be later in the week), but I bumped Suitable there, so you can in the meantime perform a local install

Thank you! :beers:


Connecting Trains

Trains which leave a station (to a different destination) within 30 minutes of another train arriving should receive bonus passengers to account for people switching trains.

closed in a88d87a


[POAE7-2487] support arrow date32(days) format in cider

What changes were proposed in this pull request?

Use arrow date32(days) in cider and cast into datetime64(seconds) only in time func
Why are the changes needed?

support arrow data format
Does this PR introduce any user-facing change?

needs to follow arrow data format

Format string
Arrow data type

tdD
date32 [days]

ttu
time64 [microseconds]

tsu
timestamp [microseconds]

How was this patch tested?

UTs
a few tests commented out. TODO:

Non-group by testcase failed
cast between date(day32) and string

Which label does this PR belong to?

@kd97 you need rebase this PR and check the build failure.

please resolve conflicts. BTW, is this pr still WIP? if not please remove it in title.

please resolve conflicts. BTW, is this pr still WIP? if not please remove it in title.

resolved conflicts and fixed failed tests.


consul: improve consul service discovery

Related to #3711

Add the ability to filter by tag and node-meta in an efficient way (/catalog/services
allow filtering by node-meta, and returns a map[string]string or service->tags).
Tags and nore-meta are also used in /catalog/service requests.
Do not require a call to the catalog if services are specified by name. This is important
because on large cluster /catalog/services changes all the time.
Add allow_stale configuration option to do stale reads. Non-stale
reads can be costly, even more when you are doing them to a remote
datacenter with 10k+ targets over WAN (which is common for federation).
Add refresh_interval to minimize the strain on the catalog and on the
service endpoint. This is needed because of that kind of behavior from
consul: https://github.com/hashicorp/consul/issues/3712 and because a catalog
on a large cluster would basically change all the time. No need to discover
targets in 1sec if we scrape them every minute.
Added plenty of unit tests.

Benchmarks
scrape_configs:

- job_name: prometheus
  scrape_interval: 60s
  static_configs:
    - targets: ["<IP_ADDRESS>:9090"]

- job_name: "observability-by-tag"
  scrape_interval: "60s"
  metrics_path: "/metrics"
  consul_sd_configs:
    - server: consul.service.par.consul.prod.crto.in:8500
      tag: marathon-user-observability  # Used in After
      refresh_interval: 30s             # Used in After+delay
  relabel_configs:
    - source_labels: [__meta_consul_tags]
      regex: ^(.*,)?marathon-user-observability(,.*)?$
      action: keep

- job_name: "observability-by-name"
  scrape_interval: "60s"
  metrics_path: "/metrics"
  consul_sd_configs:
    - server: consul.service.par.consul.prod.crto.in:8500
      services:
        - observability-cerebro
        - observability-portal-web

- job_name: "fake-fake-fake"
  scrape_interval: "15s"
  metrics_path: "/metrics"
  consul_sd_configs:
    - server: consul.service.par.consul.prod.crto.in:8500
      services:
        - fake-fake-fake

Note: tested with ~1200 services, ~5000 nodes.

Resource
Empty
Before
After
After + delay

/service-discovery size
5K
85MiB
27k
27k

go_memstats_heap_objects
100k
1M
120k
110k

go_memstats_heap_alloc_bytes
24MB
150MB
28MB
27MB

rate(go_memstats_alloc_bytes_total[5m])
0.2MB/s
28MB/s
2MB/s
0.3MB/s

rate(process_cpu_seconds_total[5m])
0.1%
15%
2%
0.01%

process_open_fds
16
1236
22
22

rate(prometheus_sd_consul_rpc_duration_seconds_count{call="services"}[5m])
~0
1
1
0.03

rate(prometheus_sd_consul_rpc_duration_seconds_count{call="service"}[5m])
0.1
80
0.5
0.5

prometheus_target_sync_length_seconds{quantile="0.9",scrape_job="observability-by-tag"}
N/A
200ms
0.2ms
0.2ms

Network bandwidth
~10kbps
~2.8Mbps
~1.6Mbps
~10kbps

Filtering by tag using relabel_configs uses 100kiB and 23kiB/s per service per job and quite a lot of CPU. Also sends and additional 1Mbps of traffic to consul.
Being a little bit smarter about this reduces the overhead quite a lot.
Limiting the number of /catalog/services queries per second almost removes the overhead of service discovery.

+1

I'm wondering if we could get away with just RefreshInterval, the other settings will be less useful/counter-productive once #2191 is in.

Update the patchs

The config documentation needs updating.

This should be done now. Please tell me if this is still unclear.

Mixing in the core change plus several optimisations on top of it makes this a bit challenging to review.

Sorry for that :/

I'm wondering if we could get away with just RefreshInterval, the other settings will be less useful/counter-productive once #2191 is in.

I guess that would still be an improvement. But I believe exposing the consul filtering capabilities has a real value because there is a non-significant overhead when using relabeling to do the filtering (as seen on the table).

We deployed this patch on our pre-production cluster (~1k services), here are the results:

Considering that Prometheus is far from being the only user of Consul, this is pretty impressive :)

lgtm, @brian-brazil ? I think we can merge before #2191 since I don't know of anyone actively working on the fix for this

I guess that would still be an improvement. But I believe exposing the consul filtering capabilities (currently tag + node-meta, soon service-meta) has a real value because there is a non-significant overhead when using relabeling to do the filtering (as seen on the table).

It'd still be interested in knowing what just the delay would give us. The other settings are going to cause support hassle, and reduce the effectiveness of #2191.

I think we can merge before #2191 since I don't know of anyone actively working on the fix for this

That was never blocking this.

It'd still be interested in knowing what just the delay and allowstale would give us.

Here is the table with the new requested column (Delay) with refresh_interval: 10s
| Resource | Empty | Before | After | After + delay | Delay |
| -------- |:-----:|:------:|:-----:|:-------------:|
|/service-discovery size|5K|85MiB|27k|27k|27k|92MiB|
|go_memstats_heap_objects|100k|1M|120k|110k|1M|
|go_memstats_heap_alloc_bytes|24MB|150MB|28MB|27MB|140MiB|
|rate(go_memstats_alloc_bytes_total[5m])|0.2MB/s|28MB/s|2MB/s|0.3MB/s|23MB/s|
|rate(process_cpu_seconds_total[5m])|0.1%|15%|2%|0.01%|15%|
|process_open_fds|16|1236|22|22|18|
|rate(prometheus_sd_consul_rpc_duration_seconds_count{call="services"}[5m])|~0|1|1|0.03|0.1|
|rate(prometheus_sd_consul_rpc_duration_seconds_count{call="service"}[5m])|0.1|80|0.5|0.5|90|
|prometheus_target_sync_length_seconds{quantile="0.9",scrape_job="observability-by-tag"}|N/A|200ms|0.2ms|0.2ms|180ms|
|Network bandwidth|~10kbps|~2.8Mbps|~1.6Mbps|~10kbps|~2Mbps|
Something weird, process_open_fds is pretty low, but I did have the socket: too many open files before I played with ulimit. This is probably because this happens only on startup and after the the time.Sleep() is preventing all the things to happen at the exact same time.. maybe (or it's a gauge, so we can't know).
The other results are what would be expected: filtering only with relabel_config means that all the information needs to be fetched by the client (prometheus) before filtering, so this is less efficient than doing it server side (even if with the patch tag filtering is partly client-side, but still more efficient).

That result doesn't seem quite consistent with the rest, for example service calls shouldn't have gone up. Can you double check you performed the test correctly?
@brancz This PR contains major changes to Consul SD. Do you think we should hold it for after 2.2 for the sake of stabilising things?

You are right, this seems counter intuitive, but let's look at the numbers
I have 1235 services. Using relabeling I'll watch each of them.
With a refresh_interval or 10s I expect to have 123 calls per second at maximum.
With a static consul cluster, with a WaitTime of 30s, this should be at minimum 41 calls per second.
This means that the expected numbers are:

A refresh_interval: 0s: [41, +inf[ (bounded by latency)
B refresh_interval: 10s: [41, 123[

But you would expect A >> B. Which doesn't seem to be the case here. In facts it looks like there are more calls when refresh_interval is non-zero, which is confirmed by the following trace:
level=debug ts=2018-02-14T14:18:12.817694053Z caller=consul.go:419 component="discovery manager scrape" discovery=consul msg="Watching service" service=observability-portal-web tag=
level=debug ts=2018-02-14T14:18:12.825106702Z caller=consul.go:428 component="discovery manager scrape" discovery=consul msg="Watching service !!done!!" service=observability-portal-web tag= meta.LastIndex=849509831 LastIndex=849509470
level=debug ts=2018-02-14T14:18:22.825509212Z caller=consul.go:419 component="discovery manager scrape" discovery=consul msg="Watching service" service=observability-portal-web tag=
level=debug ts=2018-02-14T14:18:22.826759492Z caller=consul.go:428 component="discovery manager scrape" discovery=consul msg="Watching service !!done!!" service=observability-portal-web tag= meta.LastIndex=849510183 LastIndex=849509831
level=debug ts=2018-02-14T14:18:32.827078498Z caller=consul.go:419 component="discovery manager scrape" discovery=consul msg="Watching service" service=observability-portal-web tag=
level=debug ts=2018-02-14T14:18:32.828510762Z caller=consul.go:428 component="discovery manager scrape" discovery=consul msg="Watching service !!done!!" service=observability-portal-web tag= meta.LastIndex=849510481 LastIndex=849510183
level=debug ts=2018-02-14T14:18:42.828818192Z caller=consul.go:419 component="discovery manager scrape" discovery=consul msg="Watching service" service=observability-portal-web tag=
level=debug ts=2018-02-14T14:18:42.829889865Z caller=consul.go:428 component="discovery manager scrape" discovery=consul msg="Watching service !!done!!" service=observability-portal-web tag= meta.LastIndex=849510787 LastIndex=849510481
level=debug ts=2018-02-14T14:18:52.830123327Z caller=consul.go:419 component="discovery manager scrape" discovery=consul msg="Watching service" service=observability-portal-web tag=
level=debug ts=2018-02-14T14:18:52.831354479Z caller=consul.go:428 component="discovery manager scrape" discovery=consul msg="Watching service !!done!!" service=observability-portal-web tag= meta.LastIndex=849511144 LastIndex=849510787
level=debug ts=2018-02-14T14:19:02.831627833Z caller=consul.go:419 component="discovery manager scrape" discovery=consul msg="Watching service" service=observability-portal-web tag=
level=debug ts=2018-02-14T14:19:02.832902286Z caller=consul.go:428 component="discovery manager scrape" discovery=consul msg="Watching service !!done!!" service=observability-portal-web tag= meta.LastIndex=849511497 LastIndex=849511144

This happens as soon as the refresh_interval is bigger than 2sec, meaning that it effectively becomes a regular refresh interval. I'm trying to dig into consul's code to understand what's happening.

I can confirm this behavior using the following script:
#!/bin/bash
if test ${1:-none} = "none"
then
  echo "USAGE: $0 Consul_URL"
  echo "       Example: localhost:8500/v1/health/service/MY_SUPER_SERVICE"
  exit 1
fi
url_to_check=$1

headers=$(mktemp)
content=$(mktemp)
index=0
while true;
do
  url="${url_to_check}?wait=30s&index=${index}&pretty=true&stale"
  curl -fs --dump-header "$headers" -o "${content}.new" "${url}" || { "echo Failed to query ${url}"; exit 1; }
  sleep 5
  if test $index -ne 0
  then
    diff -u "$content" "$content.new" && echo " diff: No Differences found in service"
  fi
  index=$(grep "X-Consul-Index" "$headers" | sed 's/[^0-9]*\([0-9][0-9]*\)[^0-9]*/\1/g')
  mv "$content.new" "$content"
  printf "X-Consul-Index: $index at $(date) \n"
done

the output is
$ ./consul.sh consul05-par.central.criteo.prod:8500/v1/catalog/service/graphite-relay
X-Consul-Index: 849580974 at Wed Feb 14 15:55:16 CET 2018
 diff: No Differences found in service
X-Consul-Index: 849581128 at Wed Feb 14 15:55:21 CET 2018
 diff: No Differences found in service
X-Consul-Index: 849581289 at Wed Feb 14 15:55:26 CET 2018
 diff: No Differences found in service
X-Consul-Index: 849581439 at Wed Feb 14 15:55:31 CET 2018
 diff: No Differences found in service
X-Consul-Index: 849581603 at Wed Feb 14 15:55:36 CET 2018
 diff: No Differences found in service
X-Consul-Index: 849581775 at Wed Feb 14 15:55:41 CET 2018
 diff: No Differences found in service
X-Consul-Index: 849581926 at Wed Feb 14 15:55:46 CET 2018

In other words, waiting when reusing an index makes it ineffective.
I'm not sure what to do here, if we don't change the current patch: refresh_interval is effectively a refresh interval as with current versions of consul this disables blocking queries.
It's all fine for users, except if that a low refresh_interval will be less efficient than none. Maybe this is the solution:
diff --git a/discovery/consul/consul.go b/discovery/consul/consul.go
index 80f756c6..d2a1855a 100644
--- a/discovery/consul/consul.go
+++ b/discovery/consul/consul.go
@@ -84,7 +84,7 @@ var (
                Scheme:          "http",
                Server:          "localhost:8500",
                AllowStale:      true,
-               RefreshInterval: model.Duration(15 * time.Second),
+               RefreshInterval: model.Duration(watchTimeout),
        }
 )

@@ -136,6 +136,10 @@ func (c *SDConfig) UnmarshalYAML(unmarshal func(interface{}) error) error {
        if strings.TrimSpace(c.Server) == "" {
                return fmt.Errorf("Consul SD configuration requires a server address")
        }
+       interval := time.Duration(c.RefreshInterval)
+       if interval > 0 && interval < watchTimeout {
+               return fmt.Errorf("refresh_interval (%v) can't be lower than %v", interval, watchTimeout)
+       }
        return nil
 }

The behaviour of indexes updated while no change happens seems to be linked to this behaviour of Consul: https://github.com/hashicorp/consul/issues/3712
The fact that being notified of change even when no real change occurs seems to be in the contract.
Having a sleep min delay is thus a good idea

Ok, pushed https://github.com/prometheus/prometheus/pull/3814/commits/3e9ef40136a48b5ba493c1352c0a5a6e32358a19 which I think does the correct thing:
refresh_interval now does what is advertised in the documentation, there won't be more that one update per refresh_interval. It now defaults to 30s (which was also the current waitTime in the consul query).
This also make sure we don't wait another 30s if we already waited 29s in the blocking call by substracting the number of elapsed seconds.
Hopefully this will do what people expect it does and will be safer for existing consul infrastructures.
I also updated the table with the results of setting refresh_interval: 60s (instead of 10s, which was too short). As expected memory stays more or less the same and the rest is divided by 6.

Rebased on master now that 2.2.1 is out

I'm waiting to make sure 2.2.1 is stable before merging.

rebased, using tickers.
circleci seems to fail on unrelated tests.

Thanks!


Edit alert thresholds

UI for editing the thresholds which will trigger alerts.
The UI should enable the professional to add(and remove) thresholds (in a table).
Each threshold:

selector for the type of observation
selector for type of threshold (above, below, more or equal, less or equal, equal, not equal)
value of threshold

This task should also store/share the thresholds in the appropriate wallets.

@raihan2006i we need to include this in the data model.


Added ability to pass custom labels to containers (fixes #350)

Codecov Report

Merging #351 into master will increase coverage by 0.46%.
The diff coverage is 100%.

@@            Coverage Diff            @@
##           master    #351      +/-   ##
=========================================
+ Coverage   74.34%   74.8%   +0.46%
=========================================
  Files           9       9
  Lines        1029    1048      +19
=========================================
+ Hits          765     784      +19
  Misses        216     216
  Partials       48      48

Flag
Coverage Δ

#unittests
74.8% <100%> (+0.46%)
:arrow_up:

Impacted Files
Coverage Δ

session/session.go
100% <ø> (ø)
:arrow_up:

config/config.go
90% <ø> (ø)
:arrow_up:

service/docker.go
82% <100%> (+1.48%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f262404...a9a7955. Read the comment docs.


Bundle size reductions?

Hi! Is there any way to reduce the size of markdown-it in some conditions? I know there are ways to disable plugins etc, but this only seems to prevent their runtime execution, not their inclusion. It's sort of a shame that the library is so monolithic (vs entities etc being a separate plugin?)
See<EMAIL_ADDRESS>vs<EMAIL_ADDRESS>(I know marked is not an ideal example as it is half-dead, but markdown-it minified + gzip is 4x the size)
It would be great, long term, if we could do something like
import { renderer, entities } from "markdown-it";

const markdownRenderer = renderer();

markdownRenderer.use(entities);

The idea being that the core minimal renderer logic can still be applied, and then other parts are pulled in conditionally. I don't suppose there is any roadmap or possibility for this type of composition?

There are no good ways to reduce size without loosing important features.

Why would optionally reducing size lose features? Sorry, not sure I follow. It's sort of exactly what I'm suggesting; let consumers choose their feature coverage by manual inclusion, rather than bundling it all in one monolith.

I'm very sceptical about suggestion to optimize something when no metrics provided. The same about imaginated ideal modular design. If you wish to spend your time to search such solution - no problem, but i can not participate.

Sure, it was more of an open ended question; the metrics were the bundle sizes above, which make this a troublesome library to use in the browser. There are a handful of similar issues that have been opened and subsequently closed as well, so I wondered if there were any plans.
If it doesn't interest you, then thats cool also, its your project, and I can look elsewhere. Thanks for your time :)

There are a handful of similar issues that have been opened and subsequently closed as well, so I wondered if there were any plans.

There are no plans from my side to spend significant time for changes. Everything doable with small spends is already done.
If someone decide to invest his own time - PR will be reviewed with care.
All postponed ideas are recorded in issue 125.


fix formatting on pypi

https://pypi.org/project/graspy/

Will be automatically fixed when 0.0.3 is released.
https://github.com/neurodata/graspy/blob/master/setup.py#L42

thx


Added Submariner Install

What type of PR is this?

What this PR does / why we need it:
Which issue(s) this PR fixes:
Fixes #21
Special notes for your reviewer:
Does this PR introduce a user-facing change?:

/retest

by default, a limitation of Submariner is that it doesn’t handle overlapping CIDRs (ServiceCIDR and ClusterCIDR) across clusters. If the CIDRs of two or more clusters overlapping, How do we handle it, quit the installation or install the Global Private Network component of Submariner?

Good question, the solution can only handle none overlapping scenario. We can handle this later