Add Common Table Expressions (a.k.a. CTE or WITH)

I know it's too late for 3.0-rc but anyway :)
This enables Common Table Expressions. There are two common use cases for them:

Recursive queries for retrieving tree structures (e.g. category tree or virtual file tree).
Reusable subqueries. Useful for scoping.

API may be disputable because it's more evident to use with as in plain SQL instead of cte but it may be a mess with Kernel.SpecialForms.with/1. I'm open for proposals.
Complementary PR in ecto_sql: https://github.com/elixir-ecto/ecto_sql/pull/23

Thanks @feymartynov for another great pull request!
I have added some comments more specific to the code but, before addressing them, I think we should discuss the feature in general. Non-recursive CTEs do not provide anything meaningful to Ecto, because we are capable of expressing something similar using Ecto's query syntax. So I think the focus here should be on recursive ones. I would even go to the point of supporting only recursive (and maybe call the feature recursive).
The other thing is that, for recursive CTEs, we can actually pass "arguments" to the name, which is something I did not see explored here.
What are your thoughts?

I think non recursive CTEs are still important for performance reasons at least. If you have reason to join to a single CTE more than once, having that is better than joining to a duplicated subquery multiple times. I could be wrong though, Postgres might know what to do with repeating subqueries

I agree with @zachdaniel. In my project for example I use them for scoping soft delete and data available for current user and there are cases when I join those scopes more than once.

The other thing is that, for recursive CTEs, we can actually pass "arguments" to the name, which is something I did not see explored here.

If I understand the standard correctly those "arguments" are just aliases for selected fields. We can specify them inside the query itself like I did in the test case so they don't look necessary. But if we want to stick with the standard I can add them but I need some insight about how to pass them with keyword syntax.

If I understand the standard correctly those "arguments" are just aliases
for selected field

The question is: is this a syntactic behavior or does it change the
semantics too?
José Valimwww.plataformatec.com.br
http://www.plataformatec.com.br/Founder and Director of R&D

If so I believe we can use the keyword syntax. Assuming we need both recursive CTE and non-recursive ones, then there are two thing left:

Figure out the name: I guess cte and recursive_cte are good enough?
Figure out how to do the recursive bit. It seems that, once you set one single entry to recursive, it affects all of the other ctes, so this is not quite composable, so I am not quite sure how to handle this

Thoughts?

I agree with #1 but need to some more research on #2

yeah, after looking into it I think the only option is to have cte: ... and then a separate recursive: ...  for composability. We can either take the latest declaration for recursive, or allow only one to be set and raise an error if it’s overridden.

I have been thinking about this more and, the issue with how it is implemented today, is that we lose the Ecto type information. Before I make a different proposal, are the CTE requires to appear as the source of a FROM or JOIN in SQL?

Yeah, I think we could support argument-less for now, and just do with_recursive: true. And support arguments via some other value applied to that config.  As for your question, you can use a CTE in a from/join further on in the query.

As for your question, you can use a CTE in a from/join further on in the query.

Right, but only then? Or can I use them elsewhere?
Because I am thinking that maybe we should implement this like we implement subqueries:
from p in cte(query), ...

This would allow us to leverage the type checking/binding mechanism that exists for subqueries. The only difference is that we move those out into a WITH when we build the query.

Regarding the argument, queries like this does not work without them

In MySQL it really doesn't, in PostgreSQL it's ok.

Yeah, I think we could support argument-less for now, and just do with_recursive: true.

I agree.

Right, but only then? Or can I use them elsewhere?

you can use a CTE in a from/join further on in the query.

Also in other CTEs and in all parts of UNION.

from p in cte(query), ...

This doesn't allow CTE to be used in the second part of UNION. However current implementation doesn't allow it too. I think we need a test case for that.

This doesn't allow CTE to be used in the second part of UNION. However current implementation doesn't allow it too. I think we need a test case for that.

Good point. We will need a way to figure out how bindings will work then so we can keep the type information.
Something like from f in "foo", cte: [foo: Post] should return Post structs correctly.

So what about the API? Is this ok?
Product
|> with_recursive(true)
|> cte(category_tree_query, as: "category_tree")
|> join(:inner, [p], c in "category_tree", c.id == p.category_id)

from p in Product,
  with_recursive: true,
  cte: category_tree_query,
  as: "category_tree",
  join: c in "category_tree",
  on: c.id == p.category_id

I still think we need to find a way to make the Ecto.Types work, as that is very important to how Ecto works. We could do it based on strings, but it would be too brittle.
The approach I proposed (i.e. join: c in cte(foo)) has some flaws, but maybe we can find something that mixes both together? Unsure.

I think in that context that we really just ought to make adding a cte introduce a binding the same way anything else would>
Product
|> with_recursive(true)
|> cte(category_tree in category_tree_query, as: :category_tree)
|> join(:inner, [p], c in category_tree, c.id == p.category_id)

from p in Product,
  with_recursive: true,
  cte: category_tree in category_tree_query,
  as: :category_tree,
  join: c in category_tree,
  on: c.id == p.category_id

Now, in the case that using a binding in the query also as a source in another part of the query causes insurmountable problems (something that seems likely), I think we can use the cte(foo) as a way to like...treat that as its own source. Basically the binding can be used as if it was something previously joined on, which also ought to work with how CTEs work in general, and then if used as a source it is wrapped in cte(foo). So it would work like this:
Product
|> with_recursive(true)
|> cte(category_tree in category_tree_query, as: :category_tree)
|> join(:inner, [p], c in cte(category_tree), c.id == p.category_id)

from p in Product,
  with_recursive: true,
  cte: category_tree in category_tree_query,
  as: :category_tree,
  join: c in cte(category_tree),
  on: c.id == p.category_id

So basically we just store each CTE in the query, and when its referenced as a source we use the type information from that stored query.

Thanks Zach! I have a hunch that your approach is more flexible than the
one I proposed but I can’t come up with a concrete example. Can you provide
some? We still cannot use the CTE in an union in both approaches though.
José Valimwww.plataformatec.com.br
http://www.plataformatec.com.br/Founder and Director of R&D

I really just pulled this off of the top of my head, but something like this where you might have two sources from the same CTE and one source joining on that CTE might be a good example
WITH active_users AS (
  SELECT * FROM users WHERE deleted_at IS TRUE
)
SELECT
  active_users.id,
  COUNT(employees.id) AS employee_count,
  COUNT(children.id) AS children_count,
FROM active_users
JOIN employees
ON employees.manager_id = active_users.id
JOIN active_users AS active_user_children
ON active_users.child IS TRUE
AND active_users_children.parent_id = active_users.id

I'm in the weird spot where I don't have a good dataset/database to actually build a useful query from lol

NOTE: I wrote all of this out, but I realized afterwards that it doesn't actually have an example in it of using the CTE just as a binding and not as a source. I'm leaving it here in case its worth talking about though. Skip to the bottom for the important part:
Fun fact, that actually highlights a small issue which is that you might want to use a CTE as the source of the query itself... Regardless, lets ignore that for now and do it this way instead:
WITH active_users AS (
  SELECT * FROM users WHERE deleted_at IS TRUE
)
SELECT
  active_users.id,
  COUNT(employees.id) AS employee_count,
  COUNT(children.id) AS children_count,
FROM users
JOIN employees
ON employees.manager_id = active_users.id
JOIN active_users
ON active_users.id =
JOIN active_users AS active_user_children
ON active_user_children.child IS TRUE
AND active_users_children.parent_id = active_users.id

The ecto version of that would look like:
active_users = from user in User, where: is_nil(user.deleted_at)

from user in User,
  cte: au in active_users,
  join: employee in cte(au),
  on: employee.manager_id == employee.id
  join: active_user in cte(au),
  on: active_user.id == user.id,
  join: children in cte(au),
  on: children.child == true,
  on: children.parent_id == user.id

Important part:
The above doesn't have an actual example of using it in a binding. In reality, the way I see it being used in a binding is with the in operator, since you can say something like WHERE thing IN cte in sql and that would be very useful.  aside from that, I think we might not need it available except for as a source to from as you had originally stated.

Thanks @zachdaniel!
Reading your reply made me realize that @feymartynov was correct all along. If we provide his syntax and then we want it to also propagate the types upstream, we can just bind it again:
    from c in {"category_tree", CategoryTree}, # <---- this syntax! it should work just™
      with_recursive: category_tree_query,
      as: "category_tree",
      left_join: p in assoc(c, :products),
      group_by: c.id,
      select: %{c | products_count: count(p.id)}

So assuming we are happy with this, I would go with the original proposal of moving to a keyword list and the later suggestion of handling the recursion as a separate operation. So:
    from c in {"category_tree", CategoryTree}, # <---- this syntax! it should work just™
      ctes: [category_tree: category_tree_query],
      left_join: p in assoc(c, :products),
      group_by: c.id,
      select: %{c | products_count: count(p.id)}

The only big remaining question is how to handle arguments in the CTE.

I do not think that including this is good idea as IMHO. The reason is that CTE in Postgres have slightly different semantics form any other DB:

PostgreSQL CTEs are recursive by default (what is already stated there)
PostgreSQL CTEs are optimisation fences, so WITH a AS (SELECT * FROM foos) SELECT * FROM a WHERE a.id = 1 will fetch all rows from foo and then do filtering
PostgreSQL CTEs allow DML in them (that is the reason why these are optimisation fence)

So in general in 99.9% of the cases it is desirable to use subqueries instead which Ecto supports quite well. In that singular cases when CTEs would be needed it is IMHO better to use view or just raw SQL.

@hauleth Raw SQL is fine until you have to compose queries. However I agree about performance. For example @zachdaniel's query above in real app most likely would be slow because at first PostgreSQL would do seq scan over the whole users table and then would be unable to do hash join. But there are many other ways of writing slow queries with Ecto, it's not the problem of the tool :)

@josevalim The only problem with doing a single argument of ctes: [foo: bar] is that later CTEs can reference earlier ones. So as long as we can specify that key multiple times, and/or we break it out into multiple cte:  flags, you'd lose that capability.

@hauleth I agree that it is often not the most performant, but I've also had many experiences where having CTEs available would have been my savior, it was just too long ago to recall the exact use case.

Here one more synthetic example with UNION:
WITH a AS (
  SELECT * FROM addresses WHERE deleted_at IS NULL
)

SELECT c.name, a.*

FROM providers AS p
LEFT JOIN a
ON a.id = p.address_id

With providing type information through tuples like @josevalim mentioned above it colud be like this:
from c in Customer,
  cte: addresses_query,
  as: :adresses,
  left_join: a in :addresses,
  on: a.id == c.address_id,
  select: [c.name, {a, Address}],
  union_all: (
    from p in Provider,
      cte: a in cte(addresses_query)
      left_join: a in :addresses,
      on: a.id == p.address_id,
      select: [p.name, {a, Address}]
  )

Coming back to column names list we could optionally provide them like this:
as: {:addresses, [:city, :street, :house]}

For me it still "feels unnatural". It would be better to have something like:
with foos: (from foo in Foo), bazs: (from baz in Baz) do
  from foo in foos, inner_join: baz from bazs, on: foo.id == baz.foo_id
end

Or similar as it more directly match SQL syntax and syntax proposed there can be sometimes confusing, as often (at least for me) the data returned from CTE query has (almost) nothing to do with the selected tables. Especially as I do not quite understand how it works, the top level from is the first entry in the CTE or it is the CTE query itself? If first then it can get enormously confusing, if last then how do you make query from tables introduced in CTE? For example the CTE from my application (live and running):
WITH RECURSIVE
  t AS (
    SELECT l.*

    FROM lab_tests l
    LEFT JOIN packages_to_tests pt ON pt.test_id = l.id
    INNER JOIN t ON t.id = pt.package_id OR t.id = l.parent_id
  )
SELECT * FROM t

How it would work in Ecto? Could you give me query example of such SQL? AFAIK there is no way to reference outer query in subquery.

Here are my two cents:

Let's keep this it out of the keyword query syntax. Since CTEs happen before the from (i.e. select, update, delete), it doesn't have a good match with the current DSL.

I would implement it as a simple macro, called with_cte:

query
|> with_cte("name_of_cte", as: query)
|> with_cte("this_needs_to_be_a_string_at_compile_time(foo, bar)", as: query)

We can also add a recursive_ctes/1 function (or recursive_with/1) that simply receives a query and marks it as recursive.
Thoughts?

@josevalim what would be the query here? If the actual CTE query, then the order should be reversed IMHO.

Sorry, that was a horrible example as I used query everywhere. I have updated it. Please check again.

That still leave ambiguity. Should I understand that with your syntax my query would look like:
query = from row in "t"

query
|> with_recursive_cte(
  "t",
  from(test in LabTests,
    where: test.id == ^id,
    union: (
      from test in LabTest,
      left_join: package in "packages_to_tests", on: package.test_id == test.id,
      inner_join: sub_test in "t", on: sub_test.id == test.parent_id or sub_test.id == package.package_id))

It kind of defeats the main selling point of CTEs which is the fact that you can read them from top to bottom. Ecto does quite nice job of being more like QUEL rather than SQL (which is good thing) and this syntax makes it a little bit weird again.

I am not sure it matters this much because it is unlikely you will build
the whole query at once anyway.
Plus you can always do this:
foo_cte = from ...
bar_cte = from ...
query = from ...
# wire it all up

I think trying to tie with and from somehow will just make it look weird on
the elixir side of things.
José Valimwww.plataformatec.com.br
http://www.plataformatec.com.br/Founder and Director of R&D

@josevalim even in that case it will make it weird to read
sub_tests =
    from test in LabTest, # 4
    left_join: package in "packages_to_tests",
    on: package.test_id == test.id,
    inner_join: sub_test in "t", # 5
    on: sub_test.id == test.parent_id or sub_test.id == package.package_id
test =
    from test in LabTest, # 2
    where: test.id == ^id,
    union: sub_tests # 3
query = from row in {"t", LabTest} # 6

query
|> with_recursive_cte("t", test) # 1

Where with numbers in comments I have marked order of reading to understand all query. Don't you think it looks weird? Like hopping back and forth constantly until anything of it make any sense? In such situation I think that writing raw SQL query is clearer solution, even when I need to parse results like:
results.rows
|> Enum.map(&Repo.load(LabTest, {results.columns, &1}))
|> Repo.preload([:parent])

If you think SQL is clearer in those cases then it is definitely your
choice to make. :)
I think the currently proposed API is the simplest one possible. Which
other alternatives are there?
José Valimwww.plataformatec.com.br
http://www.plataformatec.com.br/Founder and Director of R&D

@josevalim
with_cte t <- (
    from test in LabTest, where: test.id == ^id, union: (
      from test in LabTest, # 4
      left_join: package in "packages_to_tests",
      on: package.test_id == test.id,
      inner_join: sub_test in "t", # 5
      on: sub_test.id == test.parent_id or sub_test.id == package.package_id
  ),
  from: row in {t, LabTest}

Or look at it like this: we need to support the pipe based syntax anyway.
So let’s start with that, since it is simpler, and we can develop a query
based one later on.
José Valimwww.plataformatec.com.br
http://www.plataformatec.com.br/Founder and Director of R&D

But no matter what, even with my proposal, it is hard to provide support for all functionalities, because (as I said) PostgreSQL allows DML in CTEs (following SQL standard IIRC), so this is perfectly valid CTE:
WITH
foo AS (
  INSERT INTO foos (id, val) VALUES (1, 10)
)
SELECT * FROM foos

And if I understand this proposal correctly there is no way to support such query.

The way to support those would be as:
with_cte query, “foo”, as: fragment(“...”)

--
José Valimwww.plataformatec.com.br
http://www.plataformatec.com.br/Founder and Director of R&D

And I do not think that we should support all syntax via pipe form as not all SQL is meant to be written in such form. WITH is perfect (only?) example of that. IMHO good test is to rewrite this query using such syntax and check if this is still (quite) readable:
WITH RECURSIVE q(r, i, rx, ix, g) AS (
  SELECT r::DOUBLE PRECISION * 0.02, i::DOUBLE PRECISION * 0.02,
        .0::DOUBLE PRECISION      , .0::DOUBLE PRECISION, 0
  FROM generate_series(-60, 20) r, generate_series(-50, 50) i
  UNION ALL
  SELECT r, i, CASE WHEN abs(rx * rx + ix * ix) <= 2 THEN rx * rx - ix * ix END + r,
               CASE WHEN abs(rx * rx + ix * ix) <= 2 THEN 2 * rx * ix END + i, g + 1
  FROM q
  WHERE rx IS NOT NULL AND g < 99
)
SELECT array_to_string(array_agg(s ORDER BY r), '')
FROM (
  SELECT i, r, substring(' .:-=+*#%@', max(g) / 10 + 1, 1) s
  FROM q
  GROUP BY i, r
) q
GROUP BY i
ORDER BY i

This is Mandelbrot set written as PostgreSQL CTE and IMHO it is quite readable example, but quite complex.
So in my opinion it is better to not provide functionality at all instead of providing it in "not fairly usable way, because we want to fix it later".
This isn't simple task, and there will be more to come (like pivoting/unpivoting, temporal tables, row pattern matching, table samples, grouping sets, and stored procedures support). The question is how much we want to squeeze into Ecto and how much we can support from outside, because with every feature we will implement there will be more to come (especially if we want to also suport NoSQL DBs and/or Column-oriented DBs with SQL-like syntax). While CTEs are nice thing to support, I think it is exactly the feature we shouldn't support via pipe syntax. While pipes can be used for a lot of SQL, the CTEs aren't the thing as it will bring more confusion rather than a solutions (as this will be only thing in Ecto queries which needs to be read "upside down" without fragments).

So in my opinion it is better to not provide functionality at all instead
of providing it in "not fairly usable way, because we want to fix it later".

I don’t see how supporting a syntax we will have to support anyway fits
this description.
Also, you were advocating to support only strings as a starting point for
range/rows over windows. How is this any different?
José Valimwww.plataformatec.com.br
http://www.plataformatec.com.br/Founder and Director of R&D

Anyway, we don’t have to support this. Ecto 3.0 is meant to be stable API
though. So if we don’t get it now, don’t expect it later either.
José Valimwww.plataformatec.com.br
http://www.plataformatec.com.br/Founder and Director of R&D

@hauleth I agree with @jose. It is a simple trade off: composability and compartmentalizations of concerns, in exchange for not being able to read the query from top to bottom as if it were sql.

Just a question, if the pipe API si so simple then why Elixir itself decided to use with instead of "new kind" of pipe like for example ~>? I know the answer: flexibility. For me this is even more obvious case as there we can alias items created in WITH statement, so whole query could end very confusing to read, at least for me.

That's the point, isn't it? The with is an abstraction of an existing pattern. You can express what happens in Elixir's with today, just using function calls, and the regular pipe syntax. It is your choice to make between with and regular pipe plus pattern matching.
But for some reason you are saying that the only way to write this code is via a complex macro, which is what I disagree with. That would be the equivalent to say Elixir should remove pattern matching in function clauses now that we introduced with because with is more flexible.
Btw, the reason why with only arrived in Elixir v1.3 is because it took a while for us to figure out how it should look like. And since you have made this comparison, I would go ahead and say that we shouldn't rush it here either. In fact, shipping with a simple API will allow people to experiment with different syntaxes and approaches, which is exactly what happened before we added with to the language.
It is fine for us to disagree here. :) But I have made my mind that we will start with the simple API for now.

Also in DBs without WITH should we emulate them by using subqueries or throw an error about unsupported features? IIRC this is how MSSQL treats WITH, I do not know how about other DBs though.

In Ecto we generally don't try to "rewrite queries". So if WITH can be emulated with something else that is not a WITH, we typically don't do it, and just ask the adapters to raise. Some complex combinations, like using WINDOW + LOCK raising an error in Postgres, we don't check for those and we let the DB do the work.

@feymartynov I will open up a new issue with an API proposal for CTE later today. Then you can either submit a new pull request or change this one. Thanks everyone for the discussion.

@josevalim what is so complex in my macro? For me it is quite simple to create with given PR as a different syntax. Both could be easily implemented alongside each other, however the problem is that I consider "pipe" syntax plainly harmful as very easy to abuse such syntax. It is even more troublesome than named joins as you cannot check if the queried table exists until you fail at DB boundary. IMHO it is very prone to the typos. As I said, the main concern for me is that you can query table that do not exist yet, and it will be created later in code. So this can be quite confusing for the users.

Your syntax is complex for the user, because it is something new to learn. Maybe it is the best macro-based approach but I don't want to decide it on a whim. On the other hand, everyone knows how function calls work. Again, your thoughts on the pipe syntax have been clear, and we will have to agree to disagree. We will go ahead with the call syntax for now, which you may pipe to it or not.

But that is my point. Everybody knows how function calls work, so this:
foo(bar)
bar = 5

Is expected to fail, so why this:
from foo in :foo,
  cte: bar,
  as: :foo

Should work?
For me it seems counterintuitive.

Thats late binding and it is exactly what a higher level construct allow us
to do. For example, Ecto.Query.dynamic works like that too.
I agree with you that writing the straight-forward way would be nice too,
but there are still things to figure out there, so the late binding
approach, which we will have anyway, will suffice for now.
José Valimwww.plataformatec.com.br
http://www.plataformatec.com.br/Founder and Director of R&D

Should work?

Just as you referenced let/let* above, think of this like a Haskell where clause:
f x = y
  where
    y = z...

@OvermindDL1 however Haskell where do not bleed to the caller while cte does. So there is possibility that change in one function can cause other function to query wrong table. In Haskell it is not a case.

For example:
from post in query,
  cte: cte_query, as: :users

And somewhere else down (or up) the pipeline I would do:
from post in query,
  inner_join: user in assoc(post, :author)

Where schemas looks like:
defmodule Post do
  use Ecto.Schema

  schema "posts" do
    belongs_to :author, User
  end
end

defmodule User do
  use Ecto.Schema

  schema "users" do
    has_many :posts, Posts, foreign_key: :author_id
  end
end

And now we can get quite confusing error about missing columns in table users when cte_query do not return all fields that User schema needs.

I will open up a new issue with an API proposal for CTE later today. Then you can either submit a new pull request or change this one. Thanks everyone for the discussion.

Thank you, that would be nice because I'm kind of lost in this variety of opinions :)

#2769

I have updated this PR and opened a new one to ecto_sql: https://github.com/elixir-ecto/ecto_sql/pull/25
Please, check 'em out.

Thank you @feymartynov! this looks great, i have added some comments but they are minor. The only important bit is that we need to add tests for the parameter handler and make sure we recur query expressions in the planner.

You are correct. And you also implemented merge_cache, so we should be
good. Apologies!
José Valimwww.plataformatec.com.br
http://www.plataformatec.com.br/Founder and Director of R&D

Updated the code.
Tests for parameter interpolation are in the ecto_sql's PR.

Thanks @feymartynov! Now that ecto_sql is in a separate repository, it would be nice if we assert the parameters on Ecto side of things. The planner suite already checks for that, can you please update the planner tests then? I have recently added the union parameters there, so if you can rebase and increment, it would be perfect: 8c1662aed1f822f8881c296f8ba2767cb0b7b9cf

@josevalim please check out the update.

Hi @feymartynov! Right now we are focused on patch releases for Ecto. Once that is out of the way we will focus on merging non-maintenance work. :) Thanks!

close

Hey, sorry to be a nuisance, but any idea on when this could be merged? CTEs would allow so many easy solutions for current problems!

Hi @icehaunter, it is no bother at all. There are some concerns about the limited approach in this PR (only pipe syntax) but more importantly, we are mostly focused on bug fixes since v3.0.

Hi @feymartynov. Sorry for the delay on this but things have finally calmed down in the Ecto front. I have dropped some final comments. Then we only need to rebase and we should be good to go. Thank you!

Hello @josevalim! I'll do it on the weekend.

@josevalim Updated the PR.

Merged manually, thank you!
:heart: :green_heart: :blue_heart: :yellow_heart: :purple_heart:

For the record: commit is ec47bb8483dd5206bd02b68c69c15eacf744be07

Just for the record - I still think that this is bad API and "following SQL/QUEL syntax" would be better choice.


*: rename serialization.md -> config.md

Closes: https://github.com/opencontainers/image-spec/issues/283
Signed-off-by: Vincent Batts<EMAIL_ADDRESS>

LGTM
Thanks!

@vbatts Looks like there are some whitespace problems.

fixed. Not sure why newer git is not detecting these. It's a bit frustrating. :-\

LGTM, thank you

LGTM


LEGO: Pull request from lego/hb_04604851-bac4-4681-9f74-73de611d6e48_20220530163305847 to main

LEGO: Pull request from lego/hb_04604851-bac4-4681-9f74-73de611d6e48_20220530163305847 to main with localized lcls

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
4 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-0.01%) to 41.719%

Files with Coverage Reduction
New Missed Lines
%

extensions/notebook/src/jupyter/serverInstance.ts
4
75.13%

Totals

Change from base Build<PHONE_NUMBER>:
-0.01%

Covered Lines:
27741

Relevant Lines:
62042

💛  - Coveralls


Update questions.rst

typo corrected

Coverage remained the same at 82.13% when pulling bd9360b9118fbcba6f855e84191d2b0cdfd2d202 on skelligan:patch-1 into 23a46e7e1e6221e643551607c67f22ec0f0dfb86 on rdmorganiser:master.

Coverage remained the same at 82.13% when pulling bd9360b9118fbcba6f855e84191d2b0cdfd2d202 on skelligan:patch-1 into 23a46e7e1e6221e643551607c67f22ec0f0dfb86 on rdmorganiser:master.

Thanks!


fsaf

fdsadfsdfaf

@dagexiongfeng - Is there any query?

We are closing the issue. Please feel free to reopen if required.

It's a invalid issue, so closing it.


[BUG] inconsistent results of ImageBuf colorconvert in built without OCIO and with OCIO.

Describe the bug
Inconsistent results of ImageBuf colorconvert() when OpenImageIO built without OpenColorIO and with OpenColorIO.
Quite common case when oiiotools usually building in vcpkg in a minimal configuration openimageio[core,oiiotools].
Or with OCIO support openimageio[core,oiiotools,opencolorio].
As result any example from OIIO documentation give a different resul when OCIO enabled or disabled.
That happen because default color space for OCIO is ACES-AP1 (ACEScg) and no "linear" color space defined.
Libraw plugin in same time use "standard" naming for sRGB, linear, aces, etc.
And "linear" color space are alias for sRGB LInear.
Issue i feel is more complicated due to OCIO forcing to use it own conventions and ignoring the facts that a lot of work spaces still understand sRGB as default color space and Linear is a "sRGB Linear". So any fix will be just a patch to fix OCIO issue when actual OIIO code is correct.
To Reproduce
oiiotool photo.jpg --colorconvert sRGB linear -o output.exr
depend on OCIO result will be a sRGB Linear or ACEScg exr file.
Same in other direction.
oiiotool photo.jpg --colorconvert linear sRGB -o output.exr
with OCIO input will be treat as ACEScg and sRGB result will be incorrect.

Closing because in the master branch that will become 3.0, OCIO is now a required dependency, so we should no longer have to worry about behavior differences that might be present when OIIO is built without OCIO support -- that just won't happen any more.


config package is not used

The config package is listed as a dependency:
https://github.com/ubc/node-canvas-api/blob/43f5e6e56c5a46cda5ca006796686da01e2f6a00/package.json#L22
But, it does not appear to be used anywhere in the codebase. Let's remove it.

Resolved in https://github.com/ubc/node-canvas-api/pull/66


Allow first names containing non-english chars

My first name is Kévin (notice the é), and currently I cannot register on OpenCFP. This PR handles first names like last names, and will allow people with non-english characters in their names to submit CFPs with their real names.

Thanks @dunglas


IGNITE-12081 Page replacement can reload invalid page during checkpoint

Merged, https://github.com/apache/ignite/commit/91ddcd468ae0d3c828ac0677bc786b9d2a8c4989


Update of reconciler image used on Kyma control plane

Update of the reconciler image after merging pr-1317 to main branch with fix for problem of connectivity-proxy-reconciler not refreshing credentials.
https://github.com/kyma-incubator/reconciler/pull/1317

/test pre-main-control-plane-reconciler-upgrade-kyma2-latest-to-main-gardener

/test pre-main-control-plane-reconciler-upgrade-kyma2-latest-to-main-gardener

/test pre-main-control-plane-reconciler-upgrade-kyma2-latest-to-main-gardener

There is a PR for bumping the version already: https://github.com/kyma-project/control-plane/pull/2600


Prevent possible NullReferenceException

In some cases, the prevalues of the datatype may not contain the AllowedDoctypes key, resulting in settings.AllowedDoctypes to be null. This in turn, caused the entire website to be broken for me due to the NullReferenceException being thrown. Saving the DataType causes the prevalues to be there as expected, but I believe the library should not break the site.
This quickfix solves this issue. Another possibility is making sure that settings.AllowedDoctypes is initialized as an empty array by default, but I'll leave that decision up to you.

Hey @rbottema,
Thanks for this PR - sorry it's been open a while, somehow slipped through the net...
Meanwhile this was already fixed in the V7 branch and is coming in the next release!
Cheers,
Callum

Hey @rbottema,
Thanks for this PR - sorry it's been open a while, somehow slipped through the net...
Meanwhile this was already fixed in the V7 branch and is coming in the next release!
Cheers,
Callum


[CI:BUILD] Packit: run cockpit-podman tests in PRs

See https://cockpit-project.org/blog/tmt-cross-project-testing.html

.. yup, fails as expected. :heavy_check_mark:


Update pekko-sbt-paradox (#161)

(cherry picked from commit 31e1952304d72049f85824937d361483e721a971)

[error] sbt.librarymanagement.ResolveException: Error downloading org.apache.pekko:pekko-theme-paradox:1.0.1-RC1+5-13892678-SNAPSHOT
[error]   Not found
[error]   Not found
[error]   not found: /home/runner/.ivy2/localorg.apache.pekko/pekko-theme-paradox/1.0.1-RC1+5-13892678-SNAPSHOT/ivys/ivy.xml
[error]   not found: https://repo1.maven.org/maven2/org/apache/pekko/pekko-theme-paradox/1.0.1-RC1+5-13892678-SNAPSHOT/pekko-theme-paradox-1.0.1-RC1+5-13892678-SNAPSHOT.pom

[error] sbt.librarymanagement.ResolveException: Error downloading org.apache.pekko:pekko-theme-paradox:1.0.1-RC1+5-13892678-SNAPSHOT
[error]   Not found
[error]   Not found
[error]   not found: /home/runner/.ivy2/localorg.apache.pekko/pekko-theme-paradox/1.0.1-RC1+5-13892678-SNAPSHOT/ivys/ivy.xml
[error]   not found: https://repo1.maven.org/maven2/org/apache/pekko/pekko-theme-paradox/1.0.1-RC1+5-13892678-SNAPSHOT/pekko-theme-paradox-1.0.1-RC1+5-13892678-SNAPSHOT.pom

Yes this happens because the snapshots repo is being overloaded. A release is on the way but its still waiting for another binding one vote (see https://lists.apache.org/thread/dpqckc5g23fw7q6plh0d89md7tn6rr1c)

Hmm wait, I think its because this repo doesn't have snapshots repository which wasn't cherry picked, fixing


Support streaming in size estimation function in push_to_hub

This PR adds the streamable version of os.path.getsize (fsspec can return None, so we fall back to fs.open to make it more robust) to account for possible streamable paths in the nested extra_nbytes_visitor function inside push_to_hub.

would this allow to include the size in the dataset info without downloading the files? related to https://github.com/huggingface/datasets/pull/3670

@severo I don't think so. We could use this to get info.download_checksums["num_bytes"], but we must process the files to get the rest of the size info.


jarsigner error - invalid entry compressed size

Getting the following error:
jarsigner: unable to sign jar: java.util.zip.ZipException: invalid entry compressed size
Here is the more context:
Run ilharp/sign-android-release@v1
 with:
   releaseDir: android/app/build/outputs/bundle/release
   signingKey: ***
   keyAlias: ***
   keyStorePassword: ***
   keyPassword: ***
Signing file(s) in android/app/build/outputs/bundle/release with key ***...

Now sign 1 file(s):
- app-release.aab

[1/1] app-release.aab
 Signing AAB file.
 /usr/bin/jarsigner -keystore *** -storepass *** -signedjar android/app/build/outputs/bundle/release/app-release-temp.aab -keypass *** android/app/build/outputs/bundle/release/app-release.aab ***
 jarsigner: unable to sign jar: java.util.zip.ZipException: invalid entry compressed size (expected 83494 but got 85533 bytes)
Error: Error: The process '/usr/bin/jarsigner' failed with exit code 1

Normally files should be like app-release-unsigned.aab. Your aab is already signed before this action. Check your gradle build script or any other build step that runs before this action.

Yep the android/app/build.gradle was doing it...
We commented out the signingConfig from buildTypes.release and good to go


Review PooledByteBufAllocator in respect of jemalloc 4.x changes

There are some interesting changes in jemalloc. We should review our PooledByteBufAllocator and see if we can incorporate some stuff as it was written with jemalloc 3 in mind.
See also:
https://www.youtube.com/watch?v=RcWp5vwGlYU
https://github.com/jemalloc/jemalloc/blob/dev/ChangeLog
https://github.com/jemalloc/jemalloc/milestones/4.0.0

@trustin FYI as you write the initial impl :)

@Scottmitch also FYI :)

@normanmaurer - Thanks for reporting.

I found another malloc (tcmalloc) from google similar to jemalloc. it includes some useful tools (heap checker and heap profiler) as well. safari and chrome using tcmalloc.

I will try to god this over the next weeks :)

@normanmaurer, take a look at https://github.com/jemalloc/jemalloc/releases/tag/4.1.0, what about netty plan change according to new jemalloc version

As jemalloc release version 5, can you take a look and review netty's implementation.

Hi, I have some ideas. There are some inspirations from  jemalloc 4.

Split chunk into runs which are multiple of page size rather than power of 2 of page size. It can greatly reduce internal fragments. Use pairing heaps to manage runs (each heap manage the same size of runs), so that we always get a run with smaller id first to avoid dispersedly using runs.
Make cache size in PooledThreadCache dynamic. Change cache size according to previous usage of cache. It can improve cache utilization rate.
Do you think it worth a try? I’d love to contribute it :) @normanmaurer @trustin @Scottmitch

@yuanrw would love to review a PR :)

[ ] Add more size classes. Update Chunk structure. Update PooledByteBufAllocator.
[ ] Introduce low water for PoolThreadCache, make cache size dynamic.
[ ] Add more performance tests, includes multi-threaded/single-threaded benchmark, memory usage benchmark.


Errors while trying "--enable-prometheus-server" to execute APIResponsivenessPrometheusSimple and other measurements

I have a bare-metal K8S cluster v1.17.0. I tried to run some test configs (node-throughput/config.yaml), it runs successfully except that at the end I don't get APIResponsiveness measurements because it reports that "Prometheus is not enabled" (even though Prometheus was already installed on my cluster). I deleted existing Prometheus deployment namespace and I run the following command:
./clusterloader --kubeconfig=$KUBECONFIG --testconfig=$CONFIG --log_dir=$LOGDIR --alsologtostderr --enable-prometheus-server
It starts applying the manifests to create Prometheus then suddenly I get the following error :
I0227 15:53:51.029257   18906 framework.go:223] Applying /root/go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/prometheus-rules.yaml
F0227 15:54:03.147139   18906 clusterloader.go:250] Error while setting up prometheus stack: error while applying (/root/go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/prometheus-rules.yaml): timed out waiting for the condition

When I try to apply the manifest manually, this what happens:
# kubectl apply -f /root/go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/prometheus-rules.yaml
Error from server (InternalError): error when creating "/root/go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/prometheus-rules.yaml": Internal error occurred: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: service "prometheus-operator-operator" not found

I tried to debug for a while but then I got lost, I have no idea what's wrong in all of that. Is there any guidance how to get around this and get the measurements ?

Clusterloader doesn't work currently with other Prometheus instances than created by clusterloader itself. There are couple of assumptions baked in the code of clusterloader making it impossible at the moment to use other Prometheus instances (PRs welcome :) )

Based on the logs it seems, that there might a problem with Prometheus operator (we use to instantiate Prometheus in test clusters). Could you check whether the operator has been started correctly?

The Prometheus operator is running very well:
pod/prometheus-operator-6d66b7d98c-k2b24   1/1     Running             0          3m1s
The pod logs:
s=2020-02-28T23:36:58.093199649Z caller=main.go:169 msg="Starting Prometheus Operator version '0.30.0'."
level=info ts=2020-02-28T23:36:58.280004114Z caller=operator.go:213 component=alertmanageroperator msg="connection established" cluster-version=v1.17.0
level=info ts=2020-02-28T23:36:58.280843976Z caller=operator.go:391 component=prometheusoperator msg="connection established" cluster-version=v1.17.0
level=info ts=2020-02-28T23:36:58.886343299Z caller=operator.go:1478 component=prometheusoperator msg="CRD updated" crd=Prometheus
level=info ts=2020-02-28T23:36:58.901228252Z caller=operator.go:1478 component=prometheusoperator msg="CRD updated" crd=ServiceMonitor
level=info ts=2020-02-28T23:36:58.97980253Z caller=operator.go:635 component=alertmanageroperator msg="CRD updated" crd=Alertmanager
level=info ts=2020-02-28T23:36:58.99213402Z caller=operator.go:1478 component=prometheusoperator msg="CRD updated" crd=PrometheusRule
level=info ts=2020-02-28T23:37:02.081077583Z caller=operator.go:229 component=alertmanageroperator msg="CRD API endpoints ready"
level=info ts=2020-02-28T23:37:02.281619696Z caller=operator.go:184 component=alertmanageroperator msg="successfully synced all caches"
level=info ts=2020-02-28T23:37:08.192302913Z caller=operator.go:407 component=prometheusoperator msg="CRD API endpoints ready"
level=info ts=2020-02-28T23:37:08.89417053Z caller=operator.go:342 component=prometheusoperator msg="successfully synced all caches"
level=info ts=2020-02-28T23:37:08.894502876Z caller=operator.go:985 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2020-02-28T23:37:09.287938046Z caller=operator.go:985 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2020-02-28T23:37:09.403865032Z caller=operator.go:985 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s

Here is the complete log of the last API request that failed and the complete error response returned by the api-server:
I0229 00:36:57.014814   29068 request.go:942] Request Body: {"apiVersion":"monitoring.coreos.com/v1","kind":"PrometheusRule","metadata":{"labels":{"prometheus":"k8s","role":"alert-rules"},"name":"prometheus-k8s-rules","namespace":"monitoring"},"spec":{"groups":[{"name":"apiserver.rules","rules":[{"expr":"histogram_quantile(0.99, sum(rate(apiserver_request_duration_seconds_bucket[5m])) by (resource,  subresource, verb, scope, le))\n","labels":{"quantile":"0.99"},"record":"apiserver:apiserver_request_latency:histogram_quantile"},{"expr":"histogram_quantile(0.9, sum(rate(apiserver_request_duration_seconds_bucket[5m])) by (resource,  subresource, verb, scope, le))\n","labels":{"quantile":"0.90"},"record":"apiserver:apiserver_request_latency:histogram_quantile"},{"expr":"histogram_quantile(0.5, sum(rate(apiserver_request_duration_seconds_bucket[5m])) by (resource,  subresource, verb, scope, le))\n","labels":{"quantile":"0.50"},"record":"apiserver:apiserver_request_latency:histogram_quantile"}]},{"name":"probes.rules","rules":[{"expr":"histogram_quantile(0.99, sum(rate(probes_in_cluster_network_latency_seconds_bucket[5m])) by (le))\n","labels":{"quantile":"0.99"},"record":"probes:in_cluster_network_latency:histogram_quantile"},{"expr":"histogram_quantile(0.90, sum(rate(probes_in_cluster_network_latency_seconds_bucket[5m])) by (le))\n","labels":{"quantile":"0.90"},"record":"probes:in_cluster_network_latency:histogram_quantile"},{"expr":"histogram_quantile(0.50, sum(rate(probes_in_cluster_network_latency_seconds_bucket[5m])) by (le))\n","labels":{"quantile":"0.50"},"record":"probes:in_cluster_network_latency:histogram_quantile"},{"expr":"histogram_quantile(0.99, sum(rate(probes_in_cluster_dns_latency_seconds_bucket[5m])) by (le))\n","labels":{"quantile":"0.99"},"record":"probes:dns_lookup_latency:histogram_quantile"},{"expr":"histogram_quantile(0.90, sum(rate(probes_in_cluster_dns_latency_seconds_bucket[5m])) by (le))\n","labels":{"quantile":"0.90"},"record":"probes:dns_lookup_latency:histogram_quantile"},{"expr":"histogram_quantile(0.50, sum(rate(probes_in_cluster_dns_latency_seconds_bucket[5m])) by (le))\n","labels":{"quantile":"0.50"},"record":"probes:dns_lookup_latency:histogram_quantile"}]},{"name":"kube-proxy.rules","rules":[{"expr":"histogram_quantile(0.99, sum(rate(kubeproxy_network_programming_duration_seconds_bucket[5m])) by (le))\n","labels":{"quantile":"0.99"},"record":"kubeproxy:kubeproxy_network_programming_duration:histogram_quantile"},{"expr":"histogram_quantile(0.90, sum(rate(kubeproxy_network_programming_duration_seconds_bucket[5m])) by (le))\n","labels":{"quantile":"0.90"},"record":"kubeproxy:kubeproxy_network_programming_duration:histogram_quantile"},{"expr":"histogram_quantile(0.50, sum(rate(kubeproxy_network_programming_duration_seconds_bucket[5m])) by (le))\n","labels":{"quantile":"0.50"},"record":"kubeproxy:kubeproxy_network_programming_duration:histogram_quantile"},{"expr":"histogram_quantile(0.99, sum(rate(kubeproxy_network_programming_duration_seconds_bucket[5m])) by (le, pod))\n","labels":{"quantile":"0.99"},"record":"kubeproxy:kubeproxy_network_programming_duration:histogram_quantile_by_pod"},{"expr":"histogram_quantile(0.90, sum(rate(kubeproxy_network_programming_duration_seconds_bucket[5m])) by (le, pod))\n","labels":{"quantile":"0.90"},"record":"kubeproxy:kubeproxy_network_programming_duration:histogram_quantile_by_pod"},{"expr":"histogram_quantile(0.50, sum(rate(kubeproxy_network_programming_duration_seconds_bucket[5m])) by (le, pod))\n","labels":{"quantile":"0.50"},"record":"kubeproxy:kubeproxy_network_programming_duration:histogram_quantile_by_pod"}]},{"name":"apiserver.1m.rules","rules":[{"expr":"histogram_quantile(0.99, sum(rate(apiserver_request_duration_seconds_bucket[1m])) by (resource,  subresource, verb, scope, le))\n","labels":{"quantile":"0.99"},"record":"apiserver:apiserver_request_latency_1m:histogram_quantile"},{"expr":"histogram_quantile(0.9, sum(rate(apiserver_request_duration_seconds_bucket[1m])) by (resource,  subresource, verb, scope, le))\n","labels":{"quantile":"0.90"},"record":"apiserver:apiserver_request_latency_1m:histogram_quantile"},{"expr":"histogram_quantile(0.5, sum(rate(apiserver_request_duration_seconds_bucket[1m])) by (resource,  subresource, verb, scope, le))\n","labels":{"quantile":"0.50"},"record":"apiserver:apiserver_request_latency_1m:histogram_quantile"}]}]}}
I0229 00:36:57.015031   29068 round_trippers.go:419] curl -k -v -XPOST  -H "Accept: application/json" -H "User-Agent: clusterloader/v0.0.0 (linux/amd64) kubernetes/$Format" 'https://<IP_ADDRESS>:6443/apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheusrules'
I0229 00:36:57.018128   29068 round_trippers.go:438] POST https://<IP_ADDRESS>:6443/apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheusrules 500 Internal Server Error in 3 milliseconds
I0229 00:36:57.018159   29068 round_trippers.go:444] Response Headers:
I0229 00:36:57.018174   29068 round_trippers.go:447]     Content-Type: application/json
I0229 00:36:57.018181   29068 round_trippers.go:447]     Content-Length: 637
I0229 00:36:57.018188   29068 round_trippers.go:447]     Date: Fri, 28 Feb 2020 23:36:57 GMT
I0229 00:36:57.018215   29068 request.go:942] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook \"prometheusrulemutate.monitoring.coreos.com\": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: service \"prometheus-operator-operator\" not found","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook \"prometheusrulemutate.monitoring.coreos.com\": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: service \"prometheus-operator-operator\" not found"}]},"code":500}
F0229 00:36:57.018357   29068 clusterloader.go:250] Error while setting up prometheus stack: error while applying (/root/go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/prometheus-rules.yaml): timed out waiting for the condition

@oxddr what do you think about the situation ?


Allow config in snake and camel case

The PR allows the whole configuration to be defined in snake_case on camelCase.
I've also updated the readme to be consistent snake_case

Very nice, thanks!


Reported error when using FP16

HI:
I tried to run the run_custom_classifier with fp16=true and with the following error:
I1126 13:48:27.714882 545560 saver.py:1284] Restoring parameters from E:\BERT\workspace\outputmodel\model.ckpt-0
2019-11-26 13:48:28.693963: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key bad_steps not found in checkpoint
I commented the fp16 part in optimizer then it can run successfully.  I initialized from BERT base model from Google official version. My guess is that the init checkpoint does not have the bad_steps which is in used in fp16 mixed precision optimizer.
How to I fix this?
Thanks!

Hi! @secretsh
Are you using FP16 in Multi-GPU mode?
YES:
Please refer to this issue(https://github.com/HaoyuHu/bert-multi-gpu/issues/18).
  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I1127 16:54:12.318208<PHONE_NUMBER>31840 session_manager.py:500] Running local_init_op.
I1127 16:54:12.479427<PHONE_NUMBER>31840 session_manager.py:502] Done running local_init_op.
I1127 16:54:19.856697<PHONE_NUMBER>31840 basic_session_run_hooks.py:606] Saving checkpoints for 0 into /home/hahahu/bert/outputs/qqp/model.ckpt.
2019-11-27 16:54:29.908500: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
W1127 16:54:38.374780<PHONE_NUMBER>31840 basic_session_run_hooks.py:724] It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 0 vs previous value: 0. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.
W1127 16:54:38.756153<PHONE_NUMBER>31840 basic_session_run_hooks.py:724] It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 0 vs previous value: 0. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.
W1127 16:54:39.139222<PHONE_NUMBER>31840 basic_session_run_hooks.py:724] It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 0 vs previous value: 0. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.
W1127 16:54:39.529100<PHONE_NUMBER>31840 basic_session_run_hooks.py:724] It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 0 vs previous value: 0. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.
W1127 16:54:39.916643<PHONE_NUMBER>31840 basic_session_run_hooks.py:724] It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 0 vs previous value: 0. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.
I1127 16:54:47.634445<PHONE_NUMBER>31840 tpu_estimator.py:2159] global_step/sec: 0.0590434
I1127 16:54:47.636053<PHONE_NUMBER>31840 tpu_estimator.py:2160] examples/sec: 1.88939
... ...
I1127 16:55:26.539864<PHONE_NUMBER>31840 tpu_estimator.py:2159] global_step/sec: 2.48653
I1127 16:55:26.540206<PHONE_NUMBER>31840 tpu_estimator.py:2160] examples/sec: 79.5691
I1127 16:55:26.541410<PHONE_NUMBER>31840 basic_session_run_hooks.py:606] Saving checkpoints for 93 into /home/hahahu/bert/outputs/qqp/model.ckpt.
I1127 16:55:28.357516<PHONE_NUMBER>31840 estimator.py:368] Loss for final step: 0.68926907.
I1127 16:55:28.359129<PHONE_NUMBER>31840 error_handling.py:96] training_loop marked as finished

Could you pls provide a piece of code and related test dataset for me to solve these bugs? Thank you!

Hi @HaoyuHu , I changed my testing environment with GPU machine, now it's working. previously I was using CPU for testing and that may be the cause.
Thank you!


Update README.md for ohdsi-studies crawler

All repos in ohdsi-studies must conform to the template or else the crawler can't parse it correctly (see https://data.ohdsi.org//OhdsiStudies/ )


[GLUTEN-4803][UT] Add Golden Files for TPC-H Spark33 + Gluten Execution Plan

What changes were proposed in this pull request?
Close #4803
As title, follow up to add TPC-H + Spark33 Golden Files.
How was this patch tested?
unit tests

Hi @ulysses-you @PHILO-HE, please take a review and help re-trigger the fail ci.

let's hold on this pr for https://github.com/oap-project/gluten/pull/4815

Do some rebase operations, mark as draft for now to check.


[Event Request] Codeunit 5760 "Whse.-Post Receipt" – New Event  “OnBeforeCheckWhseRcptLine”

Could we please have a new publisher “OnBeforeCheckWhseRcptLine” in Function “Code”, Codeunit 5760 "Whse.-Post Receipt” including an “IsHandled” pattern?

    local procedure "Code"()
    var
        WhseManagement: Codeunit "Whse. Management";
        IsHandled: Boolean;
    begin
        with WhseRcptLine do begin
            SetCurrentKey("No.");
            SetRange("No.", "No.");
            SetFilter("Qty. to Receive", '>0');
            if Find('-') then
                repeat
*** start new code ***
                    ishandled := false;
                    OnBeforeCheckWhseRcptLine(WhseRcptLine, IsHandled);
                    if not (ishandled) then begin
*** stop new code ***
                        TestField("Unit of Measure Code");
*** start new code ***
                    end;
*** stop new code ***
                    WhseRqst.Get(
                      WhseRqst.Type::Inbound, "Location Code", "Source Type", "Source Subtype", "Source No.");
…
*** start new code ***
    [IntegrationEvent(false, false)]
    local procedure OnBeforeCheckWhseRcptLine(WhseReceiptLine: Record "Warehouse Receipt Line"; var IsHandled: Boolean)
    begin
    end;
*** stop new code ***

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. We will update this issue with information about availability.


Make to_user optional in handle_create_payload

Public content does not require a recipient.

Current coverage is 88.12% (diff: 87.50%)

Merging #43 into master will decrease coverage by 3.14%

@@             master        #43   diff @@
==========================================
  Files            13         13
  Lines           699        699
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
- Hits            638        616    -22
- Misses           61         83    +22
  Partials          0          0

Powered by Codecov. Last update b2011b3...42db264


fixing link to shared key content in REST

Addressing https://github.com/MicrosoftDocs/azure-docs/issues/10015

Microsoft authors of articles need to make updates through the private repository so the updates can be staged and validated by the current validation rules. After you review staging and fix all validation issues, you can add the sign-off comment to let the PR reviewers know the updates are ready to be merged.It's easy to access the private repo version of your articles in GitHub. Just add -pr to the URL, as follows:
Public repo URL: https://github.com/Microsoft/azure-docs/blob/master/articles/azure-portal-overview.md
Private repo URL: https://github.com/Microsoft/azure-docs-pr/blob/master/articles/azure-portal-overview.md
We have closed this pull request. Please resubmit the updates through the private repo. If you need perms to the private repo, see the contributor guide instructions


Add license mention for every crate

What this PR does / why we need it:
License information is missing from our crates metadata, this fix this and fill this field for every crate.
Special notes for your reviewer:
If applicable:

[ ] this PR has an associated PR with documentation in akri-docs
[ ] this PR contains unit tests
[ ] added code adheres to standard Rust formatting (cargo fmt)
[ ] code builds properly (cargo build)
[ ] code is free of common mistakes (cargo clippy)
[ ] all Akri tests succeed (cargo test)
[ ] inline documentation builds (cargo doc)
[x] all commits pass the DCO bot check by being signed off -- see the failing DCO check for instructions on how to retroactively sign commits

/version patch


Particle data

Is there a way for me to use an array to inject data into the particles rather than random.
I want to essentially put a different icon in each and then have a hover event which will show the title.

You can see this sample here: https://codepen.io/matteobruni/pen/YzpOZpq
It uses tsParticles for creating particles manually

Whats the difference between this and tsParticles? Im a bit confused

Can I put in more than just the position, e.g. I want to ideally put in a stroked circle with a different icon in for each particle

Can I put in more than just the position, e.g. I want to ideally put in a stroked circle with a different icon in for each particle

Yes, you can see the properties here: https://particles.js.org/docs/classes/Options_Classes_ManualParticle.ManualParticle.html
You can specify position and options, the latest one contains the same object as the main particles object


docs(TextArea): fix verbiage in TextArea docs

Change the phrase  "don't support more" to "don't support anymore" in the TextArea docs source, for proper English grammar
First PR, I reviewed the guidelines and I believe I conform. Please let me know if I missed something. Happy to fix.

Codecov Report

Merging #4317 (2a36db6) into master (e34b68a) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #4317   +/-   ##
=======================================
  Coverage   99.75%   99.75%
=======================================
  Files         180      180
  Lines        3241     3241
=======================================
  Hits         3233     3233
  Misses          8        8

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e34b68a...2a36db6. Read the comment docs.


[FEATURE] CSS variables

Did you already consider including the JS config variables in the generated lib, as css variables?
In my projects, I include the generated output of a configured tailwind as a css library, then extend it with targeted css, and I'd love to be able to make use of, say, colors as css variables.
Ex:
In Tailwind config:
let colors = {
  ...
  'black': '#22292f',
  ...
}

would generate in the css:
:root {
  --color-black: #22292f;
}

See https://github.com/tailwindcss/tailwindcss/issues/260
At the current time it makes more sense to define the variables you'd like in your CSS.

Alright, so, the other way around. Thanks.

Yeah like mentioned in the other issue we don't have concrete plans to implement anything like this, but it could totally be done with another PostCSS plugin or even Tailwind's upcoming plugin system.

I made it as plugin
https://github.com/omarkhatibco/tailwind-css-variables


NIFI-13263 - Bump ParCEFone version

Summary
NIFI-13263
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking

[X] Apache NiFi Jira issue created

Pull Request Tracking

[X] Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
[X] Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

[X] Pull Request based on current revision of the main branch
[X] Pull Request refers to a feature branch with one commit containing changes

Verification
Please indicate the verification steps performed prior to pull request creation.
Build

[X] Build completed using mvn clean install -P contrib-check

[X] JDK 21

UI Contributions

[ ] NiFi is modernizing its UI. Any contributions that update the current UI also need to be implemented in the new UI.

Licensing

[ ] New dependencies are compatible with the Apache License 2.0 according to the License Policy
[ ] New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

[ ] Documentation formatting appears as expected in rendered files

@exceptionfactory @pvillard31  FYI

Glad to see you back @trixpan !
I checked your commits upstream and everything passes here, looks good to me!


Make the filtering of duplicate segments more efficient

@exodaniel @HannesSommer I'd like to make the filterDuplicateSegmentsOfTargetMap() function more efficient:
https://github.com/ethz-asl/segmatch/blob/feature/incremental_estimator/segmatch/src/segmatch.cpp#L466
The current idea is that we get a cloud of all target segments centroids and loop over all segments centroid of the cloud to add and check if we should remove the segment from the target cloud.
Creating the target segments cloud centroid can take a few ms (50-100 ms on a big KITTI map). Brute force knn to this cloud currently takes a lot of time (500ms+). I will change by first applying a cylindrical filter to that cloud, according on the trajectory pose they are linked to.
The best would be to first grab the "near collectors" as we had before and then build the local cloud without having to iterate over all segments but only over the ones which are linked to a near collector. We could potentially achieve the same by modifying the definition of valid_segments_ in SegmentedCloud from :
std::vector<Segment> valid_segments_;  to:
typedef std::pair<SE3, std::vector<Segment> > PoseAndSegmentsPair;
std::vector<PoseAndSegmentsPair> valid_segments_;
But that would require some refactoring. Any thoughts?

Update: this is the expensive function:
https://github.com/ethz-asl/segmatch/blob/master/segmatch/src/segmented_cloud.cpp#L236
I'm gonna switch to an unordered_map

Has the efficiency improved or worsen without collectors?
I would assume that this is a case where working with limited neighborhoods is efficient, but I don't see how you would do that if all segments are in the same non-ordered data structure. Any thoughts on a sort of Octomap for segment poses? It might not make much sense if segments in some area all have pretty much the same pose, otherwise it could gain you some efficiency.
Bon courage!

@exodaniel it improved and made the integration cleaner. As I mentioned in my last post the removal is the expensive bit. Moving to unordered_map will hopefully solve that. I'll test soon.

Solved in #13

Cool !  I was wondering about this specific case,  since you're working with a single big structure. Good job on solving this!

Thanks! :-) It works fine for the current datasets. It might require improvement on bigger datasets though. The main issue was the deletion in a vector which is not efficient.


Release/1.3.1

[1.3.1]
Added

issues/50: Spatial bounds are computed correctly for grouped empty subset operations

Changed

Upgraded cmr-umm-updater to 0.2.0

Deprecated
Removed
Fixed

issues/48: get_epoch_time_var was not able to pick up the 'time' variable for the TROPOMI CH4 collection. Extra elif statement was added to get the full time variable returned.
issues/54: Skip encoding when xr dataset is empty

Security

I think we decided to increase the timeout on the cmr-umm-updater right? Let's use this release to try to get that working.

Yep, Simon has a draft PR for that in the cmr-umm-updater repo. I did re-run the build and it worked without the timeout increase, but this is a good opportunity to test that release.

The latest cmr-umm-updater release seems to be working. The last thing we're waiting on is some test data to be ingested so we can test this release on the datasets that were causing issues in 1.3.0.

I have done some manual testing in UAT for this release for MODIS A and all L2 SWOT simulation collections.

Collection Name
CID
Status
URL

MODIS_A-JPL-L2P-v2019.0
C1234724470-POCLOUD
Partial Pass
Harmony Request

SWOT_SIMULATED_L2_KARIN_SSH_GLORYS_CALVAL_V1
C1242387586-POCLOUD
Pass
Harmony Request

SWOT_SIMULATED_L2_NADIR_SSH_GLORYS_CALVAL_V1
C1242387602-POCLOUD
Pass
Harmony Request

SWOT_SIMULATED_L2_NADIR_SSH_GLORYS_SCIENCE_V1
C1242387624-POCLOUD
Pass
Harmony Request

SWOT_SIMULATED_L2_KARIN_SSH_ECCO_LLC4320_SCIENCE_V1
C1242387592-POCLOUD
Pass
Harmony Request

SWOT_SIMULATED_L2_NADIR_SSH_ECCO_LLC4320_SCIENCE_V1
C1242387600-POCLOUD
Pass
Harmony Request

SWOT_SIMULATED_L2_KARIN_SSH_ECCO_LLC4320_CALVAL_V1
C1242387601-POCLOUD
Pass
Harmony Request

SWOT_SIMULATED_L2_NADIR_SSH_ECCO_LLC4320_CALVAL_V1
C1242387620-POCLOUD
Pass
Harmony Request

Note that MODIS A is only a partial pass. The MODIS A request successfully subsetted about 80 granules before running into the following error:
Traceback (most recent call last):
  File "/home/dockeruser/.local/lib/python3.8/site-packages/harmony/cli.py", line 205, in _invoke
    (out_app_message, out_catalog) = adapter.invoke()
  File "/home/dockeruser/.local/lib/python3.8/site-packages/harmony/adapter.py", line 116, in invoke
    return (self.app_message, self._process_catalog_recursive(self.catalog))
  File "/home/dockeruser/.local/lib/python3.8/site-packages/harmony/adapter.py", line 152, in _process_catalog_recursive
    output_item = self.process_item(item.clone(), source)
  File "/worker/podaac/subsetter/subset_harmony.py", line 114, in process_item
    input_filename = download(asset.href,
  File "/home/dockeruser/.local/lib/python3.8/site-packages/harmony/util.py", line 324, in download
    http.download(cfg, source, access_token, data, destination_file, full_user_agt)
  File "/home/dockeruser/.local/lib/python3.8/site-packages/harmony/http.py", line 365, in download
    raise Exception('Unable to download: unknown error.')
Exception: Unable to download: unknown error.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/dockeruser/.local/bin/l2ss_harmony", line 8, in <module>
    sys.exit(main())
  File "/worker/podaac/subsetter/subset_harmony.py", line 229, in main
    harmony.run_cli(parser, args, L2SubsetterService, cfg=config)
  File "/home/dockeruser/.local/lib/python3.8/site-packages/harmony/cli.py", line 298, in run_cli
    _invoke(adapter, args.harmony_metadata_dir)
  File "/home/dockeruser/.local/lib/python3.8/site-packages/harmony/cli.py", line 216, in _invoke
    _write_error(metadata_dir, 'Service request failed with an unknown error')
  File "/home/dockeruser/.local/lib/python3.8/site-packages/harmony/cli.py", line 145, in _write_error
    with open(path.join(metadata_dir, 'error.json'), 'w') as file:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/metadata/4e5f95ef-325b-47eb-bee9-327b042ab50e/13724/outputs/error.json'

This is a download error which is not related to L2SS-Py.


pytorchjob doesn't have status.startTIme.

Hi,
pytorchjob doesn't provide status.startTime.
mpi and tfjob has function to update status.startTime, but pytorchjob doesn't have.
Is it deprecated? Or just missing?
if jobStatus.StartTime == nil {
		now := metav1.Now()
		jobStatus.StartTime = &now
		// enqueue a sync to check if job past ActiveDeadlineSeconds
		if tfJob.Spec.RunPolicy.ActiveDeadlineSeconds != nil {
			logger.Infof("Job with ActiveDeadlineSeconds will sync after %d seconds", *tfJob.Spec.RunPolicy.ActiveDeadlineSeconds)
			// TODO(Jeffwan): requeue job key in reconciler scenarios
			r.WorkQueue.AddAfter(tfJobKey, time.Duration(*tfJob.Spec.RunPolicy.ActiveDeadlineSeconds)*time.Second)
		}
	}

Hi @aheeruru
I think you are right. XGB job also doesn't update startTime. Let me add them.


Translate April 24 covid report into Italian

Hello @nextstrain/italian!
The situation report from 2020-04-24 is ready to be translated to Italian. The report focuses on Africa. We chose this region this week as the outbreaks in African nations are very important to understand and do not seem to be getting a lot of press. Please translate the Twitter post below! It is a bummer when the translation is great, but people do not know that because I used Google translate for the Twitter post.
Branch to translate on: sit-rep_it
File to translate: https://github.com/nextstrain/translations/blob/sit-rep_it/narratives/translations_ncov_sit-rep_it_2020-04-24.md You can translate directly in this file, but please remember to push often as other people may be working on the translation!
Link to preview: https://nextstrain.org/community/narratives/nextstrain/translations@sit-rep_it/ncov/sit-rep/it/2020-04-24
Twitter post to translate:
“We focus on Africa this week in our #COVID19 situation report. The report is now available in Italian. Thank you @, @, and @ for the rapid translation."
Logistics
Please update the translations board as you work on the translation: https://github.com/orgs/nextstrain/projects/7
Please assign yourself to the translation if you’re working on it this week.
When a translation is ready for publication -- at minimum meaning two people have worked on it and both agree that it’s ready -- please create a pull request to merge to master and tag me and the issue in it!
Discussion board for your team is here: https://github.com/orgs/nextstrain/teams/italian
Standard rules for translation:
Note that the file to translate is in Markdown. In general, it is important not to change any formatting or links. (Although it is okay to update Wikipedia or CDC links to the appropriate language version.) Comments to help with translation are noted in comment symbols like <! -- Comment -->.
Please add your name under translators: with new line, tab, and dash, formatted just like authors are. Please also add a link to your professional page (can use Twitter) under translatorLinks:.
Note: You may know of other great resources available in Italian; please don’t add them to the situation report. We need to keep content the same across all translations, so we can be confident in the accuracy of what we publish. Thanks for understanding.
At minimum, I require two sets of eyes on every translation; one of those people should have a background in phylogenetics. Although if you have more, that’s fine too.
Feel free to use a live, online editor like hackmd (sign in with GitHub) to translate together.
Thank you all so, so much! :)

Il rapporto sul COVID19 di questa settimana si concentra sull'Africa ed è ora disponibile in Italiano. Grazie a @MEKlab, @ SaraMeiqi e @alice_ledda_ per la rapida traduzione.
Hi @cassiawag, I'm working on it tomorrow and Sunday.
Thanks a lot for your work!

Hi @cassiawag
I would like to add the link to the italian reports of the 3rd of April and 27th of March mentioned in the report of this week. I cannot find the links in the Nextstrain website. Could you tell me how to find them?
thank you!
Alessia

Hi Alessia, it's just the same link as in the report but replace ...sit--rep/2020... with ...sit-rep/it/2020...

ok perfect! thank you

Just wanted to check-in on the translation? How's it going? What needs to get done in order to publish?

Hi Cassia,
we are waiting for @aliceLedda to check a few lines from a technical point of view. I understood that she's quite busy this week, but I would feel safer if we can wait another bit for her opinion.
Thanks a lot for all your work. I'm sharing the report with a cultural group in my hometown (via Facebook) and they really appreciate it!

So glad to hear that! I've been sharing the English versions via Facebook as well, and people seem to be really appreciating them.

I check in the next 15 minutes!
I have been so busy I forgot!
Alice
On Thu, Apr 30, 2020, 15:35 Sara Pilia<EMAIL_ADDRESS>wrote:

Hi all,
I gave another look at the translation and it seems clear and correct in
Italian. I'm not sure whether @aliceLedda https://github.com/aliceLedda
is reading us and has the time to see the translation.
If @al-epo https://github.com/al-epo can read it again, and it still
looks good to her, maybe we can ask @cassiawag
https://github.com/cassiawag to publish it, and if Alice spots some
problems later, she can push a pull request and correct it.
What do you think?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/nextstrain/translations/issues/89#issuecomment-621893638,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACW4WUOZH3Y4YOCJ7XVXQDTRPGEE5ANCNFSM4MQND6AQ
.

Sara ho committato...  che facciamo?
Alice
On Thu, Apr 30, 2020 at 3:37 PM alice ledda<EMAIL_ADDRESS>wrote:

I check in the next 15 minutes!
I have been so busy I forgot!
Alice
On Thu, Apr 30, 2020, 15:35 Sara Pilia<EMAIL_ADDRESS>wrote:

Hi all,
I gave another look at the translation and it seems clear and correct in
Italian. I'm not sure whether @aliceLedda https://github.com/aliceLedda
is reading us and has the time to see the translation.
If @al-epo https://github.com/al-epo can read it again, and it still
looks good to her, maybe we can ask @cassiawag
https://github.com/cassiawag to publish it, and if Alice spots some
problems later, she can push a pull request and correct it.
What do you think?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/nextstrain/translations/issues/89#issuecomment-621893638,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACW4WUOZH3Y4YOCJ7XVXQDTRPGEE5ANCNFSM4MQND6AQ
.

Ciao!
Per me è ok

Ok fatto allora!
On Thu, Apr 30, 2020, 16:05 al-epo<EMAIL_ADDRESS>wrote:

Ciao!
Per me è ok
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/nextstrain/translations/issues/89#issuecomment-621912822,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACW4WUKZNSEYG5Q43JTODH3RPGHUHANCNFSM4MQND6AQ
.

Scusate per il ritardo, e' stata una settimana impegnativa!
On Thu, Apr 30, 2020, 17:04 alice ledda<EMAIL_ADDRESS>wrote:

Ok fatto allora!
On Thu, Apr 30, 2020, 16:05 al-epo<EMAIL_ADDRESS>wrote:

Ciao!
Per me è ok
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/nextstrain/translations/issues/89#issuecomment-621912822,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACW4WUKZNSEYG5Q43JTODH3RPGHUHANCNFSM4MQND6AQ
.

@aliceLedda tranquilla, me l'ero immaginato!
Alice, if it's ok with you, we can ask @cassiawag to publish the Italian translation!

Yes! Fine!
On Thu, Apr 30, 2020, 18:00 Sara Pilia<EMAIL_ADDRESS>wrote:

@aliceLedda https://github.com/aliceLedda tranquilla, me l'ero
immaginato!
Alice, if it's ok with you, we can ask @cassiawag
https://github.com/cassiawag to publish the Italian translation!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/nextstrain/translations/issues/89#issuecomment-621979137,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACW4WUKRFJZGQLPXUM5UX6TRPGVBBANCNFSM4MQND6AQ
.

Trying to troubleshoot preview, but will get this published after I sort that out.

Published and tweeted: https://nextstrain.org/narratives/ncov/sit-rep/it/2020-04-24
Thank you!!!


Add Scholarship Confirmation to SMS Games

FEATURE OVERVIEW
User Story
As a member, I want confirmation that I have qualified for the scholarship. Currently, I don't get any. 😢 It would be great to have copy on this page (which I see after I submit my number and three friends' numbers) that notes I have been entered into the scholarship.
The copy could be a subtitle under "Thanks for sharing!" And read: "And you've entered to win the scholarship!"

As a member of the member support team, I want members to know they were qualified for the scholarship without them having to ask.
Additional Information (optional)
We already have this functionality for non-SMS campaigns.

Refs:
https://github.com/DoSomething/phoenix/issues/4121
https://github.com/DoSomething/phoenix/pull/4122
Additionally, this confirmation copy should only appear if the SMS game has a scholarship attached.
Why This Matters
The member support team has flagged "Am I qualified for the scholarship?" as a frequent question coming from members who sign up for SMS games.
cc @mikefantini @hbghidey @jamjensen

the issue here is these links are public so we would need to check

is user logged in
did the user actually sign up?
are there other steps to be eligible for being entered here? do they need to complete the game?

@angaither: interesting point re: public links! Perhaps helpful thoughts follow:

The most likely scenario is that a user is not logged in before or after they fill out the SMS game form -- they would likely hit the site from Fastweb, not be logged in, fill out their name and four cell numbers, and then hit submit, never logging in. However, someone may have just reported back on a campaign and then found the SMS game and signed up, so then they would be logged in.
Er, hmm. Could we allude to the fact that the member needed to have signed up in order to be eligible for the scholarship?
@hbghidey in your opinion, what is the ideal copy based on what would be most clear to members and most true to the scholarship rules, and encompass both our thoughts above re #2? For the moment, I've recommended the copy could be a subtitle under "Thanks for sharing!" And read: "And you've entered to win the scholarship!" -- going to add this to our product/support agenda as well! ( @angaither that meeting is today, so we will circle back after -- thanks!)

 SMS games: Scholarship confirmation

@namimody @angaither Ideal copy would be "Thanks for sharing! You've taken the first step in being entered to win the scholarship. Next step is to complete the game!" The message is relatively vague but  we can sprinkle additional context in the campaign page copy, FAQs and SMS confirmation message to let users know that the game will need to be shared and completed in order to be entered into the drawing. If we see spot additional confusion, we can readdress.

Sounds 🆗 !
I think it's probably fine to just show this message to everyone then, ya @namimody?

It looks like currently if there is a value in field_reportback_confirm_msg that is currently displayed in that subtitle section, so we would like to remove that and replace with this text?

@hbghidey just a thought -- is it alright to always say "game" even if the SMS campaign is in the adviser template? is there a word that might be more encompassing but equally user friendly?

@angaither oh wow, that's a great catch. hmm I think that it would be great to have this language hardcoded if the campaign has a scholarship, or perhaps make that field required and provide two examples of copy that could go there -- one for the advisor template and one for the choose your own adventure template.

@mikefantini do you have a preference?

@namimody since we already have this field, it might make more sense to have campaign leads fill out the text here if there's a scholarship on the game that way we don't need to worry if the advisor template doesn't need it, but games do

@angaither I think it's preferred if it's hardcoded and appears dynamically depending on if there is a scholarship or not. Both advisor and game templates can have scholarships -- and the hardcoding works really well on non-SMS campaigns. Additionally, I'd like to minimize the number of things that leads have to do manually, if this is a best practice for all SMS experiences with scholarships.
I think the best solution is to have this text appear below "Thanks for sharing!":
"You've taken the first step in being entered to win the scholarship. Next step is to complete the experience!"
This should work for all scenarios.

@namimody @angaither Just want to confirm that If there is no scholarship, it should display the text in the field_reportback_confirm_msg but if there is a scholarship, we display the text outlined above?


Conflict with RefinedStorage v 1.3.5, specifically solders

Minecraft - 1.11.2
refinedstorage-1.3.5
mcflux-2.0.1  vs mcflux-2.0.0-b7
Updated mcflux from 2.0.0-b7 to 2.0.1. No other changes.
Refined Storage Solders would no longer accept items in top or bottom slot.
Reverted to mcflux-2.0.0-b7, no other changes, and Refine Storage again worked as expected.

Additional info:
forge-1.11.2-<IP_ADDRESS>6
Exact same results stand alone and client-server.

Same results with mcflux-2.0.2.
Refine Storage Solders fail when upgraded to mcflux-2.0.2. No other changes made.
Reverting to mcflux-2.0.0-b7 restores normal behaviors.

Did you receive some warnings or errors like "NBT data cannot be saved or read" or something similar? If you do, please send these logs (via Pastebin or Gist). I can't reproduce that problem so I need more information about it.

Negative, no log message of any kind, standalone nor client-server.
The selected items simply will not click into their respective top or bottom slots in the RS Solder. The middle slot behaves normally. I have tried a number of different items, both RS specific and Minecraft vanilla that are valid recipe items. The same exact test world ( and actual game worlds ) behave as expected when mcflux-2.0.0-b7 is swapped back in the place of mcflux-2.0.1+ and the world restarted.
I'm running a large, custom private mod pack with over 120 mods. There may well be an unfortunate  synergy taking place. I am going to pull all mods updated in the past two weeks and test to see if another mod is the actual issue. This will take several hours and I won't begin until after evening my time. I'll advise my results.

I tested with the following mods:
ActuallyAdditions-1.11.2-r99
CommonCapabilities-1.11.2-1.3.0
ConvenientAdditions-1.0.17-build27
CyclopsCore-1.11.2-0.9.1
EverlastingAbilities-1.11.2-1.2.4
ExponentialPower-1.1.1
Growable Cells-0.3.3
IvToolkit-1.3.2-1.10
RecurrentComplex-1.2.8
Supply Drop-1.11.2-1.0.1
UtilityWorlds-1.11.2_1.1.0
ViesCraft-1.11.2-4.11.0
environmentaltech-1.11.2-0.11.0d
extrautils2-1.11.2-1.2.0
justenoughbuttons-1.11.2-1.4.1
llibrary-1.7.4-1.11
mcjtylib-1.1x-2.3.5
multistorage-1.11.0-1.3.4
pmdumper-1.11.2-1.5.1
rftools-1.1x-5.82
shetiphiancore-1.11.0-3.4.2
simplegenerators-1.11.2-0.11.0a
teslafied-1.11-1.1.0
thedragonlib-1.11.2-2.1.0
theoneprobe-1.1x-1.4.1
valkyrielib-1.11.2-0.11.0c
zz-backpacks 1.11.2 - 3.1.7
I removed all of the mods updated in the past ten days, listed above, and none has any effect on mcflux v2.0.1+ and Refined Storage.
I understand you can not replicate this issue. Been there, done that, still have the scars...
I will continue to test my private mod pack as new versions  are released. This is a promising mod and I would like to use it if I can resolve this issue.

Updated to mcflux 2.0.5, no change.
Updated:
CommonCapabilities-1.11.2-1.3.0 to CommonCapabilities-1.11.2-1.3.1
CyclopsCore-1.11.2-0.9.1 to CyclopsCore-1.11.2-0.9.2
and built RefinedStorage 1.4 from the source.
And the problem has been resolved.
Obviously, it was a slight conflict amongst one of those three mods and mcflux.
Please close this issue.

Confirmed here, no console messges of any kind. TAke out your mod and the pack works.


Is there planned a comment system for the blog?

Not a third party service such as Disqus, but integrated with Formwork. Are becoming rarer CMS without database with managing blog comments natively.
Thank to reading ;)

Hi @MiFrance, thank you for the question.
I'm sorry but a native comment system for the blog is not planned since it would be difficult to maintain and would have to deal with spam and potential security issues. So using a third party service is reasonably a better and more secure option.

Hmm it's a shame :(
Yet with a moderation of comments (optional) and a simple captcha with questions / answers configured in backend, it would do the trick.
The problem with third-party services is that we do not know what they are doing with our data ... No matter if the service closes.


Feature/preserve original path

Ticket: https://github.com/TestArmada/flank/issues/611

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.Tahir Hajizada seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.

Codecov Report

Merging #615 into master will decrease coverage by 0.04%.
The diff coverage is 78.94%.

@@             Coverage Diff              @@
##             master     #615      +/-   ##
============================================
- Coverage     76.35%   76.31%   -0.05%
- Complexity      606      610       +4
============================================
  Files            81       81
  Lines          2284     2297      +13
  Branches        324      327       +3
============================================
+ Hits           1744     1753       +9
- Misses          323      324       +1
- Partials        217      220       +3

Thanks for submitting a pull request! I'll take a look soon.

I'm happy to make the suggested changes if that'd be easier.

I am ok with your suggestions. Makes sense. It would be nice if you apply them or I can do after 9-10 hours :)

Sounds good. I'll apply the fixes in ~4 hours after work.

I updated, tested, and merged your pull request! Nice work.
I released Flank v8.1.0 with the feature and updated the change log.
Thanks for contributing.

Thank you for the changes and release. Welcome.


Simplify promotion after A/B(/n) test

Is your feature request related to a problem? Please describe.
Promoting a winning version requires several steps to avoid dropping user requests:

unset iter8.tools/abn label on default version to step traffic to it
upgrade the default version
set iter8.tools/abn label to true to restart traffic to (upgraded) default version
unset iter8.tools/abn on the candidate version
delete candidate resources

Iter8 is not responsible for these steps. However, it is reasonable to ask: is there something Iter8 can do that will reduce the effort for the user?
Describe the solution you'd like
Enable simpler user interaction for promotion, perhaps as simple as:

upgrading the default version and
deleting the candidate version

Other Information
Most of the user effort is in checking for version readiness and setting the label iter8.tools/abn.
If Iter8 could compute version readiness itself, then the user interaction would be simplified to just 2 steps as desired.
To compute readiness, the Iter8 service needs to know, in advance, all of the resources that comprise a version since it is necessary to determine when all of the necessary resources are available. Iter8 must be able to identify for each application/track pair the list of resources (GVR, namespace/name)
The user can provide this list as part of the configuration of the Iter8 Service. For example, the following configuration might suffice:
ns:
  hello:
    baseline:
    - group: apps
      version: v1
      resource: deployments
      name: hello-deployment
    - version: v1
      resource: services
      name: hello-svc
    candidate:

User configuration can be simplified if several simplifying assumptions are made:

All versions of an application are deployed to the same namespace
Tracks have names are fixed as: baseline, candidate-0, candidate-1, etc
There is a single resource of each type
Resource names have the form <app_name>-<resource_type>
There is a known mapping between a set of “resource types” to GVR

Then the user configuration might be as simple as:
--set ns1.hello=service-deploy
--set ns2.goodbye.knative=goodbye

This approach simplifies the steps needed to promote a version. However, it is more complex to specify a resources to watch. Indeed, for each version, all the resources should be explicitly identified by type and name (instead of just type).
This can be simplified by assuming a naming convention such as: `<app_name>-track-<track_index>’.
Configuration can then be specified something like:
--set ns1.hello=service-deploy \
--set ns2.recommend=kserve

In this case, the Iter8 service will watch

service and deployment resources named hello-track-i in the namespace ns1.
knative service resources named recommend-track-i in the namespace ns2
Here the i is a track index.

List of work needed:

Changes in hub — handle config & RBAC template
Changes in Iter8 — configuration of the informers change; in order to determine readiness of a track — check if each of its trigger resource exists, check if its readiness conditions are satisfied, has right ABn labels (e.g., version labels), has a finalizer string on it (-- iter8.tools/abn), does not have deletionTimestamp on it (readiness conditions need implementation; base can provide some subroutines);
Changes in labels — no more ready or track label; only version label is required (track index is now part of the k8s resource object name)
Finalizer is always required in every trigger resource for every track (never remove it yourself after adding it; dear user — because, removal of finalizers are always meant to be done by the finalizer controllers).
Trigger handler — if there is a deletionTimestamp for any trigger for a track — time to delete this track — remove the track from Iter8 ABn state — and removes the finalizer from the trigger objects for the track (more RBACs are needed for this to work).

Given a change in resource, we

using name/namespace lookup configuration to identify app/track and from that all resources
Get these resources from the cluster (if some missing, not ready)
Evaluate for readiness
If all ready then record as ready
otherwise, not ready

To evaluate readiness:
for all resources r specified for track:
  get resource
  if not present → return not ready
  if being deleted → return not ready
  return isReady (r)

This has been released in v0.13


Log levels order

Hi,
Shouldn't the log levels be ordered more traditionally with 'info' be less than 'debug' ? While in production, setting the log level to 'debug' seems inappropriate, while by shrinking it to 'warn' you currently is missing important 'info' messages (server startup, socket handshakings, etc…).

You should really consider fixing this.  The debug/info log level order is different from practically every other framework out there (and really doesn't make sense).
For breaking changes like this, do what the Rails team does: add a deprecation warning and a provide a configuration variable that lets a developer jump to the new behavior early, and also allows someone to stay on the old behavior if necessary once you change the defaults.


feature: support alternative dashboard layout

Task/Issue URL: https://app.asana.com/0/0/1205648391652872/f
Description
The updated Dashboard will have the following new interface, designed to allow platforms to opt-in to additional UI features/layouts in a unified way. For now just primaryScreen.layout is supported to enable an experiment to increase adoption of the protections toggle
/**
 * Call this function to provide the dashboard with RemoteFeatureSettings
 */
declare function onChangeFeatureSettings(settings: RemoteFeatureSettings): void

/**
 * This describes the payload for feature settings
 */
export interface RemoteFeatureSettings {
  primaryScreen?: PrimaryScreen;
}

export interface PrimaryScreen {
  /**
   * A string to represent different screen layouts
   */
  layout: "default" | "highlighted-protections-toggle";
}

To provide different layout values, the PR includes support for a new Privacy Configuration key, privacyDashboard

Example PR adding the dashboard to privacy config the https://github.com/duckduckgo/privacy-configuration/pull/1349
Example PR enabling the experiment https://github.com/duckduckgo/privacy-configuration/pull/1350

So, the experiment will be enabled remotely, and then the PrivacyDashboardViewModel will use the values from the privacy config to choose which layout the dashboard should show.

Pixel Params
This PR also adds the param dashboard_highlighted_toggle, to the following pixels

mp_wla (adding to the allowlist, pressing the toggle)
mp_wlr (removing from the allowlist)
m_bsr (feedback form is submitted)
Privacy Triage: https://app.asana.com/0/0/1205648391652879/f

Steps to test this PR
Feature 1

[ ]
[ ]

UI changes

Normal
Experiment

Current dependencies on/for this PR:

develop

PR #3656   👈

This comment was auto-generated by Graphite.

@shakyShane did you mean to update all those assets in this PR? I’m all in for removing 20k lines of code, but it doesn’t seem to be the main goal of this change.


RFC: possibility to log obtained data

Hi! I figured out that there is one main difference between @urql/react and @urql/svelte. Its about logging requested data - in react I can see in console what data came to me and when, but in svelte I cannot do that. It always shows {
isPaused,
pause,
resume,
subscribe
} when using queryStore({}) and {
subscribe
} when using mutationStore({}). I can see received data only when its rendered or in network.
Summary
It would be good if logging of obtained data into console would be available
Proposed Solution
Perhaps just rewriting react source code into svelte would do
Requirements

Hiya 👋
Sorry, heads up, I'll proactively close this issue after answering, because the necessary features for this already exist. However, feel free to continue using this thread for any ongoing discussion or questions ✌
We currently already have two main tools available for this:

The debugExchange: Which can be imported from core and added in the list of exchanges (depending on where it's placed it can also see operations before deduplication) https://formidable.com/open-source/urql/docs/api/core/#debugexchange
The urql devtools: Which are a browser extension and an exchange together, visualising any incoming or outgoing activity https://github.com/FormidableLabs/urql-devtools

These two basically expand on information your can gather on what urql gives you beyond the browser network devtools.
If you then need to get information about your bindings, we prefer to recommend manual logging tools (in your case, just subscribe to the store and then log), or your framework's browser devtools


TokenCleanupService using full table scan

We have tens of millions of rows in the PersistedGrants table and this block is resulting in a full scan during clean up:
https://github.com/IdentityServer/IdentityServer4/blob/07898a61dc22ce9a95e5fc5611fa674feac4230e/src/EntityFramework.Storage/src/TokenCleanup/TokenCleanupService.cs#L77-L81
Sorting by the same column used in the predicate fixes it
                var expiredGrants = await _persistedGrantDbContext.PersistedGrants
                    .Where(x => x.Expiration < DateTime.UtcNow)
                    .OrderBy(x => x.Expiration)
                    .Take(_options.TokenCleanupBatchSize)
                    .ToArrayAsync();

Submitted PR 5092

Hmm, I guess there is an index on the key, but given that that's the ordering it still needs to do a full table scan, despite there being a index on expiration?

It's not always a full scan, but it does read many more rows than it needs to (see the query plan linked in the PR).
If the optimizer chose to use the index on Expiration, it would have to read all the expired rows, and then re-order them based on the key before taking the top TokenCleanupBatchSize.  This would be memory intensive and probably spill over into tempdb.  The optimizer (wisely) decides against this and instead runs through the key index in the correct order, throwing away records that don't meet the criteria, and stopping when it has returned TokenCleanupBatchSize number of records.
In application terms, the job has to read and skip over all the older but longer living tokens (refresh) to get to the newer expired (access) tokens.  Also, any time there are fewer expired records than TokenCleanupBatchSize it will do a full scan.  This will happen regularly if the clean up job is keeping up.
Ordering and filtering by the Expiration, the query will only need to read TokenCleanupBatchSize rows.

Yep, the sort by expiration is certainly the better choice. We'll get that merged in soon.

I'm having the exact same issue as described by @edmacdonald,  and did some tests with the purposed fix.
It actually gives a great performance boost!
Is there any expectation of this being merged really soon ?  we are start facing some nasty problems .
I'm seeing that this is still waiting for a review in  PR https://github.com/IdentityServer/IdentityServer4/pull/5092

Many thanks

thanks @edmacdonald for the info .  @brockallen  is there any chance that this PR will be merged soon ?