[SPARK-27188][SS] FileStreamSink: provide a new option to have retention on output files

What changes were proposed in this pull request?
This patch proposes to provide a new option to specify time-to-live (TTL) for output file entries in FileStreamSink.
The metadata log greatly helps to easily achieve exactly-once but given the output path is open to arbitrary readers, there's no way to compact the metadata log, which ends up growing the metadata file as query runs for long time, especially for compacted batch.
(There're some reports from end users which include their workarounds: SPARK-24295)
This patch will filter out outdated output files in metadata while compacting batches, which helps metadata to not grow linearly, as well as filtered out files will be "eventually" no longer seen in reader queries which leverage File(Stream)Source.
How was this patch tested?
Added unit tests.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Test build #103609 has started for PR 24128 at commit 3e82d95.

cc. @tdas @zsxwing @jose-torres @arunmahadevan @gaborgsomogyi

Test build #103609 has finished for PR 24128 at commit 3e82d95.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103609/
Test PASSed.

The FileStreamSink was designed to work with the metadata log. I'm not comfortable adding an option to just turn it off; there are all sorts of ways that could cause more subtle issues than at-least-once semantics.

I'm not comfortable adding an option to just turn it off; there are all sorts of ways that could cause more subtle issues than at-least-once semantics.

I completely understand about uncomfortable of disabling the metadata, but as I described in JIRA issue and description of PR there's no workaround except letting end users deal with dirty thing by their hands. I'd give it another try to let FileStreamSink checks deleted output files in background and exclude when compacting metadata (I guess it's ideal one to go), but that definitely brings overhead and maybe some configurations as well.
Regarding subtle issues it would be better for us to share possible issues (instead of 'something might happen') if we can imagine any: it would help to lead our direction to the right way.

Btw, my 2 cents, ideally output of FileStreamSink should be able to also read from other than Spark, which means metadata should only affect semantics. End users may try to do it: mail thread in previous comment is one of example - querying output with Hive.

Another possible workaround is alternative of #23840 - add an option to force FileStreamSink to filter out all entries when compacting batches. It definitely needs option to source side which force File(Stream)Source to ignore metadata. I'm not sure the new option is pretty different from this proposal though.

There's also another possible workaround - if we really feel OK to exclude old output files in metadata even they're not actually deleted in its directory, we can expose option to set retention policy (mostly time to live) and force FileStreamSink to filter out entries which becomes old based on TTL. Readers cannot read some of output files even these are not deleted, but it's a policy being set from end users, so that might be OK. We can still expose option for File(Stream)Sources to ignore metadata.
So we have some alternatives on this patch and all of things have its trade-off. What's our preference? I believe this is ongoing issue end users are already struggling with, so we have to take one of approach even it's not that ideal.

I'm not sure I agree with the premise. Adding a feature, even behind a flag, represents a judgment by the Spark community that the feature works well and an ongoing obligation for Spark committers to maintain and fix bugs in the feature. We shouldn't implement ad-hoc changes with unclear behavior and semantics just to fix an immediate symptom some end-users face.
A retention policy generally makes sense to me, although I don't personally know enough about how Hadoop-compatible filesystems deal with timestamps to effectively review such a change.

This is a commit based on master for the last alternative (exclude old output files in metadata):
https://github.com/HeartSaVioR/spark/commit/8564ab44a45db0cbf5d09dcc3b3c993f440b264e
I guess this is the least intrusive approach and simple. One thing to note is that excluding output files are reflected when compacting batches so it depends on compaction interval as well as batch interval. We can safely document it as "eventually happens".
@jose-torres What do you think about this approach?

We shouldn't implement ad-hoc changes with unclear behavior and semantics just to fix an immediate symptom some end-users face.

I tend to agree your point, but what I would like to do is investigating production issues end-users are facing and providing possible solutions which even may be seen as a workaround. We can still reject the approach which doesn't feel safe and clear. We might want to share why it is not safe/clear then, so that we don't take that way again.

Test build #103643 has started for PR 24128 at commit 8564ab4.

Rebased to the approach: applying retention.

Test build #103643 has finished for PR 24128 at commit 8564ab4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103643/
Test PASSed.

Test build #103656 has started for PR 24128 at commit db2fa60.

Test build #103656 has finished for PR 24128 at commit db2fa60.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103656/
Test FAILed.

retest this, please

Test build #103662 has started for PR 24128 at commit db2fa60.

Test build #103662 has finished for PR 24128 at commit db2fa60.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103662/
Test PASSed.

Could I kindly ask for reviewing on new approach? That would not be intrusive unless end users configure the retention badly.

Kindly reminder.

Ping again, as Spark+AI Summit 2019 in SF is end.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

@tdas @zsxwing @jose-torres @gaborgsomogyi Kindly reminder.

Ping.

Can one of the admins verify this patch?

ok to test

Test build #110688 has started for PR 24128 at commit db2fa60.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/15850/
Test PASSed.

Test build #110688 has finished for PR 24128 at commit db2fa60.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110688/
Test FAILed.

Test build #110713 has started for PR 24128 at commit 2a3bad5.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/15869/
Test PASSed.

Test build #110713 has finished for PR 24128 at commit 2a3bad5.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110713/
Test FAILed.

retest this, please

Test build #111279 has started for PR 24128 at commit 2a3bad5.

Test build #111279 has finished for PR 24128 at commit 2a3bad5.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

retest this, please

Test build #111307 has started for PR 24128 at commit 2a3bad5.

Test build #111307 has finished for PR 24128 at commit 2a3bad5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111307/
Test PASSed.

SPARK-29995 is just filed which denotes same issues SPARK-24295.

There are many end users facing this problem. So I give +1 to resolve this problem. But the ttl option can just resolve the problems of some scenarios. This dose not resolve scenarios where end-users does not set ttl, or just want to read files from earliest offset with ttl set.

Maybe we can differentiate two major cases:

downstream query to read the output directory is also Spark (leverages metadata)

In this case, technically we never be able to delete any entries in metadata if we want to ensure the downstream query provides same result during multiple runs (unless inputs are added in real time).
We know that's only ideal - if the streaming query runs longer and writes gigantic number/size of files for a long time, we would want to get rid of some part to gain speed and save storage with fully understanding that we are throwing out some inputs which will affect the result of query.
Assume we decided to get rid of some output files. How to do it safely? The only safe way to do it is, getting rid of them in metadata first, and delete actual files. (Downstream query relies on the metadata to get the list of files, so if we don't make sure deleting them in metadata first, the downstream query will try to read the file which no longer exist, and fails - depending on the option.)
That means running streaming query should deal with the deletion, as we don't have any official offline tool to modify metadata, and you may find difficulties to "how" to let streaming query know which files to delete. That's why I just simply pick "retention" which is generally acceptable approach (Kafka also applies retention policy by default).

we never let Spark read the output directory - we let other frameworks to read the directory

In this case we don't need to build metadata - though this means end users will need to deal with "at-least-once" guarantee. Given the file sink doesn't overwrite the file, it may leave corrupted records on partial output as well. If that's acceptable, we may be able to add an option to "disable" metadata, though there was some comments worried about doing it: https://github.com/apache/spark/pull/24128#issuecomment-474109068
So I guess there're not many options here and I guess I picked the viable one, but I'd be really appreciated for more ideas!

IMHO, the core problem is the compact metadata log grows bigger and bigger, and it is a time-consuming work to compact the metadata log. So why not limit the size of compact metadata log? I mean we can split the compact log to multiple log files. We may just need write limited sink file path into single compact log. And every 10 batches, we just compact delta logs with latest compact log.
batch N:

10 11 12.compact

batch N+10

10 11 12.compact 13 14 ... 22.compact.1 22.compact

batch N+20

10 11 12.compact 13 14 ... 22.compact.1 22.compact 23 24 ... 32.compact.1 32.compact

batch N+30

10 11 12.compact 13 14 ... 22.compact.1 22.compact 23 24 ... 32.compact.1 32.compact 33 34 ... 42.compact.2 42.compact.1 42.compact

In batch N+10, we rename 12.compact to 22.compact, and compact delta log (13,14 ... 21) into 22.compact. If the size of 22.compact is beyond the limit, we need to 22.compact to 22.compact.1 and then create a new 22.compact. The cost of rename is low or not higher than creating new file. And we just need to compact delta logs with latest compact log file. The biggest gain is the time cost of compacting is predictable, and does not scales linearly for increases.
But this may broke the compatibility when use old version spark to read stream sink files. This is just my rough ideas, please advise if there is any mistake.

IMHO, the core problem is the compact metadata log grows bigger and bigger, and it is a time-consuming work to compact the metadata log, because it will read old compact log file and then write to new compact log file.

I agree with you that the problem is that compact metadata log just grows most of the times, though taking plenty of time building metadata log is just a one of multiple major issues. The other major issue, reading metadata log won't decrease unless we optimize the format of file or just get rid of entities like this patch is proposing.
One thing we have to consider is, when compact phase happens, Spark is able to get rid of some entities which have been existing - that's the feature this patch leverages. That requires full read and rewrite of entities per each compact phase, and that's why we can't just simply add two compact files.
Looks like CompactibleFileStreamLog is introduced to avoid "small files problem", which seems to be possible to tweak a bit to change the approach to maintain "ranged delta" which might be more similar with what you proposed. That's no longer be a "snapshot", but in most cases the entities are not removed so it also makes sense to me. I'm expecting the logic more complicated than current one, but that might be acceptable given the issue has been affecting badly for end users.

@tdas @zsxwing @jose-torres @gaborgsomogyi Kindly reminder.

@uncleGen
Hi, do you plan to go ahead with your idea? I have been thinking about this issue, and your idea seems to be a realistic solution which doesn't introduce too much changes. While we may also want to find the solution which could deal with most of things, but for now it would be great even only with your idea.
Otherwise, would you mind if I pick your idea up if you're not planning to do it?

@HeartSaVioR OK

@HeartSaVioR
Would you mind if I ask to elaborate your answer? IMHO it's not clear which one (or both?) you are OK with.

@HeartSaVioR Sorry to make you confused! Feel free to pick my idea please.


Segfault when trying to manually build a dynamo-like function

I tried to disassemble and understand the process of how dynamos are working, and reasoned about what would be equivalent the following dynamo:
IRTools.@dynamo function test_dynamo(a...)
    ir = empty(IRTools.IR(a...))
    push!(ir, IRTools.xcall(Main, :println, "hi"))
    IRTools.return!(ir, nothing)
    return ir
end

Naively substituting the code pieces in dynamo.jl, I thought the following should work:
@generated function test1(f, args...)
    ir = IRTools.empty(IRTools.IR(f, args...))
    push!(ir, IRTools.xcall(Main, :println, "hi"))
    IRTools.return!(ir, nothing)

    m = ir.meta::IRTools.Meta
    ir = IRTools.varargs!(m, ir)
    IRTools.argnames!(m, :args)
    _self = IRTools.splicearg!(m, ir, Symbol("#self#"))
    IRTools.prewalk!(x -> x === IRTools.self ? _self : x, ir)
    @show result = IRTools.update!(m.code, ir)
    return result
end

But instead, it crashes:
julia> test1(identity, 1)
result = IRTools.update!(m.code, ir) = CodeInfo(
 1 ─      (Main.println)("hi")
 │   %2 = nothing
 └──      return %2
)
Internal error: encountered unexpected error in runtime:
BoundsError(a=Array{Core.Compiler.SlotInfo, (2,)}[
  Core.Compiler.SlotInfo(defs=Array{Int64, (0,)}[], uses=Array{Int64, (0,)}[], any_newvar=false),
  Core.Compiler.SlotInfo(defs=Array{Int64, (0,)}[], uses=Array{Int64, (0,)}[], any_newvar=false)], i=(Core.Compiler.UnitRange{Int64}(start=1, stop=3),))
rec_backtrace at /buildworker/worker/package_linux64/build/src/stackwalk.c:94
record_backtrace at /buildworker/worker/package_linux64/build/src/task.c:246
jl_throw at /buildworker/worker/package_linux64/build/src/task.c:577
throw_boundserror at ./abstractarray.jl:484
checkbounds at ./abstractarray.jl:449 [inlined]
getindex at ./array.jl:737 [inlined]
scan_slot_def_use at ./compiler/ssair/slot2ssa.jl:51
just_construct_ssa at ./compiler/ssair/driver.jl:103
run_passes at ./compiler/ssair/driver.jl:114
optimize at ./compiler/optimize.jl:162
typeinf at ./compiler/typeinfer.jl:35
typeinf_ext at ./compiler/typeinfer.jl:567
typeinf_ext at ./compiler/typeinfer.jl:604
jfptr_typeinf_ext_1 at /usr/local/julia-1.0.0/lib/julia/sys.so (unknown line)
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2182
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1536 [inlined]
jl_apply_with_saved_exception_state at /buildworker/worker/package_linux64/build/src/rtutils.c:257
jl_type_infer at /buildworker/worker/package_linux64/build/src/gf.c:275
jl_compile_method_internal at /buildworker/worker/package_linux64/build/src/gf.c:1784 [inlined]
jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1828
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2182
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:324
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:428
eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:363 [inlined]
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:686
jl_interpret_toplevel_thunk_callback at /buildworker/worker/package_linux64/build/src/interpreter.c:799
unknown function (ip: 0xfffffffffffffffe)
unknown function (ip: 0x7f69e453496f)
unknown function (ip: 0xffffffffffffffff)
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:808
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:787
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/builtins.c:622
eval at ./boot.jl:319
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2182
eval_user_input at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/REPL/src/REPL.jl:85
run_backend at /home/philipp/.julia/packages/Revise/agmgx/src/Revise.jl:949
#75 at ./task.jl:259
jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1829
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2182
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1536 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:268
unknown function (ip: 0xffffffffffffffff)
result = IRTools.update!(m.code, ir) = CodeInfo(
 1 ─      (Main.println)("hi")
 │   %2 = nothing
 └──      return %2
)

signal (11): Segmentation fault
in expression starting at no file:0
cast_or_null<llvm::Instruction, llvm::Instruction> at /buildworker/worker/package_linux64/build/usr/include/llvm/Support/Casting.h:298 [inlined]
emit_function at /buildworker/worker/package_linux64/build/src/codegen.cpp:6552
jl_compile_linfo at /buildworker/worker/package_linux64/build/src/codegen.cpp:1159
jl_generate_fptr at /buildworker/worker/package_linux64/build/src/codegen.cpp:1394
jl_compile_method_internal at /buildworker/worker/package_linux64/build/src/gf.c:1821 [inlined]
jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1828
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2182
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:324
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:428
eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:363 [inlined]
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:686
jl_interpret_toplevel_thunk_callback at /buildworker/worker/package_linux64/build/src/interpreter.c:799
unknown function (ip: 0xfffffffffffffffe)
unknown function (ip: 0x7f69e453496f)
unknown function (ip: 0xffffffffffffffff)
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:808
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:787
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/builtins.c:622
eval at ./boot.jl:319
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2182
eval_user_input at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/REPL/src/REPL.jl:85
run_backend at /home/philipp/.julia/packages/Revise/agmgx/src/Revise.jl:949
#75 at ./task.jl:259
jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1829
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2182
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1536 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:268
unknown function (ip: 0xffffffffffffffff)
Allocations: 21604532 (Pool: 21601220; Big: 3312); GC: 48
[1]    26428 segmentation fault (core dumped)  julia

No idea if that's a legitimate use case that should have worked.  What I actually wanted to achieve was to get a dynamo that adds a new argument to the given method.

This is possible, of course, but really fiddly. @dynamo exists because it's pretty much impossible to get this right first try (it always takes me multiple attempts as well).
We could dig into why this isn't working, but if you just want to add a new argument you probably don't need to, you can do this with @dynamo. You just need something like
@dynamo function foo(f, cx, args...)
  ir = IR(f, args...)
  cx = argument!(ir, at = 1)
  ...
end

Hm. For some reason, just using argument! didn't work when I tried it, but now it does :)  Thanks, that question is settled now.


Return params instead of cloning it

I tried the latest version and is not setting any search param, not even the scope!
But I copied to code to my app and did this change and it worked, looks like doing new URLSearchParams(params) is, for some reason, not copying the params correctly.

Thanks for test that! Weird thing 😅


Fix NPE in :performance:test when building in detched HEAD state

When building in a detached HEAD state (eg. after git checkout REL_2.5) a number of tests in :performance:test fail with a NPE in Git.groovy as follows:
can use 'last' baseline version to refer to most recently released version

java.lang.NullPointerException: Cannot get property 'objectId' on null object
    at org.gradle.performance.fixture.Git.<init>(Git.groovy:37)
    at org.gradle.performance.fixture.Git.current(Git.groovy:28)
    at org.gradle.performance.fixture.CrossVersionPerformanceTestRunner.run(CrossVersionPerformanceTestRunner.groovy:64)
    at org.gradle.performance.fixture.CrossVersionPerformanceTestRunnerTest.can use 'last' baseline version to refer to most recently released version(CrossVersionPerformanceTestRunnerTest.groovy:88)

Hey @tonyabbott, could you sign the CLA? http://gradle.org/contributor-license-agreement/

Hi @big-guy - We signed a corporate agreement last week that covers me, company name is Attix5. Let me know if there's a problem with it

Thanks.  Found it, I was looking for an individual contributor CLA.


Support for imagePullSecrets For fluentBit daemonset

This PR add supports for imagePullSecrets for FluentBit Daemonset.
issue - https://github.com/kubesphere/fluentbit-operator/issues/92

/cc @wanjunlei

Some changes are overlapping with https://github.com/kubesphere/fluentbit-operator/pull/93

I guess #93 is limited to operator and this introdues changes for daemonset

Thanks a lot. This may need to be revised as well(fluentbit-operator/api/fluentbitoperator/v1alpha2/zz_generated.deepcopy.go )
like this
if in.ImagePullSecrets != nil {
in, out := &in.ImagePullSecrets, &out.ImagePullSecrets
*out = make([]v1.LocalObjectReference, len(*in))
copy(*out, *in)
}

ohh I thought It was auto-generated. I will update

@wenchajun I have updated it, Can you please have a look?

You need to run make manifest and update the files.

@wanjunlei  Updated files.


About WaterNeRF

Hello, I'm also currently trying to reproduce WaterNeRF, and I've encountered some issues in the process. May I ask if I can seek your assistance or advice?

replied in your email thread. thanks for interests


pull request for e2e test at 01/03/2017 15:15:07

:white_check_mark: Validation status: passed
For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Add support for DuckDB struct literal syntax

Fixes #1129
Existing Struct expression supports the BigQuery semantics, which allow for optionally typed fields, that if typed, also have an optional name.
DuckDB has incompatible semantics: only named fields without any type specification.
Should I create a separate expression, perharps NamedFieldsStruct, or change the Struct to have an inner enum Struct(enum { MaybeTyped(...), Named(...) }) ?
Thanks!

You can ask the question in issue. The PR draft doesn't send the notification.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

105 of 111 (94.59%)  changed or added relevant lines in 4 files are covered.
25 unchanged lines in 1 file lost coverage.
Overall coverage increased (+0.03%) to 87.909%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/ast/mod.rs
10
13
76.92%

src/parser/mod.rs
22
25
88.0%

Files with Coverage Reduction
New Missed Lines
%

tests/sqlparser_bigquery.rs
25
89.28%

Totals

Change from base Build<PHONE_NUMBER>:
0.03%

Covered Lines:
20583

Relevant Lines:
23414

💛  - Coveralls

Should I create a separate expression, perharps NamedFieldsStruct, or change the Struct to have an inner enum Struct(enum { MaybeTyped(...), Named(...) }) ?

I think given the difference in syntax, having a new struct like NamedFieldsStruct would be better / less confusing / easier to handle
Thank you again for working on this @gstvg  and @jayzhan211

thanks @jayzhan211 and @alamb
I created a new Expr named Dictionary, which currently only supports struct literals.
I tried adding supporting for DuckDB MAP too, but since map keys can be any expression, it conflicted with Snowflake-only JsonAccess with colon json_field:key, which is currently parsed for all dialects. Restricting it to Snowflake solves the problem. Should i push it here PR too?

thanks @jayzhan211 and @alamb I created a new Expr named Dictionary, which currently only supports struct literals.
I tried adding supporting for DuckDB MAP too, but since map keys can be any expression, it conflicted with Snowflake-only JsonAccess with colon json_field:key, which is currently parsed for all dialects. Restricting it to Snowflake solves the problem. Should i push it here PR too?

I don't understand what you mean by Snowflake-only JsonAccess with colon json_field:key, can you elaborate it more.
It seems the difference between struct and map in Duckdb is that the key_type can be different for map in each row. They are both json-like objects.

Sure. I would like to solve all this at once, to be honest.
Snowflake JSON access uses : to query data

https://docs.snowflake.com/en/user-guide/json-basics-tutorial-query
DuckDB maps key can be any type, not only strings, but may have different set of keys for each row

https://duckdb.org/docs/sql/data_types/map.html
To support any type of key, I used Parser::parse_expr instead of parse_identifier
Currently, because of the following arms in parse_infix and get_next_precedence, when parse_expr is called to get the key expr, it parses the whole key: value as a JsonAccess
If those two arms are moved to the Snowflake dialect, or conditioned to a dialect_of!(Snowflake), parse_expr will return the key only, not the whole key value, but the json_column:json_key would be restricted to the Snowflake dialect
I didn't find any other way to do that
I've pushed it to another branch if you have interest

which is currently parsed for all dialects

I think we have not yet supported any.

Sorry, I mean the json_column:json_key is parsed for all dialects, not restricted to the Generic and/or Snowflake

Others dialects that uses this syntax:
Snowflake

Syntax limited to Object constants, so parse_value or parse_identifier can be called instead of parse_expr
ClickHouse

Has the same conflicts as DuckDB map, which also means it has the same solution 😀

How about considering Expr::JsonAcess as your Expr::DictionaryField, it seems to me they are equivalent but the json operator is colon. Similar to parsing [e1, e2...] but {e1, e2...} instead.
Expr::Dictionary(Vec<Expr>), where we can get Expr::JsonAccess (left colon right)

I met the issue while trying out my idea #1197

I also pushed d9d16ce to fix the doc test failues

@jayzhan211  does this change look good for you?

I think either struct and map and other dictionaries like syntax {"key": "value"} is possible to share the same parsing rule.
It is fine to do it specifically for now, but I think we can share the similarity for other dialects or syntax later on.

Thanks @jayzhan211  and @gstvg !

How about considering Expr::JsonAcess as your Expr::DictionaryField, it seems to me they are equivalent but the json operator is colon. Similar to parsing [e1, e2...] but {e1, e2...} instead.
Expr::Dictionary(Vec<Expr>), where we can get Expr::JsonAccess (left colon right)

Good idea. But making Dictionary a Vec<Expr> wouldn't be unergonomic for users? Extending it, what about something like this:
    fn parse_duckdb_dictionary_field(&mut self) -> Result<DictionaryField, ParserError> {
        if let Expr::JsonAccess { left, operator: JsonOperator::Colon, right } = self.parse_expr()? {
            Ok(DictionaryField {
                key: left,
                value: right,
            })
        } else {
            self.expected(":", self.peek_token())
        }
    }

How about considering Expr::JsonAcess as your Expr::DictionaryField, it seems to me they are equivalent but the json operator is colon. Similar to parsing [e1, e2...] but {e1, e2...} instead.
Expr::Dictionary(Vec<Expr>), where we can get Expr::JsonAccess (left colon right)

Good idea. But making Dictionary a Vec<Expr> wouldn't be unergonomic for users? Extending it, what about something like this:
    fn parse_duckdb_dictionary_field(&mut self) -> Result<DictionaryField, ParserError> {
        if let Expr::JsonAccess { left, operator: JsonOperator::Colon, right } = self.parse_expr()? {
            Ok(DictionaryField {
                key: left,
                value: right,
            })
        } else {
            self.expected(":", self.peek_token())
        }
    }

It looks good to me.


Update Custom Step : Add privacy risk measure for SMOTE (SDG)

Please include answers to these questions as part of your pull request
In the GitHub webUI, use the Write tab to modify the Markdown text that is part of the pull request. For each question simply place an X inside the square brackets, [X], that represents your answer. Make sure there are no blanks inside the brackets, otherwise MarkDown doesn't render properly. Using the Preview tab while editing this form, you can see the formatted/rendered version of the message.

Q1: Confirm that you have the right to submit the code that is being contributed. Please consider the origin of your code and confirm you have the appropriate rights to make the submission subject to the Apache 2.0 license that applies to everything in this repository of custom steps. If so, follow the instructions for the Contributor Agreement (which is based on the industry-standard Developer Certificate of Origin (DCO)).

[X] Yes, I have the right to submit the contributed code on behalf of myself, my company, or any other owner of the code. I have also attached my signed copy of the DCO to this message.
[ ] No

Q2: Confirm that your contribution does not include any personally identifiable information (PII), for example, in any examples used in your README file.

[X] My contribution does NOT include PII data
[ ] My contribution includes PII data

Q3: Confirm your contribution does not include any encryption or other export-controlled technology.

[X] My contribution does NOT contain encryption or other export-controlled technology
[ ] My contribution includes encryption or other export-controlled technology

ContributorAgreement.txt

@josiah-chua, as this is your first contribution (code which helped towards the privacy risk calc), just informing you that you need to attach a Contributor Agreement.

Note the text file above
Make a copy of the same
Change last line to include your email address and name
Attach to this PR.

Thanks.

ContributorAgreement.txt


Add a memory limit processor

This processor would stop the consumer when the memory usage gets too close from the PHP memory limit, to provide a graceful shutdown (with logging) in such case instead of waiting until the memory limit is reached and the process crashes. This would help in case your processor is leaking memory.
what do you think @odolbeau ?

Yes, it's definitively a good idea!


nix compile warning resolved

@acautin

there are still some warnings (maybe outside the scope of this pr)

Also with -O2 in CentOs6
$ make -f c_src/Makefile
cd c_src/odpi && make
make[1]: Entering directory `/home/bikram/projects/oranif/c_src/odpi'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory `/home/bikram/projects/oranif/c_src/odpi'
gcc -o priv/dpi_nif.so -ggdb -O2 -Wall -fPIC -shared -std=c11 -Ic_src -I"/usr/lib/erlang/erts-9.2/include" -I"c_src/
odpi/include" -Lc_src/odpi/lib c_src/dpiConn_nif.c c_src/dpiContext_nif.c c_src/dpiData_nif.c c_src/dpi_nif.c c_src/
dpiQueryInfo_nif.c c_src/dpiStmt_nif.c c_src/dpiVar_nif.c -lodpic
c_src/dpiConn_nif.c: In function ‘conn_close’:
c_src/dpiConn_nif.c:317:18: warning: ‘m’ may be used uninitialized in this function [-Wmaybe-uninitialized]
c_src/dpiStmt_nif.c: In function ‘stmt_execute’:
c_src/dpiStmt_nif.c:42:18: warning: ‘m’ may be used uninitialized in this function [-Wmaybe-uninitialized]
c_src/dpiStmt_nif.c: In function ‘stmt_bindValueByPos’:
c_src/dpiStmt_nif.c:187:5: warning: ‘bindType’ may be used uninitialized in this function [-Wmaybe-uninitialized]
c_src/dpiStmt_nif.c: In function ‘stmt_bindValueByName’:
c_src/dpiStmt_nif.c:213:5: warning: ‘bindType’ may be used uninitialized in this function [-Wmaybe-uninitialized]
c_src/dpiStmt_nif.c: In function ‘stmt_defineValue’:
c_src/dpiStmt_nif.c:332:5: warning: ‘nativeType’ may be used uninitialized in this function [-Wmaybe-uninitialized]
c_src/dpiStmt_nif.c:332:5: warning: ‘oraType’ may be used uninitialized in this function [-Wmaybe-uninitialized]

@acautin Can you please try once more with Mac?

@c-bik this is how it looks now in OS X:
cd c_src/odpi && make
make[1]: Nothing to be done for `all'.
gcc -o priv/dpi_nif.so -Ic_src -I"/Users/agustin/kerl/20.2/erts-9.2/include" -I"c_src/odpi/include" -O2 -ggdb -Wall -fPIC -std=c11 -dynamiclib  c_src/dpiConn_nif.c c_src/dpiContext_nif.c c_src/dpiData_nif.c c_src/dpiQueryInfo_nif.c c_src/dpiStmt_nif.c c_src/dpiVar_nif.c c_src/dpi_nif.c -Lc_src/odpi/lib -shared -lodpic -flat_namespace -undefined suppress
c_src/dpiData_nif.c:187:39: warning: incompatible pointer types passing 'int64_t *' (aka 'long long *') to parameter of type 'long *' [-Wincompatible-pointer-types]
    if (!enif_get_int64(env, argv[1], &amount))
                                      ^~~~~~~
/Users/agustin/kerl/20.2/erts-9.2/include/erl_nif_api_funcs.h:104:79: note: passing argument to parameter 'ip' here
ERL_NIF_API_FUNC_DECL(int,enif_get_long,(ErlNifEnv*, ERL_NIF_TERM term, long* ip));
                                                                              ^
1 warning generated.

@acautin

@c-bik this is how it looks now in OS X:
cd c_src/odpi && make
make[1]: Nothing to be done for `all'.
gcc -o priv/dpi_nif.so -Ic_src -I"/Users/agustin/kerl/20.2/erts-9.2/include" -I"c_src/odpi/include" -O2 -ggdb -Wall -fPIC -std=c11 -dynamiclib  c_src/dpiConn_nif.c c_src/dpiContext_nif.c c_src/dpiData_nif.c c_src/dpiQueryInfo_nif.c c_src/dpiStmt_nif.c c_src/dpiVar_nif.c c_src/dpi_nif.c -Lc_src/odpi/lib -shared -lodpic -flat_namespace -undefined suppress
c_src/dpiData_nif.c:187:39: warning: incompatible pointer types passing 'int64_t *' (aka 'long long *') to parameter of type 'long *' [-Wincompatible-pointer-types]
    if (!enif_get_int64(env, argv[1], &amount))
                                      ^~~~~~~
/Users/agustin/kerl/20.2/erts-9.2/include/erl_nif_api_funcs.h:104:79: note: passing argument to parameter 'ip' here
ERL_NIF_API_FUNC_DECL(int,enif_get_long,(ErlNifEnv*, ERL_NIF_TERM term, long* ip));
                                                                              ^
1 warning generated.

It seems ERL_NIF_TERM enif_make_int64(ErlNifEnv* env, ErlNifSInt64 i) is internally mapped to enif_get_long(ErlNifEnv*, ERL_NIF_TERM term, long* ip) where (long *) != (int64_t *) (in Mac).
Unfortunately ErlNifSInt64 isn't a public NIF variable (not with the current Erlang version anyway).
Perhaps I can't fix this one yet, have wait for erlang version upgrade and revisit. @acautin Can you please create this into a issue?
and if there  aren't anything else, please approve too

@acautin and @KarlKeiser resolved some merge conflicts with master, can you please test?


CI: golangci is broken with go canary (1.24RC)

Description
The core of the issue is that golangci is unable to work properly with a go version that is higher than the one it has been built with, resulting in hogging memory / CPU.
This can be reproduced locally on a laptop:

install go 1.24RC
download binary curl -sSfLO https://github.com/golangci/golangci-lint/releases/download/v1.62.2/golangci-lint-1.62.2-darwin-arm64.tar.gz

Then:
dmp@Synchronicity:~/Projects/go/nerd/nerdctl$ go version
go version go1.24rc1 darwin/arm64

dmp@Synchronicity:~/Projects/go/nerd/nerdctl$ time GOOS=linux /Users/dmp/tmp/golangci-lint-1.62.2-darwin-arm64/golangci-lint run
^C^C

real	2m10.275s
user	5m25.968s
sys	3m23.894s

vs.
dmp@Synchronicity:~/Projects/go/nerd/nerdctl$ go version
go version go1.23.4 darwin/arm64

dmp@Synchronicity:~/Projects/go/nerd/nerdctl$ time GOOS=linux /Users/dmp/tmp/golangci-lint-1.62.2-darwin-arm64/golangci-lint run

real	0m27.558s
user	1m38.488s
sys	0m47.445s

I have no idea what kind of voodoo golangci is doing, but clearly their binary installs are not usable for our purpose.
Solutions are either: the golangci github action offers to build / install instead of downloading pre-built (and that would work with go1.24), or we figure out how to install manually and remove the action, or we stop linting for canary (would be sad...).
It is unclear to me ATM if there will be unforeseen issues with 1.24 wrt golangci - from a glimpse, it seems that it is just fine and maps are just sugar.
Steps to reproduce the issue
No response
Describe the results you received and expected
na
What version of nerdctl are you using?
main
Are you using a variant of nerdctl? (e.g., Rancher Desktop)
None
Host information
No response

Added notes:

golangci DOES compile just fine against golang 1.24
using the action to go install is a no-go, as you cannot separate GOOS for install and runtime phases... :/

The action install mechanism does not provide much value.
Suggesting we just drop that part and install normally.


Check if IPv6 is enabled before restoring the network pools

Related to https://github.com/docker/docker/issues/26075
Fix reservePool logic to be cover the case where a restored network was created with a IPv6 subnet specified but no --ipv6.
Signed-off-by: Alessandro Boch<EMAIL_ADDRESS>

LGTM

LGTM

FYI: I realize this is correct behaviour but this broke the API as it affected when libnetwork requests an IPv6 address from the IPAM plugin.


SVG Import support

While trying to update the logo in my fork I tried replacing public/HULL-Logo.svg and quickly noticed it did not work, as this was the actual logo being used:
https://github.com/ndimatteo/HULL/blob/e7f5ec9acec80b6e0810bb408d8f9ae53d6811ba/components/icon.js#L5
I was wondering if adding SVG support is something to consider here (I have a working prototype in a branch, it's just basically adding the @svgr/webpack package). It would help with duplications like that and the file becoming cumbersome once you add more SVGs. To keep the color parameter we could just use something like currentColor. It might also be unnecessary optimization.
I'll make a PR if it's something that could be worthwhile.

Hey there @hyldmo,
Thanks for the suggestion! I believe using inline SVG's is a better solution than importing it as a URL like this.
Here's an old (but good) article for some reasons why: https://css-tricks.com/pretty-good-svg-icon-system/
I know it's a bit more cumbersome to have something as simple as a logo abstracted away into a react component with it's innards stripped out, but I find the consistency of using <Icon /> creates a solid pattern for displaying your site's icon system wherever needed, and gives you one place to go to edit/adjust :)


Fix NaN payload unittest in errorfunction (ltsmaster)

#25 retargeted to ldc-ltsmaster.

Since it's already merged upstream, just cherry-pick it from there, that's what I did for my upstream PRs.

That's what I thought I did.  Did I miss something? The cherry-picked commits are what these from https://github.com/D-Programming-Language/phobos/pull/3976/commits

No, I mean cherry-pick and merge without a PR. No need to review again.


Remove references to specific menu entries from menubar.ts

As discussed in https://github.com/Microsoft/vscode/pull/54826
I believe all references to specific menu entries should be removed from menubar.ts
These menu entries should somehow be registered from their respective areas. 'debug' from debug.contribution.ts and so on.

I think this can probably be closed by now, at least I cannot find specific debug entries anymore.

Yes this looks better now, closing


Added Grokking Web Application Security

Hi,
Stjepan here from Manning. I thought this title might be a good match for your list. Thank you for considering it.
Best,

Thank you.
Best,
On Fri, Feb 9, 2024 at 2:02 AM Daniel Romero @.***>
wrote:

Merged #73 https://github.com/infoslack/awesome-web-hacking/pull/73
into master.
—
Reply to this email directly, view it on GitHub
https://github.com/infoslack/awesome-web-hacking/pull/73#event-11752794027,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ANDIEMO6SJT7ECLWVUMPTALYSVYRPAVCNFSM6AAAAABC74MSCKVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJRG42TENZZGQYDENY
.
You are receiving this because you authored the thread.Message ID:
@.***>

--
Stjepan Jureković
Manning Publications - Marketing
@.***


#1709 Generate C# TimeSpan as a DayJs duration

Attempt to generate a Dayjs duration for a C# TimeSpan.
See #1709
Im not familiar with the code however I wanted to attempt to help out.
DayJS duration takes a ISO 8601 string or an object as a constructor.
I chose to use a function to convert the 'D.HH:mm:ss.SSS' string to the object.
Im not sure if that is a good idea or if converting it to a ISO 8601 and passing that to the constructor would be more reusable.
Copilot suggests the following conversion function:
function convertToISO8601(input: string): string {
  const [days, rest] = input.split('.');
  const [hours, minutes, seconds, milliseconds] = rest.split(/[:.]/);

  // Constructing the ISO 8601 duration string
  let isoDuration = 'P';
  if (parseInt(days) > 0) isoDuration += `${parseInt(days)}D`;
  if (hours !== '00' || minutes !== '00' || seconds !== '00' || milliseconds !== '000') {
    isoDuration += 'T';
    if (parseInt(hours) > 0) isoDuration += `${parseInt(hours)}H`;
    if (parseInt(minutes) > 0) isoDuration += `${parseInt(minutes)}M`;
    if (parseInt(seconds) > 0 || parseInt(milliseconds) > 0) {
      let totalSeconds = parseInt(seconds);
      if (parseInt(milliseconds) > 0) {
        totalSeconds += parseInt(milliseconds) / 1000;
      }
      isoDuration += `${totalSeconds}S`;
    }
  }

  return isoDuration;
}

It would be best if the conversion function would be added as a function to the file as a utility which can then be called in the generated classes instead of inlined like i did here.
But im not sure how to do this and i hope someone can pick this up and polish it up.

Please dont merge this. I will make a new pull request.
I realized that the dayjs duration can also be constructed with a  ISO 8601.


foundry/eslint override for max-len not working

eslint max-len rule not merged into config
When using @sumup/foundry/eslint and providing overrides to the max-len rule, no change in behavior is observed.
Steps to reproduce
Steps to reproduce the behavior:

create a new nextjs circuit-ui project: npx create-next-app --example "https://github.com/sumup-oss/circuit-ui/tree/main/packages/cna-template/template" my-project
Add a .eslintrc.js with max-len override

module.exports = require('@sumup/foundry/eslint')({
  rules: {
    'max-len': ['error', {code: 100}],
  },
});

create a .tsx file with a line longer than 85 chars but less than 100 chars
run eslint npx foundry run eslint --ext .tsx .
Observe the error: This line has a length of 97. Maximum allowed is 80    max-len

Expected behavior

The override value should be merged in with foundry's baseline sharedRules config, and eslint should permit line lengths of up to 100 characters
foundry's override-merging logic should detect the eslint rule object, then merge the keys/values into its baseline config

Specifications

Package version: 6.1.0
Node version: 18.18.0
OS name & version: MacOS Monterey 12.6.9

Additional context
None

Seems like the issue is that the foundry ruleset defines an eslint overrides section for files: ['**/*.{ts,tsx}'] that has its own max-len declaration.
I see in the eslint documentation for "how do overrides work":

Glob pattern overrides have higher precedence than the regular configuration in the same config file. Multiple overrides within the same config are applied in order. That is, the last override block in a config file always has the highest precedence.

So my workaround is to define my own overrides section like so:
module.exports = require('@sumup/foundry/eslint')({
  rules: {
    'max-len': ['error', { code: 100 }],
  },
  overrides: [
    {
      files: ['**/*.{ts,tsx}'],
      rules: {
        'max-len': ['error', { code: 100 }],
      },
    },
  ],
});

Closing out this issue, as there's no bug present.


making 'sim_enabled' a launch argument

@ipa-bnm
allows to start simulation without Warning

Serial connection on %s failed

fixes https://github.com/ipa320/cob_driver/issues/236
related to https://github.com/ipa320/cob_simulation/issues/87
@xwhm

yes, this makes sense. lgtm

LGTM


CVE-2024-21626

Version: Helm v3.16.3
AWS Security Inspector return vulnerability
Affected packages
CVE-2024-21626 - github.com/containerd/containerd
runc is a CLI tool for spawning and running containers on Linux according to the OCI specification. In runc 1.1.11 and earlier, due to an internal file descriptor leak, an attacker could cause a newly-spawned container process (from runc exec) to have a working directory in the host filesystem namespace, allowing for a container escape by giving access to the host filesystem ("attack 2"). The same attack could be used by a malicious image to allow a container process to gain access to the host filesystem through runc run ("attack 1"). Variants of attacks 1 and 2 could be also be used to overwrite semi-arbitrary host binaries, allowing for complete container escapes ("attack 3a" and "attack 3b"). runc 1.1.12 includes patches for this issue.
Installed version
1.7.12
Fixed version:
1.7.13

CVE-2024-21626 is for runc, and doesn't affect helm


Update langgraph_self_rag_local.ipynb

replaced 'pprint' with 'print' as it caused an error during the tutorial, and pprint had not been imported.

@vbarda
please check and merge.

@unworld11 thanks for fixing!

would love to contribute more often,
love the work you guys are doing
thanks for langchain :)


FMUv2, FMUv4 is out of RAM when SYS_AUTOSTART == 0

Describe the bug
when SYS_AUTOSTART == 0 FMUv4 is out of ram in startup
[[boot] Fault Log info File No 4 Length 3177 flags:0x01 state:1
[boot] Fault Log is Armed
sercon: Registering CDC/ACM serial driver
sercon: Successfully registered the CDC/ACM serial driver
HW arch: PX4_FMU_V4
FW git-hash: cfad556a1609048029e66f540c7e87e87f2d8905
FW version: 1.9.0 0 (17367040)
FW git-branch: master
OS: NuttX
OS version: Release 7.22.0 (118882559)
OS git-hash: d07a1d459ee9b5b56dbf06a160e8faa4c49fcd10
Build datetime: Dec 13 2018 09:05:30
Build uri: localhost
Toolchain: GNU GCC, 7.2.1 20170904 (release) [ARM/embedded-7-branch revision 255204]
PX4GUID:<PHONE_NUMBER>003535323035375110003e0039
MCU: STM32F42x, rev. 3
[hardfault_log] Fault Log is Armed
INFO  [tune_control] Publishing standard tune 1
INFO  [dataman] Unknown restart, data manager file '/fs/microsd/dataman' size is 350216 bytes
INFO  [load_mon] stack check enabled
INFO  [param] selected parameter default file /fs/mtd_params
WARN  [rgbled] no RGB led on bus #1
WARN  [blinkm] I2C init failed
WARN  [blinkm] init failed
WARN  [rgbled_ncp5623c] no RGB led on bus #1
nsh: rgbled_pwm: command not found
Board init: /etc/init.d/rc.board
WARN  [drivers_board] reset done, 50 ms
WARN  [hmc5883] no device on bus 1 (type: 2)
WARN  [lis3mdl] no device on bus 2
INFO  [ist8310] no device on bus 1
BMP280_I2C on I2C bus 1 at 0x76 (bus: 100 KHz, max: 100 KHz)
WARN  [bmp280] id of your baro is not: 0x58
WARN  [bmp280] bus option number is 2
ERROR [bmp280] driver start failed
ERROR [bmm150] Internal I2C not available
HMC5883_SPI on SPI bus 1 at 5 (11000 KHz)
WARN  [hmc5883] no device on bus 1 (type: 3)
WARN  [lis3mdl] no device on bus 2
LIS3MDL_SPI on SPI bus 1 at 7 (11000 KHz)
INFO  [lis3mdl] Poll rate set to max (80hz)
WARN  [mpu6000] no device on bus #3 (SPI1)
MPU6000 on SPI bus 1 at 6 (1000 KHz)
INFO  [mpu9250] Bus probed: 2
WARN  [mpu9250] no device on bus 2
INFO  [mpu9250] Bus probed: 3
MPU9250 on SPI bus 1 at 4 (1000 KHz)
INFO  [mpu9250] accel cutoff set to 30.00 Hz
INFO  [mpu9250] gyro cutoff set to 80.00 Hz
INFO  [mavlink] mode: Normal, data rate: 20000 B/s on /dev/ttyS0 @ 921600B
WARN  [mavlink] hardware flow control not supported
px4flow [46:100]
INFO  [px4flow] scanning I2C buses for device..
ERROR [param] Parameter SYS_USE_IO not found
MS5611_SPI on SPI bus 2 at 3 (20000 KHz)
WARN  [bst] no devices found
INFO  [mpu9250] Bus probed: 2
WARN  [mpu9250] no device on bus 2
INFO  [mavlink] mode: Config, data rate: 800000 B/s on /dev/ttyACM0 @ 57600B
Starting Main GPS on /dev/ttyS3
Starting MAVLink on /dev/ttyS1
INFO  [mavlink] mode: Normal, data rate: 1200 B/s on /dev/ttyS1 @ 57600B
No autostart ID found
INFO  [logger] logger started (mode=all)
nsh: unset: unsetenv failed: Out of memory

nsh: unset: unsetenv failed: Out of memory
nsh> param set SYS_AUTOSTART 4001
  SYS_AUTOSTART: curr: 0 -> new: 4001
nsh> reboot
WARN  [platforms__common] Reboot NOW.
[boot] Fault Log info File No 4 Length 3177 flags:0x01 state:1
[boot] Fault Log is Armed
sercon: Registering CDC/ACM serial driver
sercon: Successfully registered the CDC/ACM serial driver
HW arch: PX4_FMU_V4
FW git-hash: cfad556a1609048029e66f540c7e87e87f2d8905
FW version: 1.9.0 0 (17367040)
FW git-branch: master
OS: NuttX
OS version: Release 7.22.0 (118882559)
OS git-hash: d07a1d459ee9b5b56dbf06a160e8faa4c49fcd10
Build datetime: Dec 13 2018 09:05:30
Build uri: localhost
Toolchain: GNU GCC, 7.2.1 20170904 (release) [ARM/embedded-7-branch revision 255204]
PX4GUID:<PHONE_NUMBER>003535323035375110003e0039
MCU: STM32F42x, rev. 3
[hardfault_log] Fault Log is Armed
INFO  [tune_control] Publishing standard tune 1
INFO  [dataman] Unknown restart, data manager file '/fs/microsd/dataman' size is 350216 bytes
INFO  [load_mon] stack check enabled
INFO  [param] selected parameter default file /fs/mtd_params
WARN  [rgbled] no RGB led on bus #1
WARN  [blinkm] I2C init failed
WARN  [blinkm] init failed
WARN  [rgbled_ncp5623c] no RGB led on bus #1
nsh: rgbled_pwm: command not found
Board init: /etc/init.d/rc.board
WARN  [drivers_board] reset done, 50 ms
WARN  [hmc5883] no device on bus 1 (type: 2)
WARN  [lis3mdl] no device on bus 2
INFO  [ist8310] no device on bus 1
BMP280_I2C on I2C bus 1 at 0x76 (bus: 100 KHz, max: 100 KHz)
WARN  [bmp280] id of your baro is not: 0x58
WARN  [bmp280] bus option number is 2
ERROR [bmp280] driver start failed
ERROR [bmm150] Internal I2C not available
HMC5883_SPI on SPI bus 1 at 5 (11000 KHz)
WARN  [hmc5883] no device on bus 1 (type: 3)
WARN  [lis3mdl] no device on bus 2
LIS3MDL_SPI on SPI bus 1 at 7 (11000 KHz)
INFO  [lis3mdl] Poll rate set to max (80hz)
WARN  [mpu6000] no device on bus #3 (SPI1)
MPU6000 on SPI bus 1 at 6 (1000 KHz)
INFO  [mpu9250] Bus probed: 2
WARN  [mpu9250] no device on bus 2
INFO  [mpu9250] Bus probed: 3
MPU9250 on SPI bus 1 at 4 (1000 KHz)
INFO  [mpu9250] accel cutoff set to 30.00 Hz
INFO  [mpu9250] gyro cutoff set to 80.00 Hz
INFO  [mavlink] mode: Normal, data rate: 20000 B/s on /dev/ttyS0 @ 921600B
WARN  [mavlink] hardware flow control not supported
px4flow [46:100]
INFO  [px4flow] scanning I2C buses for device..
ERROR [param] Parameter SYS_USE_IO not found
MS5611_SPI on SPI bus 2 at 3 (20000 KHz)
WARN  [bst] no devices found
INFO  [mpu9250] Bus probed: 2
WARN  [mpu9250] no device on bus 2
INFO  [mavlink] mode: Config, data rate: 800000 B/s on /dev/ttyACM0 @ 57600B
Starting Main GPS on /dev/ttyS3
Starting MAVLink on /dev/ttyS1
INFO  [mavlink] mode: Normal, data rate: 1200 B/s on /dev/ttyS1 @ 57600B
INFO  [init] Mixer: /etc/mixers/quad_x.main.mix on /dev/pwm_output0
INFO  [logger] logger started (mode=all)

NuttShell (NSH)
nsh> INFO  [ecl/EKF] EKF aligned, (pressure height, IMU buf: 22, OBS buf: 14)
INFO  [frsky_telemetry] Scanning timeout: exiting

@davids5 the problem is in NuttX: after the last unset, unsetenv will use newsize==0 in https://github.com/PX4/NuttX/blob/px4_firmware_nuttx-7.22%2B/sched/environ/env_unsetenv.c#L97.
The realloc call then returns NULL in https://github.com/PX4/NuttX/blob/px4_firmware_nuttx-7.22%2B/mm/mm_heap/mm_realloc.c#L100, which then leads to a ENOMEM error.

@davids5 can you look into this please?

@bkueng Sorry I should have acknowledged your findings - Thank you for debugging it.
It is on my list for today! I had an massive fragmentation issue with the upgrade to 7.72 that I am working on. I originally thought this was the genesis until I found the cause that I am working with Greg on resolving.


bevy_render2: Add support for #else for shader defs

Objective

Add support for #else for shader defs

Solution

When entering a scope with #ifdef or #ifndef, if the parent scope is truthy, and the shader definition is also truthy, then the a new scope is pushed onto the scope stack that is also truthy, else falsy. When encountering a subsequent else clause within a scope, if the parent is truthy and the current scope is truthy, then it should become falsy. If the parent scope is truthy and the current scope is falsy then it should become truthy. If the parent scope is falsy, then the current scope should remain falsy as the parent scope takes precedent.
I added a simple test for an else case.

looks good to me!

bors r+


Add support for Winsen ZH03B

Added support for a Winsen ZH03B air quality sensor, updated the version number and fixed a bug in the demo code.

@usul27 Can you take look when you get a chance?

@usul27 you are listed as the PyPi owner for this, are you interested in maintaining pmsensor still?  I would be happy to take it if you cannot support it anymore.


Support processing of FHIR JSON with unexpected fields

Is your feature request related to a problem? Please describe.
For some use cases, it would be nice to be able to parse a JSON representation of a resource that has some unexpected fields. This is somewhat normal for JSON APIs and allows for greater forward compatibility (i.e. adding fields can be backwards compatible with older versions of the API).
From https://www.hl7.org/fhir/R4/json.html#2.6.2

Given the way extensions work, applications reading JSON resources will never encounter unknown properties. However, once an application starts trading with other applications that conform to later versions of this specification, unknown properties may be encountered. Applications MAY choose to ignore unknown properties in order to foster forwards compatibility in this regard, but may also choose not to.

Describe the solution you'd like
Firstly, we'd need to add support for "lenient mode" in the parser so that these resources could be parsed. We used to have something like this in the DSTU2 implementation (fhirServer/core/jsonParserLenient).
Next, we'd need some way to configure the server to set the parse mode to either "strict" or "lenient" by default.  If we don't find the config setting, then we should use "strict" mode to match the current behavior.
Finally, I think it would be cool to honor the client's "handling" preference (optionally passed in via the Prefer header) in order for clients to override the default behavior of the server.
Perhaps 4 different settings for the config property:

strict-only ("strict mode" with no option for client to override via the Prefer header)
strict* (default to "strict mode" but honors the client handling preference)
lenient (default to "lenient mode" but honors the client handling preference)
lenient-only ("lenient mode"  with no option for client to override via the Prefer header)

Where strict would be the default for the propery.
Additional context
Currently, parsing a resource with invalid fields will result in a parse error and I think that is a good default.

Would require changes to:

Parser interface to support the config param
CodeGenerator that generates JSON and XML parsers


QUnit tests that are not in a module are ignored by the service

The generateTestCases function only looks at tests in childSuites:
https://github.com/mauriciolauffer/wdio-qunit-service/blob/48c10ba8398fc9406032f5fea24e71e4b985f7c9/src/index.ts#L43C10-L43C27
I have a test file where only one of the four tests is defined under a module. The resulting qunitResults data structure looks like this:
[0-0] 2024-05-29T21:11:54.390Z INFO webdriver: RESULT {
[0-0]   childSuites: [
[0-0]     {
[0-0]       childSuites: [],
[0-0]       fullName: [Array],
[0-0]       name: 'violations-test-module',
[0-0]       runtime: 4031,
[0-0]       status: 'passed',
[0-0]       testCounts: [Object],
[0-0]       tests: [Array]
[0-0]     }
[0-0]   ],
[0-0]   fullName: [],
[0-0]   name: null,
[0-0]   runtime: 11128,
[0-0]   status: 'failed',
[0-0]   testCounts: { failed: 2, passed: 4, skipped: 0, todo: 0, total: 6 },
[0-0]   tests: [
[0-0]     {
[0-0]       assertions: [Array],
[0-0]       errors: [],
[0-0]       fullName: [Array],
[0-0]       name: 'qoe-stuff',
[0-0]       runtime: 1003,
[0-0]       status: 'passed',
[0-0]       suiteName: null
[0-0]     },
[0-0]     {
[0-0]       assertions: [Array],
[0-0]       errors: [],
[0-0]       fullName: [Array],
[0-0]       name: 'js-error-stuff',
[0-0]       runtime: 3125,
[0-0]       status: 'passed',
[0-0]       suiteName: null
[0-0]     },
[0-0]     {
[0-0]       assertions: [Array],
[0-0]       errors: [],
[0-0]       fullName: [Array],
[0-0]       name: 'qoe-loaded',
[0-0]       runtime: 2968,
[0-0]       status: 'passed',
[0-0]       suiteName: null
[0-0]     }
[0-0]   ]
[0-0] }

Only the test in childSuites was processed.

@bburke4 thanks for sharing it. I've run a couple tests with no qunit modules and couldn't replicate your issue. Would be possible to share a gist/repo with your scenario, pls?

I could replicate this function at a higher level in the code, which probably would fix your issue. I just want to replicate it before making any changes.
I'll also include the scenario to the examples/tests

Thanks for the quick reply. I will work on modifying my tests in such a way that I can share them publicly. But FWIW, the tests were written out in a script tag in an html file and I was loading QUnit via http://code.jquery.com/qunit/qunit-2.20.1.js

It seems to be related to async tests. Here is the test file that should reproduce it:

        <title> Tests</title>
        <link rel="stylesheet" href="//code.jquery.com/qunit/qunit-2.20.1.css">

Test file was simple:
    describe('Unit Tests', () => {
        it('should pass', async () => {
            await browser.url('http://localhost/sample-test.html');
            const qunitResults = await browser.getQUnitResults();
            expect(qunitResults).toBeTruthy();
        });
    });

Thanks!

On a potentially unrelated note, I also ran into an issue of tests being ignored when i was loading some asynchronously. I noticed that the service was detecting that the runEnd must have already been emitted so it collected the results before all the tests had finished running.
Looking into the code a bit, I was wondering if it would be better if the service checked QUnit.config.pq.finished instead of QUnit.config.queue.length here, but I haven't tested this out yet.

@bburke4 thanks for sharing all these details, I appreciate it. Both issues should be fixed now (https://github.com/mauriciolauffer/wdio-qunit-service/pull/13). I've also included more tests that should cover these scenarios.

@bburke4  thanks for sharing all these details, I appreciate it. Both issues should be fixed now (https://github.com/mauriciolauffer/wdio-qunit-service/pull/13). I've also included more tests that should cover these scenarios.

Thanks so much @mauriciolauffer! I see all of my tests included in the results now and it even seems to run a little quicker. Thank you!


Compare by date when version compare is not possible in latest.py

Updates were ignored when versions could not be compared (e.g. if at least one of the version did not conform to PEP 440). This improves the process by comparing by version dates in that case.

Without this changes updates for products such as unity don't work at all.


Improve Dockerfile.revad-eos

Use EOS 4.8.91
Bring back multi-stage build
Copy the resulting revad from the builder container

Resulting image size (unpacked on disk) is 2.59GB

eos-all:4.8.91 is 2.47GB
existing revad:latest-eos is 6.18GB

/cc @SamuAlfageme can we get this merged, so that it can be used for Sciencebox?

@ebocchi I'm sorry to have screwed part of this change :-P can you please rebase it on top of master ?

@glpatcern
Rebased against master. Should be good to go now ;)

@ebocchi can you fix the linter (Codacy Static Code Analysis)?

As per @glpatcern, going back to eos-client-4.8.91-1.el7.cern.x86_64 as the only package needed is eos-client.
eos-client-4.8.91-1.el7.cern.x86_64 include:
# rpm -qa | grep ^eos | sort
eos-client-4.8.91-1.el7.cern.x86_64
eos-folly-2<IP_ADDRESS>-1.el7.cern.x86_64
eos-folly-deps-2<IP_ADDRESS>-1.el7.cern.x86_64
eos-fusex-4.8.91-1.el7.cern.x86_64
eos-fusex-core-4.8.91-1.el7.cern.x86_64
eos-fusex-selinux-4.8.91-1.el7.cern.x86_64
eos-protobuf3-3.5.1-5.el7.cern.eos.x86_64
eos-xrootd-4.12.8-1.el7.cern.x86_64


fix read visium with space ranger 2.0

IMPORTANT: Please search among the Pull requests before creating one.
Description
Fix :func:squidpy.read.visium with spacerangerV2.0 update.
How has this been tested?

Closes

close #599

Codecov Report

Merging #600 (ec78faf) into main (d24dd13) will decrease coverage by 0.05%.
The diff coverage is 40.00%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #600      +/-   ##
==========================================
- Coverage   72.31%   72.26%   -0.06%
==========================================
  Files          37       37
  Lines        5177     5185       +8
  Branches      976      984       +8
==========================================
+ Hits         3744     3747       +3
- Misses       1181     1185       +4
- Partials      252      253       +1

Impacted Files
Coverage Δ

squidpy/read/_read.py
37.23% <40.00%> (-1.14%)
:arrow_down:

squidpy/im/_io.py
67.14% <0.00%> (+0.71%)
:arrow_up:

will close this following this conversation https://github.com/scverse/scanpy/pull/2296#issuecomment-1273309536


Retry metadata deployments after failure due to temporary org locking

There can be an error returned from metadata deployments that really doesn't indicate an issue with the code.  It is intermittent and is usually due to some background task not yet completing from the previous build.  The SOAP response message contains the following XML that could be parsed for:
<errorMessage>null: The changes you requested require salesforce.com to temporarily lock your organization's administration setup. However, the administration setup has already been locked by another change. Please wait for the previous action to finish, then try again later.</errorMessage><errorStatusCode>ALREADY_IN_PROCESS</errorStatusCode>

This should probably behave like the retries option on the cumulusci.tasks.salesforce.InstallPackageVersion task does if a beta version is not yet available.

This may still happen, but hasn't been a practical issue since we're mostly using scratch orgs.


[Map] DM-Dangerous

Hash: d88bdaa90f0843451768e31865452b5d0d9d0e02
Current name: DM-Dangerous
The author's name is actually Dave "Dangerus" O.
From the readme it looks like The First means it is his first map.
Real name : Dave Otten

Updated


Adds support for falling back to the partner profile image on a stubbed show

Before:

After:

Fixes LD-347

Fails

:no_entry_sign:

No CHANGELOG added.

  Generated by :no_entry_sign: dangerJS


feat(SPV-1249) paymail list & assert with JSON template

Pull Request Checklist

[ ] 📖 I created my PR using provided  : CODE_STANDARDS
[ ] 📖 I have read the short Code of Conduct: CODE_OF_CONDUCT
[ ] 🏠 I tested my changes locally.
[ ] ✅ I have provided tests for my changes.
[ ] 📝 I have used conventional commits.
[ ] 📗 I have updated any related documentation.
[ ] 💾 PR was issued based on the Github or Jira issue.

:warning: Please install the  to ensure uploads and comments are reliably processed by Codecov.
Codecov Report
Attention: Patch coverage is 61.83575% with 79 lines in your changes missing coverage. Please review.

Project coverage is 45.35%. Comparing base (a4547fb) to head (f457c2b).

Files with missing lines
Patch %
Lines

actions/testabilities/json_placeholders.go
64.00%
41 Missing and 13 partials :warning:

actions/users/paymails.go
65.71%
9 Missing and 3 partials :warning:

...ions/testabilities/assert_spvwallet_application.go
0.00%
10 Missing :warning:

models/filter/paymail_filter.go
72.72%
2 Missing and 1 partial :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #793      +/-   ##
==========================================
+ Coverage   45.16%   45.35%   +0.18%
==========================================
  Files         358      360       +2
  Lines       17360    17565     +205
==========================================
+ Hits         7841     7966     +125
- Misses       8949     9008      +59
- Partials      570      591      +21

Flag
Coverage Δ

unittests
45.35% <61.83%> (+0.18%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines
Coverage Δ

actions/users/routes.go
100.00% <100.00%> (ø)

models/filter/paymail_filter.go
73.91% <72.72%> (-4.66%)
:arrow_down:

...ions/testabilities/assert_spvwallet_application.go
0.00% <0.00%> (-68.34%)
:arrow_down:

actions/users/paymails.go
65.71% <65.71%> (ø)

actions/testabilities/json_placeholders.go
64.00% <64.00%> (ø)

... and 10 files with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a4547fb...f457c2b. Read the comment docs.


(MAINT) Handle username of PE masters > 3.7.0

Thanks. I'd seen the email be truncated, so the README was correct.


Add support for Selinux mount context labels

Signed-off-by: Mrunal Patel<EMAIL_ADDRESS>

@crosbymichael @LK4D4 ping
cc: @rhatdan

Why does this not get placed in the Mount.Options array?

@crosbymichael This is for convenience. We could pass it to individual mounts as well. (We had a MountLabel in libcontainer configuration).

An argument in favor is that the value of the label is the same for all the mount points (that we need to label).

Could it every need to be different or will it always be the same?

AFAIK it wouldn't be but would defer to @rhatdan on that.

ok, looking at the existing code this looks good.  Only thing is that your json example has this as an array and not just a string

@crosbymichael Fixed.

LGTM

Right now we are still always planning on labeling content inside of the container with the same label.
LGTM

@mrunalp would you mind adding it to the JSON schema too? otherwise I can do a follow-up PR.

LGTM based on your response ^^

I'll add it.
Sent from my iPhone

On Apr 22, 2016, at 10:56 AM, Vincent Batts<EMAIL_ADDRESS>wrote:
@mrunalp would you mind adding it to the JSON schema too? otherwise I can do a follow-up PR.
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

@vbatts Added to schema.

LGTM


cleaning up a unit test InvalidPipelineRuns to use YAML parser

Changes
Replacing task, pipeline, pipelineRun, objects initialization with the YAMLParser for readability in TestReconcile_InvalidPipelineRuns.
Signed-off-by: pritidesai<EMAIL_ADDRESS>/kind cleanup
part of issue #4610
Submitter Checklist
As the author of this PR, please check off the items in this checklist:

[ ] Docs included if any changes are user facing
[x] Tests included if any functionality added or changed
[x] Follows the commit message standard
[x] Meets the Tekton contributor standards (including
functionality, content, code)
[x] Release notes block below has been filled in or deleted (only if no user facing changes)

Release Notes
NONE

/lgtm


Add Re-infer metadata link to alerts at coverage pages

Sometimes IPT displays an alert that the metadata could not be inferred. Often it is enough to simply do re-inferring.

related https://github.com/gbif/ipt/issues/2368


Card doesn't work with new GitHub integration

As the title says: due to an updated GitHub integration, the github-flexi-card does not work anymore. See also https://www.home-assistant.io/blog/2022/02/02/release-20222/#breaking-changes
HA version: 2022.2

The new integration is completely different and it doesn't expose all the stats (e.g. clones, views). It sucks that it lists only the starred repos and it is not possible to add a custom one (which you haven't starred already - e.g. your own repos)
I have created a topic with feature request so please vote if you're interested
https://community.home-assistant.io/t/github-integration-improvements-ha-2020-2/388098
FYI I'm working on the change to add the support for attribs/repo-properties in the separate entities

Just voted, thank you for all the effort you are putting in this!


[fuzz result] blank list item causes spurrious split

-

-

-

-

Pulldown-cmark thinks there's three lists here, but commonmark.js and commonmark-hs both think there's only one.
spec.commonmark.org dingus
pandoc.org try

Out of curiosity, how did this not appear during fuzzing? It seems exactly the kind of thing fuzzing would catch.


python313Packages.standard-mailcap: init at 3.13.0

This is a replacement for the mailcap module from the python standard lib which was removed for python 3.13.  This package will be needed for alot to work under python 3.13.
Repo: https://github.com/youknowone/python-deadlib
PyPI: https://pypi.org/project/standard-mailcap/
(There is no proper homepage)
This fixes #362878.
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[x] sandbox = true

[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

@dotlambda I tried to address your comments. Do you mind having another look?


CodeException:500

Issue when running the ptc command.
Traceback (most recent call last):
File "C:\Python27\Scripts\ptc-script.py", line 9, in
load_entry_point('PTCAccount==1.2.1', 'console_scripts', 'ptc')()
File "c:\python27\lib\site-packages\ptcaccount\console.py", line 45, in entry
args.username, args.password, args.email, args.email_tag
File "c:\python27\lib\site-packages\ptcaccount\accountcreator.py", line 379, in random_account
try_username, password, try_email
File "c:\python27\lib\site-packages\ptcaccount\accountcreator.py", line 226, in create_account
resp_code=200
File "c:\python27\lib\site-packages\ptcaccount\accountcreator.py", line 88, in request
raise PTCInvalidStatusCodeException(str(resp.status_code))
ptcaccount.ptcexceptions.PTCInvalidStatusCodeException: 500

Python 2.7

I had an extra space or something.


Data Protection migrations for PostgreSql

I've implemented the migrations of the data protection tables for PostgresSql

@skoruba I've ran the unit tests locally and they work. I have no idea why it's failing here.
Can you rerun it?

Thanks for this PR, but I already did it - for PostgreSQL and Mysql as well. :-)

It's not on the dev branch.

I merge it now.


data missing on finance_budgets table

For both Cornell (on LDP 1.8.2) and Chicago, the finance_budgets table is not showing the data consistently for these columns.
allocated
cash_balance
total_funding
available
unavailable
over_expended
over_encumbrance
This is also true for the budget_t table in Chicago's implementation of metadb.

If you would, please provide a specific example.

When I run a query like this one, I do not see a problem:
SELECT id,
       cash_balance,
       json_extract_path_text(data, 'cashBalance') AS cash_balance_json
    FROM finance_budgets
    WHERE cash_balance IS NOT NULL OR
        json_extract_path_text(data, 'cashBalance') IS NOT NULL;

Just to confirm, are you using ldp1_folio_snapshot to test this, and https://folio-snapshot.dev.folio.org/ the front end for this LDP1 instance?

I was not able to find a problem in any LDP1 instance.  Could you give a specific example of missing data?

Assuming ldp1_folio_snapshot is using https://folio-snapshot.dev.folio.org/ as its source, you will see this issue for the AFRICAHIST fund in FY23.

If you run this quick script in ldp1_folio_snapshot, you will see that the data from the fields mentioned above is NULL and it should be populated with the values you see in FOLIO.
SELECT *
FROM finance_budgets fb
WHERE name LIKE 'AFRICAHIST-FY2023'
;

Nancy Bolduc found this issue, so feel free to talk to her in more depth about this problem.

Here are the data as they are stored in FOLIO in the budget table, which is the source table for finance_budgets:
 {
     "id": "d71635df-c08c-49cc-9082-197a30fd0392",
     "name": "AFRICAHIST-FY2023",
     "fundId": "7fbd5d84-62d1-44c6-9c45-6cb173998bbd",
     "_version": 17,
     "metadata": {
         "createdDate": "2023-01-12T01:54:36.514Z",
         "updatedDate": "2023-01-12T18:38:26.774Z",
         "updatedByUserId": "3cdbc9a2-ea43-5054-9953-69840cccc27d"
     },
     "acqUnitIds": [
     ],
     "encumbered": 28.0,
     "allocationTo": 0.0,
     "budgetStatus": "Active",
     "expenditures": 14.0,
     "fiscalYearId": "0bb73cfd-9251-44ce-a7de-3ef16875ba6a",
     "netTransfers": -2500.0,
     "allocationFrom": 0.0,
     "awaitingPayment": 0.0,
     "initialAllocation": 21000.0,
     "allowableEncumbrance": 110.0,
     "allowableExpenditure": 100.0
 }

As you can see, FOLIO is not storing those columns in this table.

Perhaps calculations to another set of stored columns elsewhere?

Possibly.  I suggest talking with the PO.

Good morning all!  I find it misleading to have columns in the LDP where the data is not populated.  Why do we have these columns then?

For columns transformed from JSON, a column exists if at least one of the records has data.

Okay, that makes sense. So I believe our next step is to work with Folio. Thanks Nassib!
From: Nassib Nassar @.>
Sent: Friday, January 13, 2023 9:15 AM
To: library-data-platform/ldp @.>
Cc: Nancy Bolduc @.>; Comment @.>
Subject: Re: [library-data-platform/ldp] data missing on finance_budgets table (Issue #226)
For columns transformed from JSON, a column exists if at least one of the records has data.
—
Reply to this email directly, view it on GitHubhttps://github.com/library-data-platform/ldp/issues/226#issuecomment-1381919392, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ANYDJNJU652CHS4F4I3IC2DWSFPQDANCNFSM6AAAAAATZOIAXU.
You are receiving this because you commented.Message ID<EMAIL_ADDRESS>


Allow an easy copy of adapter/aggregator between chains

@KelvinThai I am gonna take over this one, do small change and merge it!


Add nginx config for RTL

Related to #255

Is/was there a reason that LNbits and RTL are/were not started after activation?

Is/was there a reason that LNbits and RTL are/were not started after activation?

RTL was causing errors during recovery when it was activated in the install script and started before the LND wallet finished the setup.


Argentinian ID not working

Hi! I'm developing an interpreter for the PDF417 code from argentinian IDs. I've try this library, but none seems to work. Could it be something about colour? Size? The pictures are FullHD, scanned... Is there a way to force the "TRY HARDER" mode when invoking the decoder? Thnaks!

The decoder is very sensitive to image quality. Do you have a very high resolution sample to start from?

Also check out this thread: https://github.com/PeculiarVentures/js-zxing-pdf417/issues/13

Thank you for your answer! Yes, I've changed the file pdf417/decoder/DecodedBitStreamParser.js but nothing changes.

Im not sure what to tell you.

I was also trying an Argentinan ID and run into this problem, it might be related https://github.com/PeculiarVentures/js-zxing-pdf417/pull/16

Merged.
I'm curious since your both trying for the same ID, what independently motivated you both down this path?

Well, the Argentinian ID has a PDF417 and it's not so uncommon to require one when registering for some services.
I'm trying a client-side instead of server-side to validate the image quality earlier in the process.

You may find this interesting: https://demo.identityfront.com
We have not integrated the PDF417 library yet but MRZ is in the works currently.

I guess that @seppo0010 wants something to extract the data more than something to validate identity, just as we want.

Yes, we also want extraction, we just decided to do MRZ first as we have higher confidence that we will be able to do it robustly.
PDF417 extraction is on the roadmap.

Also, and here is my personal toughts about the issue, here in Argentina we're used to try solutions that give us enough freedom to adapt, improve and also that won't affect out budget. That's another motivation for using XZing :)

@RodolfoEBDR of course; that is universal and why we OSS so much of what we do :)

@RodolfoEBDR @seppo0010 Were you able to make it work?

It worked for my with the PR I sent that is now merged.

One thing I noticed using server-side zxing is that often resizing the original image (e.g.: 2x) helped a lot. I'm not sure if that's possible in the browser, I believe it is. However we did not go with this version for the project I was working on due to unrelated reasons.


feat: +git tools

Features

Add git tool lib

Codecov Report
Attention: Patch coverage is 84.09091% with 7 lines in your changes are missing coverage. Please review.

Project coverage is 81.62%. Comparing base (fdf53ac) to head (e8cf329).

Files
Patch %
Lines

metagpt/utils/git_repository.py
80.64%
6 Missing :warning:

metagpt/tools/libs/git.py
92.30%
1 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1068   +/-   ##
=======================================
  Coverage   81.62%   81.62%
=======================================
  Files         248      249    +1
  Lines       13885    13929   +44
=======================================
+ Hits        11333    11370   +37
- Misses       2552     2559    +7

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Addition of a required package for azure fencing configuration on SLES

[Enter feedback here]
Based on SLES documentation: https://www.suse.com/support/kb/doc/?id=000020377,
We need to deploy python-azure-identity(SLES 12. xx) or python3-azure-identity ( SLES 15. xx ) in order for the Azure fencing agent to work.
Without installing this package, the setup for azure fencing on SLES will result in a "Stopped" status without any useful information in logs or traces.  Deeper troubleshooting would indicate that the fencing agent is unable to connect to Azure AD for authentication.
This package( python-azure-identity or python3-azure-identity) is currently not deployed as a dependency when installing python3-azure-mgmt-compute or python-azure-mgmt-compute, and hence it should be included in the documentation.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: a20453e6-5273-2119-69f7-85b5108a8f7d
Version Independent ID: 4ac93e53-6969-4c8d-5b5c-0db47a8976ce
Content: Set up Pacemaker on SUSE Linux Enterprise Server (SLES) in Azure - Azure Virtual Machines
Content Source: articles/virtual-machines/workloads/sap/high-availability-guide-suse-pacemaker.md
Service: virtual-machines-sap
GitHub Login: @rdeltcheva
Microsoft Alias: radeltch

@shekharsorot
Thanks for the feedback! I have assigned the issue to content author to check and update the document as appropriate.

@rdeltcheva
Can you please check and add your comments on this doc update request as applicable.

We Already link to the SUSE KBA from the documentation : "If your python-azure-mgmt-compute or python3-azure-mgmt-compute package version is 17.0.0-6.7.1, follow the instructions in SUSE KBA to update the fence-agents version and install the Azure Identity client library for Python module if it is missing.".
Since the need to install the Azure Identity client library is dependent on the version of the python management compute package we chose to link to the SUSE KB.

Just want to point out, currently, the dependency chain for "python3-azure-mgmt-compute" does not includes "python3-azure-identity", not it's installed by default with any other azure  packages as well unless explicitly specified. This seems to be an issue from SUSE side.
It might make sense to add this package to the line  "sudo zypper install python3-azure-mgmt-compute"  since end users / admins who are configuring pacemaker would prefer to use a single documentation link as reference for step by step instructions. In fact, this is an actual feedback from an end cutomer.
Either way, much appreciate your quick response on the same.

Hello, I have installed fence-agents-4.7 and was running in described issue with  python3-azure-mgmt-compute package version is 17.0.0-6.7.1. I have updated azure-mgmt-compute to version 23.0.0 .
After this, fecing agent - still on 4.7 is running fine.
Is this an valid alternative solution and if so, will it be documented somewhere?

Thanks for your dedication to our documentation. Unfortunately, at this time we have been unable to review your issue in a timely manner and we sincerely apologize for the delayed response. Since these packages are managed by SUSE, the preference is to point to SUSE KB that explains and documents the dependencies (for the different SLES versions). We are closing this issue for now, but feel free to comment here as necessary.
#please-close


add a requirement not to register the secret password with null character (fixes #3504)

Bug（バグ） #3504
新規登録時に「秘密の質問への答え」が空欄のままでも登録できてしまう
https://redmine.openpne.jp/issues/3504
Closes #228

Since the target version was changed, the ticket termination


Transpile JS for use in Safari < 13

PhotoSwipe uses optional chaining, so it does not work in Safari < 13.
(Maybe this is too hard or too inefficient)

Won’t fix, won’t support obsolete browsers.


[python] Attempt MacOS M1 wheel build

Issue and/or context:
#794
Changes:
Notes for Reviewer:
This is currently just an experiment.

Codecov Report
Base: 79.31% // Head: 66.10% // Decreases project coverage by -13.21% :warning:

Coverage data is based on head (6064656) compared to base (21f92c2).
 Consider uploading reports for the commit 82d5334 to get more accurate results

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
Additional details and impacted files
@@             Coverage Diff             @@
##             main     #810       +/-   ##
===========================================
- Coverage   79.31%   66.10%   -13.21%
===========================================
  Files          70       40       -30
  Lines        3490     1599     -1891
===========================================
- Hits         2768     1057     -1711
+ Misses        722      542      -180

Flag
Coverage Δ

python
?

r
66.10% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

apis/python/src/tiledbsoma/metadata_mapping.py

apis/python/src/tiledbsoma/logging.py

apis/python/src/tiledbsoma/factory.py

apis/python/src/tiledbsoma/general_utilities.py

apis/python/src/tiledbsoma/util_ann.py

apis/python/src/tiledbsoma/exception.py

apis/python/src/tiledbsoma/measurement.py

...n/src/tiledbsoma/options/tiledb_platform_config.py

apis/python/src/tiledbsoma/dense_nd_array.py

apis/python/src/tiledbsoma/dataframe.py

... and 20 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

reattempting

#794 is the real thing


Issue #53: Updated docusaurus.config.js

Added Configurations:
  showLastUpdateTime: true,
  showLastUpdateAuthor: true,

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

@andresgalante I have added the configuration options showLastUpdateTime: true and showLastUpdateAuthor: true, but the changes are not being reflected on the page.

@ShivamPandey00 : I was able to show the lastUpdateTime and Author showing up on the docs standalone pages but not able to pull from the remote files. With the current priorities, I don't have the bandwidth to research and add this feature for the remote files. But, will consider it in the future. I will close this issue and continue to track it under PR: https://github.com/hirosystems/docs/pull/372.
Thanks for your contribution!


ERROR: failed to solve: process "/bin/sh -c pip3 install --upgrade awscli" did not complete successfully: exit code: 1

ERROR: failed to solve: process "/bin/sh -c pip3 install --upgrade awscli" did not complete successfully: exit code: 1

New version 1.1 removes the need for AWSCLI altogether. Please try V1.1 and let me know if any issues remain.


Remove bad assert in destroyJoint

pls is possible make new version to pub.dev where is this fix. I am stuck with this error.

@rmasarovic thanks for the reminder, just published it for forge2d will soon publish flame_forge2d

Now flame_forge2d is updated to 0.6.3-rc3 and forge2d to 0.6.3, enjoy!