Unused vmap GPU memory allocation causes RESOURCE_EXHAUSTED for versions >0.4.14

Description
Overview
The script below works when using an NVIDIA GPU with Jax version 0.4.14, but after upgrading to 0.4.31 (and trying a few other versions in between) it is triggering the following error:
E0910 20:24:00.097739   38257 pjrt_stream_executor_client.cc:3067] Execution of replica 0 failed: RESOURCE_EXHAUSTED: Out of memory while trying to allocate X bytes
jaxlib.xla_extension.XlaRuntimeError: RESOURCE_EXHAUSTED: Out of memory while trying to allocate X bytes.
where the value of X ranges from ~5GB (e.g.<PHONE_NUMBER>) to 20GB+ depending on the shape of the dls variable (set to 3540 in the script below).
jax<=0.4.14 - no error
jax>0.4.14 - error
Not sure if this is a bug or if there is some code/syntax in the example below that is no longer supported in versions > 0.4.14 that is responsible for this behavior.
Allocation vs. pprof usage
The GPU has 6GB of memory and after some trial and error it appears that setting the dls variable to a shape of 1590 succeeds and uses only ~500kB of memory according to pprof (following https://jax.readthedocs.io/en/latest/device_memory_profiling.html), but a shape of 1600 gives an error trying to allocate ~5GB. If pprof is in fact showing GPU memory usage this could suggest memory is being allocated but not used.
jnp.exp removal
Trial and error also showed that removing the jnp.exp calls inside the function m seem to resolve the issue. For example, the script below with dls shape set to 10000 fails trying to allocate 30GB, but removing the jnp.exp calls succeeds and shows as using only ~2MB by pprof.
Script
import jax
import jax.numpy as jnp
from jax import vmap

def wp(y0, ts, rng, tidx_offset):
    t0, t1 = ts[0], ts[-1]
    y = jnp.ones((11, 3)) * (t0 * t1)
    y = jnp.vstack(
        (
            y,
            jnp.ones((71, 3)) * y0,
        )
    )
    y = jnp.roll(y, tidx_offset, axis=0)
    y = y[:71]
    y = y.at[:, 2].set(jnp.abs(y[:, 2] - 0.03) + 0.03)
    return y

def ps(ys, ts, tidx_offset):
    t = jnp.maximum(ts - ts[0], 0)
    t = jnp.hstack(
        (
            t,
            jnp.zeros(71 - 11),
        )
    )
    t = jnp.roll(t, tidx_offset, axis=0)
    t = t[:71]
    ds = jnp.sqrt(jnp.sum((ys[1:, :] - ys[:-1, :]) ** 2, axis=-1))
    d = jnp.cumsum(jnp.hstack((jnp.array([0.0]), ds)), axis=-1)

    s_xyz = jnp.array([0.123, 0.345, 0.456])
    s_xyz = jnp.exp(jnp.array(-2.0)) + s_xyz
    scale = t * jnp.exp(jnp.array(-2.2)) + d * jnp.exp(jnp.array(-2.5)) + 1e-6
    s = jnp.einsum("i,x->ix", scale, s_xyz)

    return s

def m(s, d, d_mirror, rate):
    scale = 0.5 * (15.0 / 75)
    m = (
        scale
        * rate
        / ((2 * jnp.pi) ** (3 / 2) * jnp.prod(s))
        * (
            # removing these two jnp.exp calls appears to resolve the issue
            (jnp.exp(-0.5 * jnp.sum(d**2 / s**2)))
            + (jnp.exp(-0.5 * jnp.sum(d_mirror**2 / s**2)))
        )
    )
    return m

def func(y0, dl, tss, rng, rate, tidx_offset):
    ys = wp(y0, tss, rng, tidx_offset)
    d = ys - dl
    A = jnp.array([[1, 0, 0], [0, 1, 0], [0, 0, -1]])
    ys_mirror = jnp.matmul(ys, A)
    d_mirror = ys_mirror - dl
    scale = 0.5 * (15.0 / 75)
    rate = rate[tidx_offset]
    s = jnp.ones((71, 3)) * y0 * 2.3

    results = vmap(m, in_axes=(0, 0, 0, None))(s, d, d_mirror, rate)
    return results

@jax.jit
def run():
    y0s = jnp.ones(shape=(1, 3))
    # dls shape of ~1600+ fails on 6GB GPU trying to allocate 5GB+
    # dls shape of <1590 succeeds on 6GB GPU and uses only ~476kB memory according to pprof
    # dls shape of 10000 fails trying to allocate 30GB, but passes and only uses ~2MB when removing jnp.exp calls above
    dls = jnp.ones(shape=(3540, 3))
    rates = jnp.ones(shape=(1, 71))
    rngs = jnp.ones(shape=(71, 75, 2), dtype="uint32")
    tss = jnp.ones(shape=(71, 11, 75))
    tidx_offsets = jnp.arange(len(tss))

    output = vmap(
        vmap(
            vmap(
                vmap(func, in_axes=(None, None, 1, 0, None, None)),
                in_axes=(None, None, 0, 0, None, 0),
            ),
            in_axes=(None, 0, None, None, None, None),
        ),
        in_axes=(0, None, None, None, 0, None),
    )(y0s, dls, tss, rngs, rates, tidx_offsets)
    result = jnp.sum(output, axis=(0, 2, 3))
    return result

result = run()
jax.profiler.save_device_memory_profile("memory.prof")
print(result)
print(result.shape)

System info (python version, jaxlib version, accelerator, etc.)
Pip versions:
# jax
jax==0.4.31
jax-cuda12-pjrt==0.4.31
jax-cuda12-plugin==0.4.31
jaxlib==0.4.31
jaxtyping==0.2.34
# nvidia
nvidia-cublas-cu12==<IP_ADDRESS>
nvidia-cuda-cupti-cu12==12.6.68
nvidia-cuda-nvcc-cu12==12.6.68
nvidia-cuda-runtime-cu12==12.6.68
nvidia-cudnn-cu12==<IP_ADDRESS>
nvidia-cufft-cu12==<IP_ADDRESS>
nvidia-cusolver-cu12==<IP_ADDRESS>
nvidia-cusparse-cu12==<IP_ADDRESS>
nvidia-nccl-cu12==2.22.3
nvidia-nvjitlink-cu12==12.6.68

Output of jax.print_environment_info(), it is running inside a container based on nvidia/cuda:12.3.2-base-ubuntu22.04:
jax:    0.4.31
jaxlib: 0.4.31
numpy:  1.26.4
python: 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0]
jax.devices (1 total, 1 local): [CudaDevice(id=0)]
process_count: 1
platform: uname_result(system='Linux', node='docker-desktop', release='<IP_ADDRESS>-microsoft-standard-WSL2', version='#1 SMP Fri Mar 29 23:14:13 UTC 2024', machine='x86_64')

$ nvidia-smi
Tue Sep 10 20:44:01 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.23.06              Driver Version: 545.92       CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 4050 ...    On  | 00000000:01:00.0 Off |                  N/A |
| N/A   43C    P3              11W /  35W |     78MiB /  6141MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A     46151      C   /python3.10                               N/A      |
+---------------------------------------------------------------------------------------+

Pip versions of latest version that does not show the error (v0.4.14):
# jax
jax==0.4.14
jaxlib==0.4.14+cuda12.cudnn89
jaxtyping==0.2.23
# nvidia
nvidia-cublas-cu12==<IP_ADDRESS>
nvidia-cuda-cupti-cu12==12.1.105
nvidia-cuda-nvcc-cu12==12.1.105
nvidia-cuda-runtime-cu12==12.1.105
nvidia-cudnn-cu12==<IP_ADDRESS>
nvidia-cufft-cu12==<IP_ADDRESS>
nvidia-cusolver-cu12==<IP_ADDRESS>
nvidia-cusparse-cu12==<IP_ADDRESS>
nvidia-nccl-cu12==2.18.1
nvidia-nvjitlink-cu12==12.1.105

I checked the HLO when using dls=jnp.ones(shape=(10000, 3)) but it does indeed look like some very large tensors are being generated by your program (1 x 10000 x 71 x 75 x 71 x3 ~= 40GB)
ENTRY main.152 {
  constant.27 = f32[] constant(1)
  broadcast.28 = f32[1,71]{1,0} broadcast(constant.27), dimensions={}
  iota.29 = s32[71]{0} iota(), iota_dimension=0
  ...
  constant.15 = f32[3,3]{1,0} constant({ { 1, 0, 0 }, { 0, 1, 0 }, { 0, 0, -1 } })
  dot.95 = f32[1,71,75,71,3]{4,3,2,1,0} dot(scatter.89, constant.15), lhs_contracting_dims={4}, rhs_contracting_dims={0}
  reshape.96 = f32[1,1,71,75,71,3]{5,4,3,2,1,0} reshape(dot.95)
  broadcast.97 = f32[1,1,71,75,71,3]{5,4,3,2,1,0} broadcast(reshape.96), dimensions={0,1,2,3,4,5}
  reshape.98 = f32[1,71,75,71,3]{4,3,2,1,0} reshape(broadcast.97)
  broadcast.99 = f32[1,10000,71,75,71,3]{5,4,3,2,1,0} broadcast(reshape.98), dimensions={0,2,3,4,5}
  subtract.100 = f32[1,10000,71,75,71,3]{5,4,3,2,1,0} subtract(broadcast.99, broadcast.17)
  multiply.132 = f32[1,10000,71,75,71,3]{5,4,3,2,1,0} multiply(subtract.100, subtract.100)
  divide.133 = f32[1,10000,71,75,71,3]{5,4,3,2,1,0} divide(multiply.132, broadcast.4)
  reduce.138 = f32[1,10000,71,75,71]{4,3,2,1,0} reduce(divide.133, constant.25), dimensions={5}, to_apply=region_3.134
  multiply.139 = f32[1,10000,71,75,71]{4,3,2,1,0} multiply(reduce.138, broadcast.2)
  exponential.140 = f32[1,10000,71,75,71]{4,3,2,1,0} exponential(multiply.139)
  add.141 = f32[1,10000,71,75,71]{4,3,2,1,0} add(exponential.131, exponential.140)
  multiply.146 = f32[1,10000,71,75,71]{4,3,2,1,0} multiply(broadcast.145, add.141)
  ROOT reduce.151 = f32[10000,71]{1,0} reduce(multiply.146, constant.25), dimensions={0,2,3}, to_apply=region_4.147
}

After commenting out the two lines containing exp these large tensors are not materialized:
  ...
  constant.12 = f32[] constant(1)
  reduce.27 = f32[1,71]{1,0} reduce(broadcast.6, constant.12), dimensions={2}, to_apply=region_0.23
  constant.1 = f32[] constant(15.7496099)
  broadcast.2 = f32[1,71]{1,0} broadcast(constant.1), dimensions={}
  multiply.28 = f32[1,71]{1,0} multiply(reduce.27, broadcast.2)
  reshape.29 = f32[1,1,71]{2,1,0} reshape(multiply.28)
  broadcast.34 = f32[1,1,71]{2,1,0} broadcast(reshape.29), dimensions={0,1,2}
  reshape.35 = f32[1,71]{1,0} reshape(broadcast.34)
  broadcast.36 = f32[1,71,71]{2,1,0} broadcast(reshape.35), dimensions={0,2}
  divide.37 = f32[1,71,71]{2,1,0} divide(broadcast.33, broadcast.36)
  broadcast.38 = f32[1,10000,71,75,71]{4,3,2,1,0} broadcast(divide.37), dimensions={0,2,4}
  constant.11 = f32[] constant(0)
  ROOT reduce.43 = f32[10000,71]{1,0} reduce(broadcast.38, constant.11), dimensions={0,2,3}, to_apply=region_1.39
}

I'm not sure why thus code runs on Jax <0.4.14... it's possible there's some optimizations being done differently. You can inspect the compiled code yourself using:
run.lower().compiler_ir(dialect='hlo').as_hlo_text() (for >=0.4.30)
jax.xla_computation(run)().as_hlo_text() (for <0.4.30)

Thanks for the response. I'm starting to think it is some change in openxla or lower that is responsible rather than jax itself. A few questions:

what part of the hlo text shows when a tensor is "materialized"? is there docs/links on how to read these outputs?
what's the difference between func.lower().as_text() and run.lower().compiler_ir(dialect='hlo').as_hlo_text()?
how do you determine expected memory based on tensor shape?

Does this seem like a bug or just an old edge case not working anymore do you think? When using dls=jnp.ones(shape=(1590, 3)) the program ran successfully and pprof reported ~500kB of memory usage, but increasing to  dls=jnp.ones(shape=(1600, 3)) fails trying to allocate ~5GB, which seems like strange behavior.


[WIP][SPARK-31036][SQL] Use stringArgs in Expression.toString to respect hidden parameters

What changes were proposed in this pull request?
This PR proposes to respect hidden parameters by using stringArgs  in Expression.toString . By this, we can show the strings properly in some cases such as NonSQLExpression.
Why are the changes needed?
To respect "hidden" arguments in the string representation.
Does this PR introduce any user-facing change?
Yes, for example, on the top of https://github.com/apache/spark/pull/27657,
val identify = udf((input: Seq[Int]) => input)
spark.range(10).select(identify(array("id"))).show()

shows hidden parameter useStringTypeWhenEmpty.
+---------------------+
|UDF(array(id, false))|
+---------------------+
|                  [0]|
|                  [1]|
...

whereas:
spark.range(10).select(array("id")).show()

+---------+
|array(id)|
+---------+
|      [0]|
|      [1]|
...

How was this patch tested?
By existing tests.

Test build #119296 has started for PR 27788 at commit 41085bd.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24037/
Test PASSed.

Test build #119296 has finished for PR 27788 at commit 41085bd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119296/
Test PASSed.

Test build #119380 has started for PR 27788 at commit e886efd.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24117/
Test PASSed.

cc @cloud-fan

Test build #119380 has finished for PR 27788 at commit e886efd.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119380/
Test FAILed.

retest this please

Test build #119400 has started for PR 27788 at commit e886efd.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24137/
Test PASSed.

Test build #119400 has finished for PR 27788 at commit e886efd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119400/
Test PASSed.


rendering in pygame

Steps to do:

[x] Implement EnvRenderer with pygame
[x] Add wheels to visualize the rotation
[x] Implement zoom-in and follow a given vehicle (e.g., ego)
[ ] Implement a set of callbacks to visualize global/local plans

Current implementation in branch dev_rendering:

The car is drawn with line segments connecting the vehicle vertices and additional segments for the wheels. The reason to use lines instead of rectangles is that rotating rectangles in pygame results in deformation during execution (the car looks like slightly changing its size).
The only way I found is to blit a surface with the map layout at each call, this is not so efficient and represents the main bottleneck at the moment. Is there any way to draw the map once at the beginning and then only draw the vehicles? I have tried to draw only the point of the track but this is even worse (it halves the fps)
To avoid zooming in at each call, I rescale the map according to the zoom factor and then center the window on the agent to follow. For large zooming factor (>3), this slows down the rendering.
Performance on my laptop running the example with waypoint follower:

rendering
simulation time
real time

None
15.0
3.7

human
15.0
18.7

rgb_array
15.0
26.9

rgb_array + save video
15.0
48.3

Still to:

check rendering with multiple agents
add rendering callbacks to visualize trajectories

Any idea? @nandantumu @hzheng40

I have added 3 rendering callbacks to:

draw closed line from a sequence of points (e.g., raceline)
draw unclosed line from a sequence of points (e.g., local plan)
draw points (e.g., pose, marker for lookahead point)

It looks like this

and the overhead is the following in "human" mode:
|---------------|---------------|---------------|---------------|

render
sim_time
elapsed_time
fps

no callbacks
15.0
15.19
98.73

waypoints
15.0
18.46
81.24

waypoints+lookahed
15.0
16.61
90.30

There are now two configuration files for rendering, "rendering.yaml" and "rendering_fast.yaml".
The size of the produced windows, zoom levels, etc are different to speed up the rendering.
On my laptop, I can render a single PP agent at 100 Hz for "human" and 4-500Hz for "human_fast"
Updated benchmarking:

Single PP Agent, different render modes

key
sim_time
elapsed_time
fps

None
15.0
0.77
1951.13

human
15.0
16.48
91.03

human_fast
15.0
3.47
432.02

rgb_array
15.0
18.63
80.52

rgb_array_list
15.0
18.17
82.57

Human render mode, different number of agents

key
sim_time
elapsed_time
fps

1
15.0
18.71
80.19

2
15.0
15.99
93.82

3
15.0
15.62
96.03

4
15.0
23.90
62.76

5
15.0
20.13
74.50

...
...
...
...

10
15.0
37.09
40.45


Adds a setting that allows disabling the 'failIfMajorPerformanceCaveat' flag used in the 'isWebGLSupported' function

https://github.com/pixijs/pixi.js/issues/5774
https://github.com/pixijs/pixi.js/issues/5778

Codecov Report

Merging #5811 into dev will decrease coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##              dev    #5811      +/-   ##
==========================================
- Coverage   70.56%   70.56%   -0.01%
==========================================
  Files         205      205
  Lines       10383    10384       +1
==========================================
  Hits         7327     7327
- Misses       3056     3057       +1

Impacted Files
Coverage Δ

packages/utils/src/settings.js
100% <100%> (ø)
:arrow_up:

packages/utils/src/browser/isWebGLSupported.js
88.23% <100%> (ø)
:arrow_up:

packages/ticker/src/Ticker.js
87.9% <0%> (-0.81%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5cd5a51...47913bf. Read the comment docs.


Fixes bug in flatten for Sec

This fixes issue https://github.com/granule-project/granule/issues/223 where the flatten operation for Sec was set to "meet" which only works for Level not Sec (there was some accidental conflation between the two here  which is not separated by the SecurityLevels pragma).

Adds the test from the issue.


Driver hangs with empty result set

Reproducer available.  Execute via ./go-mssql-repro-1 --hostname <ip> --username <username> --password <password>
When executing queries (with certain attributes, detailed below) that do not return a result set the driver hangs, expecting to read data that isn't ever coming.
Using this based-on-real-world-issue query as an exemplar:
SELECT  TOP 50 'Top 50 queries' as Description,
	                  a.*,
	                  SUBSTRING(SqlText, (qs.statement_start_offset/2)+1,
	   	((CASE qs.statement_end_offset
	   		WHEN -1 THEN DATALENGTH(SqlText)
	   		ELSE qs.statement_end_offset
	   		END - qs.statement_start_offset)/2) + 1) as statement,
	       		qs.*,
	       		queryplan.query_plan as query_plan_ext_xml
	   FROM (SELECT DB_NAME(dbid) as [Database],
	                plan_handle,
	                UseCounts,
	                RefCounts,
	                size_in_bytes,
	                Cacheobjtype,
	                Objtype,
	                st.text as SqlText
	         FROM sys.dm_exec_cached_plans cp
	                  CROSS APPLY sys.dm_exec_sql_text(plan_handle) st
	         WHERE (LEFT(TEXT,300) LIKE '%SOME_MATCHING_TEXT%')) a
	            CROSS APPLY sys.dm_exec_query_plan(a.plan_handle) queryplan
	            INNER JOIN sys.dm_exec_query_stats qs on qs.plan_handle = a.plan_handle
	   	  WHERE queryplan.query_plan IS NOT NULL AND DATEDIFF(hour,qs.last_execution_time,GETDATE()) < 12
	         ORDER BY qs.total_elapsed_time DESC

As written the driver hangs executing that query, never to return.  Removing the ORDER BY or TOP clause prevents hanging.  Other nearly identical queries (with WHERE clauses that match results) do not hang.
To Reproduce
Reproducer available.  Execute via ./go-mssql-repro-1 --hostname <ip> --username <username> --password <password>
Expected behavior
Expecting that empty result sets are returned successfully, with rows.Next() returning false on first call.
Further technical details
SQL Server version: SQL Server 2019
Operating system: MacOS 13.1 M1 Pro

Originally the query had a ROW_NUMBER construct that also appeared to cause (at least, part) the hanging.

Possibly related. Driver hangs even when not expecting a result set.
For example, a simple call to sp_addarticle hangs indefinitely when using the Exec method, which doesn't expect results.
E.g...
qry = "exec sp_addarticle @force_invalidate_snapshot = 1, @publication = N'EXMPL', @article = N'" + filter.Table + "', @source_owner = N'dbo', @source_object = N'" + filter.Table + "', @type = N'logbased', @description = N'', @creation_script = N'', @pre_creation_cmd = N'drop', @schema_option = 0x000000000803509F, @identityrangemanagementoption = N'manual', @destination_table = N'" + filter.Table + "', @destination_owner = N'dbo', @status = 24, @vertical_partition = N'false', @ins_cmd = N'CALL [sp_MSins_dbo" + filter.Table + "]', @del_cmd = N'CALL [sp_MSdel_dbo" + filter.Table + "]', @upd_cmd = N'SCALL [sp_MSupd_dbo" + filter.Table + "]'" 				fmt.Println("Adding to replication:\n", qry) 				_, err = sqlserv.Exec(qry) 				if err != nil { 					log.Println(err) 					break 				}


V1 migration creates duplicate cognito user pool

Describe the bug
Running the v1 auto migration created a new cognito userpool and attached it to my API, making it impossible for my users to log in.
To Reproduce
Steps to reproduce the behavior:

Run v1 auto migration on project with AppSync and Cognito user pools
Note: this userpool was initially created with the amplify-cli as part of my GraphQL API project

Expected behavior
Migration should use existing configured userpool, not create a new one
Desktop (please complete the following information):

OS: macOS

Additional context
amplify-cli: v1.0.5
Help!
How can I hook back up the original userpool?

You can visit the user pool console and place the cognito ID in your aws_exports.js file for your app to use the intended user pool. That being said did you manually configure the cognito user pool CloudFormation template previously?

I'll try that. I generated the userpool using the amplify cli. However, I did manually modify the template add email as the username attribute.

It looks like it will be more than just that. It also deleted the app clients from the original userpool, except for the client I manually created for using from the AppSync console.
And if I replace the values in aws-exports.js, will they be overridden by the cli at some point?

I have not migrated yet to v1 but it seems I might run into this as well because I also manually modified the cognito cloudformation file. @flybayer it's not clear from your comments, but were you able to resolve this / find a workaround? I wouldn't want my cognito user pool to be permanently borked and I hope that did not happen.

@hisham So the CLI migrates your AppSync API first (to make use of custom resolvers), then it changes auth CFN files locally and then before pushing, you would be prompted whether you want to push the resources or not. I would recommend you to say 'No' in your case for 'pushing' the changed CFN files and make sure you do a git diff and incorporate changes in the modified CFN to reflect your current cloudformation modifications (specially tied to immutable CFN attributes like sign in/sign up attributes).

Sounds good, will do when I migrate. Thanks @kaustavghosh06


Can Mantle be open sourced?

Can Mantle to PAL layer be open sourced?
Thanks!

I didn't hear there is any plan.


nix flake lock --update haskellNix

Update haskellNix in preparation to the node update.
The PR only changes flake.lock file as a result of running nix flake lock --update haskellNix
Issue Number
ADP-2956

This thing is failing no matter how hard I try: haskell-language-server won't compile with the updated haskellNix 😠


[Bug] The new SA used by autoscaler caused permission errors

Search before asking

[X] I searched the issues and found no similar issues.

KubeRay Component
ray-operator
What happened + What you expected to happen
When I set enableInTreeAutoscaling to true for my Ray cluster, the kuberay-operator creates a new k8s service account with the same name of the cluster (if it's not available) and uses it for the head node pod (see here). Our Ray clusters on Kubernetes use a specific pre-configured service account in the namespace. Switching to a different service account would cause the permission issues when users try to use Ray to read the data. The autoscaler should use the existing service account defined in the RayCluster resource object for the head node instead of creating a new one to use.
Reproduction script
The bug can be reproduced by following these steps.

Define a k8s service account named default-sa
Create a RayCluster with the name of foo and use default-sa for its head node
Set enableInTreeAutoscaling to true for the Ray cluster you just created
The Kuberay operator will restart the head node pod, inject the autoscaler container, and use foo as its service account

Anything else
No response
Are you willing to submit a PR?

[x] Yes I am willing to submit a PR!

@daikeshi I think this is a problem happen in your environment? BTW, The reason you have pre-configure service account is because it binds to GCP Service Account that has the permission to read the data?
Let me try to understand the problem deeply. The ray program which reads the data is scheduled either on head or worker. Autoscaler container is embedded inside head pod and it creates worker without specifying the service account. It should use default one by default.
ideally we should ask both head and worker use your preconfigured SA, right? So the problem is autoscaler create the pods without delegate sa to worker template?
Is it ok to reuse the service account if RayCluster service account is provided?

Yes, we should reuse the service account if RayCluster service account is provided.
In our case, the service account is pre-configured in each namespace with GKE workload identity. We currently deploy RayCluster directly via raw YAML manifests, so we are able to configure service account for the head node and workers. I think the workers launched by the autoscaler was able to pick up the correct service account defined in the RayCluster. It's just the head node that tries to use a new one when it's injecting the autoscaler container. We could simply check and reuse the existing one if RayCluster has the service account defined.
We are happy to take a stab of this bug.


python311Packages.agate-sql: 0.5.9 -> 0.7.0

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for python311Packages.agate-sql is: Adds SQL read/write support to agate.
meta.homepage for python311Packages.agate-sql is: https://github.com/wireservice/agate-sql
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found 0.7.0 with grep in /nix/store/gv9r4vs16pgfjwhn9jhrlj96npa4ycwp-python3.11-agate-sql-0.7.0
found 0.7.0 in filename of file in /nix/store/gv9r4vs16pgfjwhn9jhrlj96npa4ycwp-python3.11-agate-sql-0.7.0

Rebuild report (if merged into master) (click to expand)

3 total rebuild path(s)

3 package rebuild(s)

First fifty rebuilds by attrpath
csvkit
python310Packages.agate-sql
python311Packages.agate-sql

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/gv9r4vs16pgfjwhn9jhrlj96npa4ycwp-python3.11-agate-sql-0.7.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A python311Packages.agate-sql https://github.com/r-ryantm/nixpkgs/archive/201fd804acfb91e4f08c27074f5cedf248e3ccd8.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/201fd804acfb91e4f08c27074f5cedf248e3ccd8#python311Packages.agate-sql

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/gv9r4vs16pgfjwhn9jhrlj96npa4ycwp-python3.11-agate-sql-0.7.0
ls -la /nix/store/gv9r4vs16pgfjwhn9jhrlj96npa4ycwp-python3.11-agate-sql-0.7.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

2 packages marked as broken and skipped:

python310Packages.agate-sql
python311Packages.agate-sql

1 package built:

csvkit

Maintainer pings
cc @vrthra for testing.

Result of nixpkgs-review pr 261972 run on x86_64-linux 1

6 packages built:

csvkit
csvkit.dist
python310Packages.agate-sql
python310Packages.agate-sql.dist
python311Packages.agate-sql
python311Packages.agate-sql.dist


Thread 1: EXC_BAD_ACCESS (code=1, address=0x0)

Hi,
I am getting:
"Thread 1: EXC_BAD_ACCESS (code=1, address=0x0)" error when I build from XCode on real device(iPhone 6s Plus).
Also I'm not sure if thats relevant but I also get following error on xcode console:
"2019-01-15 17:50:11.646908+0300 MyAppName[3979:1856266] [] nw_connection_get_connected_socket 152 Connection has no connected handler
2019-01-15 17:50:11.647018+0300 MyAppName[3979:1856266] TCP Conn 0x1c416edc0 Failed : error 0:61 [61]"
My unity version: 2018.2.20f1 and RN version: 0.57.5
Any help is appreciated!

@alisherakb  I get this error  "Thread 1: EXC_BAD_ACCESS (code=1, address=0x0)"  by
UnityInitRuntime(g_argc, g_argv);  inside UnityUtils.mm
Unity v2018.3.13f1,
react-native-unity-view: v1.3.3
xcode 10

@alisherakb I get this error "Thread 1: EXC_BAD_ACCESS (code=1, address=0x0)" by
UnityInitRuntime(g_argc, g_argv); inside UnityUtils.mm
Unity v2018.3.13f1,
react-native-unity-view: v1.3.3
xcode 10
Any help would be appreciated

Getting the same error
Xcode 9
Unity 2018.2.17

@OgieBen Is the issue still relevant? I can assist you.


{lamb} instead of lambda sign (possibly a UTF-8 problem?)

Instead of getting a pretty lambda sign, I get {lamb} as my prompt. :disappointed:

I have the same problem when running Cmder as root, but everything is fine when I run it as me. In a root tab I get a warning that translates to

System can't find the given path.
File not found - C:\Users\User\Copy\random\config\aliases

I have it in a directory like "C:\Users\User\Copy\random stuff", so it looks like spaces aren't handled properly in root mode.

Same issue on Windows 10 Technical Preview is solved by upgrading to Clink 0.4.3

Windows 10 RTM I am having this issue. I installed/upgraded to Clink 0.4.3 via Chocolatey and I am still stuck with {lamb}

Same problem for me on latest Windows 10, I have no other Clink installed next to the one included with Cmder

any solution?

Download the latest pre-release, do not install anything from chocolatey.

i downloaded the latest version of clink and still have this issue
(using 1.2.9)

Clink is included with cmder already.

The latest prerelease didn't fix the issue.

yeah got that problem in Windows 10 on my work computer like 2 days ago aswell, when opening up a new cmd.exe :(

@MartiUK people still having issues, maybe this should be re-opened?

i fix the issue by uninstall clink on my PC before i use cmder

I fixed it by installing a updated version to the clink folder in cmder.

I also did the above to fix it, the pre-release didn't solve it for me


Fixed possible buffer overflow in framer for bogus headers on 32b platforms.

Updated to remove that unintended arch condition. The bounds checking should now be 100% covered, since SetBuffer() guarantees both minimum and maximum capacity now.


Automated Updates for latest releases

Make use of scrapers and nightly pipelines to check and update info for the different tools. At https://github.com/projectlint/OS-lifecycle I have something like this, just only I need to finish to tune-up :-)

Worked a bit on this and tried to leverage GitHub Dependabot instead of doing the hard bits on our own (scraping). The idea was to:

Have a separate directory for each product release cycle
That gets independently updated by GitHub on a daily schedule
Using the final output from the version manager (lockfiles) as the input to Jekyll.

For some tools like bootstrap/django/rails (which are available on package managers) this is fairly straight-forward.
It gets tricky however with lower level projects (such as node or PHP). Turns out that Dependabot supports terraform, which (by virtue of not having a manifest file) lets you include any git repository as a module.
Additionally, while Terraform does not support version constraints or lockfiles for modules, these can be configured using Dependabot. So if you write the following file:
module "playtube" {
  source = "git::https://github.com/nodejs/node?ref=v16.0.0"
}

and configure Dependabot correctly, it will file the correct PRs for you to upgrade this to the latest node release. See the open PRs and dependabot configuration here: https://github.com/captn3m0/dependabot-github-test
I haven't tested it yet, but terraform supports arbitary git repos, svn and hg as well. I don't think hg/svn are supported by Dependabot, but generic git repos might work well for us.
Overall, the idea is to get to a workflow where major version bumps are maintained by contributors while minor and patch releases get taken up by this automatic update cycle.

Filed an issue on github/docs, because it doesn't specify what all sources are valid: https://github.com/github/docs/issues/8419
Tested against mercurial, and it doesn't seem supported.

It would have been great if dependabot supported Terraform JSON, but alas - it doesn't (yet). If it were, we could read the final data directly into Jekyll, without any processing, thus making the pipeline much simpler to understand.

Our automation for minor releases is live 🎉  and functional. You can see the update PR at #1260, #1258 for reference.
All of this is documented at https://github.com/endoflife-date/endoflife.date/wiki/Automation with some suggestions on what can be taken up next.
If anyone would like to contribute a new product, you can start by:

Filing a PR providing automation information in the product.md file. See the auto key defined in CONTRIBUTING.md for the same. This will track Git/DockerHub for now, but we can add more (such as packagist/rubygems etc if needed).
If the data is available elsewhere (such as a webpage), you can contribute a small script in the release-data directory that generates a JSON file containing versions alongside date of the release. Sample scripts: unrealircd, haproxy).
Providing details about where this information can be found and creating an issue, or noting in the relevant talk page.


Static brotli files on production build

Please consider adding generation of static brotli files during production build
$ nuxt build

Hi @OlzhasAlexandrov
Actually we compress the files on nuxt generate with html-minifier since the others files are minified already. The files with nuxt build will soon be compressed too.
Thanks for your interest!


OOP school library: refactor your code

### OOP school library: refactor your code
This pull request addresses the need for code refactoring in the OOP School Library project. The goal of this refactoring is to improve code quality, maintainability, and readability, ensuring that the project adheres to best practices and design principles.

@ichala thanks for the review


workspace switch action works unreasonable when focus popup window

Observed behavior
when focus in popup window , some action works unreasonable

some keybinding not effect like switch-workspace
switch-workspace in popup widow case other workspace layout crash

Expected behavior
works like common sense
Steps to reproduce

create multiple workspace
action for popup window like popup-scrach/magit/help
then switch focuse in popup window
press M-1..9 to switch workspace not work
M-x +workspace:switch-next or others commands

System information

Click to expand
- OS: gnu/linux (x86_64-pc-linux-gnu)
- Emacs: 25.3.1 (Feb 23, 2018)
- Doom: 2.0.9 (develop https://github.com/hlissner/doom-emacs/commit/dfb6bc162d14f16b9031458f58f695fb33f81ff1)
- Graphic display: t (daemon: nil)
- System features: XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS NOTIFY GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB TOOLKIT_SCROLL_BARS GTK3 X11
- Details:
  ```elisp
  modules:   (:config :feature debugger eval evil file-templates (lookup (+devdocs +docsets)) services snippets spellcheck syntax-checker version-control workspaces :completion ivy :ui doom-dashboard doom-modeline doom-quit hl-todo nav-flash evil-goggles vi-tilde-fringe window-select :tools electric-indent eshell gist imenu impatient-mode macos make neotree pdf rotate-text term tmux upload :lang emacs-lisp go hy javascript latex ledger lua markdown (org (+attach +babel +capture +export +present +publish)) plantuml python rest rust sh web :app :config :private org chinese)
  packages:  (ace-link ace-window adaptive-wrap all-the-icons anaconda-mode async atomic-chrome auctex auto-compile auto-yasnippet avy (centered-window :recipe (centered-window :fetcher github :repo anler/centered-window-mode)) coffee-mode command-log-mode company company-anaconda company-auctex company-dict company-go company-lua company-quickhelp company-racer company-restclient company-shell company-statistics company-tern company-web counsel (counsel-css :recipe (counsel-css :fetcher github :repo hlissner/emacs-counsel-css)) counsel-dash counsel-projectile counsel-tramp devdocs dired-k docker-tramp dockerfile-mode doom-themes dumb-jump editorconfig (emacs-snippets :recipe (emacs-snippets :fetcher github :repo hlissner/emacs-snippets :files (*))) emmet-mode eslintd-fix evil evil-anzu evil-args evil-commentary evil-easymotion evil-embrace evil-escape evil-exchange evil-goggles evil-indent-plus evil-ledger evil-magit evil-matchit evil-mc evil-multiedit evil-numbers evil-org evil-snipe evil-surround evil-textobj-anyblock evil-vimish-fold evil-visualstar (exec-path-from-shell :ignore t) expand-region fcitx flycheck flycheck-cask flycheck-ledger flycheck-plantuml flycheck-popup-tip flycheck-pos-tip flycheck-rust flyspell flyspell-correct flyspell-correct-ivy fringe-helper gist git-gutter-fringe git-link git-timemachine gitconfig-mode gitignore-mode go-eldoc go-guru go-mode gorepl-mode graphviz-dot-mode haml-mode helpful highlight-indentation highlight-numbers highlight-quoted hl-todo htmlize hy-mode hydra imenu-anywhere imenu-list impatient-mode ivy ivy-bibtex ivy-hydra ivy-rich ivy-xref js2-mode js2-refactor json-mode keyfreq langtool ledger-mode less-css-mode lsp-python lua-mode macrostep magit magithub makefile-executor markdown-mode markdown-toc mixed-pitch moonscript multi-term nav-flash neotree nlinum nlinum-hl nlinum-relative nodejs-repl nose ob-go ob-mongo ob-rust ob-sql-mode ob-translate org-brain (org-bullets :recipe (org-bullets :fetcher github :repo Kaligule/org-bullets)) org-download org-plus-contrib org-tree-slide orgit (osx-clipboard :ignore t) overseer ox-pandoc ox-reveal pcre2el pdf-tools persistent-soft persp-mode pip-requirements pipenv plantuml-mode prodigy prog-fill projectile pug-mode py-isort quelpa quickrun racer rainbow-delimiters rainbow-mode realgud restclient rjsx-mode (rotate-text :recipe (rotate-text :fetcher github :repo debug-ito/rotate-text.el)) rust-mode sass-mode shackle shrink-path skewer-mode slime smart-forward smartparens smex solaire-mode ssh-deploy stylus-mode swiper synosaurus tern toc-org toml-mode undo-tree use-package vi-tilde-fringe vimrc-mode visual-fill-column web-beautify web-mode wgrep which-key whitespace-cleanup-mode wordnut xref-js2 yaml-mode (yapfify :recipe (yapfify :fetcher github :repo JorisE/yapfify)) yasnippet youdao-dictionary)
  elc dirs:  n/a
  exec-path: (/home/liubin/.cargo/bin /home/liubin/.pyenv/plugins/pyenv-virtualenv/shims /home/liubin/.pyenv/shims /home/liubin/.pyenv/bin /home/liubin/.local/bin /usr/local/sbin /usr/local/bin /usr/sbin /usr/bin /sbin /bin /usr/games /usr/local/games /snap/bin /usr/local/libexec/emacs/25.3/x86_64-pc-linux-gnu)

</details>

Just FYI, I use the following code as a partial workaround of this problem:
(defun +my-workspace/goto-main-window (pname frame)
  (let ((window (car (+my-doom-visible-windows))))
    (if (window-live-p window)
        (select-window window))))
(add-hook 'persp-before-switch-functions '+my-workspace/goto-main-window)

(defun +my-workspace/doom-visible-windows (&optional window-list)
  "Return a list of the visible, non-popup windows."
  (cl-loop for window in (or window-list (window-list))
           unless (window-dedicated-p window)
           collect window))

(defun +my-workspace/close-window-or-workspace ()
  "Close the selected window. If it's the last window in the workspace, close
the workspace and move to the next."
  (interactive)
  (let ((delete-window-fn (if (featurep 'evil) #'evil-window-delete #'delete-window)))
    (if (window-dedicated-p)
        (funcall delete-window-fn)
      (let ((current-persp-name (+workspace-current-name)))
        (cond ((or (+workspace--protected-p current-persp-name)
                   (cdr (+my-workspace/doom-visible-windows)))
               (funcall delete-window-fn))
              ((cdr (+workspace-list-names))
               (+workspace/delete current-persp-name)))))))

I've changes those bindings, which should make them available in popup windows. Give it a try and let me know if it doesn't fix your issue.

key binding available in popup window. but action still unnormal when switch workspace.

@fuxialexander These works fine @hlissner , thanks  for snippet,  why don't you  create PR?

@driftcrow I think it's a partial solution as I did not store the information of whether a point is in a pop up window before switch (and restore it after switch back). Although I do not need that, I'm not sure whether others will care about it.

Sorry for the late followup. A lot has changed in all this time. On the latest commit the switch-workspace keys appear to work from inside popups, so I'll consider this resolved. Let me know if that isn't the case and I will reopen the thread.


Installation Failure

Expected Behavior
Installation wirks
Actual Behavior
Installation fails, s. logiles
Steps to Reproduce
Install FarmBot Webserver on Ubuntu 18.04.
Results of git log -1
farmbot_logs.zip
farmbot_logs.zip
Run git log -1 and paste the results here.

@patricknoack
A few things:

It appears that you ran the commands outside of the installation directory. For commands like npm run test or rails api:start to work, you need to use the "cd" command to enter the installation directory where Farmbot-Web-App resides. Most likely cd ~/Farmbot-Web-App.
You did not delete unused configuration variables. The application will not run correctly with extra / unused config vars. Try pasting this into application.yml instead:

API_HOST:         "<IP_ADDRESS>"
API_PORT:         "3000"
MQTT_HOST:        "<IP_ADDRESS>"
NO_EMAILS:        "TRUE"
OS_UPDATE_SERVER: "https://api.github.com/repos/farmbot/farmbot_os/releases/late
SECRET_KEY_BASE:  "RUN `rake secret` AND PASTE THE RESULTS HERE"


⚠ Bofrost.lu has degraded performance

In 47720fb, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 1408 ms

Resolved: Bofrost.lu performance has improved in 7c3f705 after 13 minutes.


Get the hardware MAC

I can get the unique identifier of the device through the API interface. I need to use 3 or more slave nodes. Each slave node has a temperature sensor and transmits the temperature to the root node. I need to identify which temperature value comes from. node

You could use otLinkGetFactoryAssignedIeeeEui64() and include that as part of the application payload along with the temperature sensor.

Thank you so fast reply, I will try now

@NOA-Ben, any updates on this issue?


Unseen not working.

Description
When call /mark-unseen api.
Environment

Wppconnect-server version(s): v1.3.3
Browser: Chromium 99
OS: MAC OS BIG SUR
Node version: Node 14
WhatsApp version: 2.2214.12
MultiDevice (BETA): yes

Steps to Reproduce
POST - /mark-unseen
Return.
{ status: 'error', message: 'Error on mark unseen' }
Console is empty. Nothing happened.

Closed by  commit c3f6420f8129dd4069d2ba05aa911a352914afb4

I've tested.
The unseen method now is working but with a error...
It don't mark as unseen (unread), it marks as seen (read).
Even if we try to call the method again it keep marking as seen (read) not as unread (unseen).


Issue with Playing MCAP File in ROS 2

Description
I am trying to play an MCAP file using the following command:
ros2 bag play -s mcap diagnostic_0.mcap

Expected Behavior
The bag should play as expected without any errors.
Actual Behavior
I receive the following error:
yaml-cpp: error at line 1, column 12: bad conversion

System Details

Docker Image: Ubuntu 22.04
ROS 2 Distribution: Iron
Python Version: 3.10.12

Steps Taken
I have tried running the command with different ROS 2 distributions (Humble and Iron), but I receive the same error. I also created a simple MCAP file with some test topics, and it works as expected, which suggests that the read/write functionality for MCAP files is working.
Suspected Cause
It seems like the issue could be related to a custom YAML file that is used with the --storage-config-file option to decode the MCAP file. If this is the case, could you please provide this YAML configuration file, or advise how I can proceed?
Request
I would appreciate any help in resolving this error. If this is part of the test or if this issue is outside the scope of regular support, please ignore this question.
Thank you for your assistance!

Hi @DuneV
Thanks for noticing! Rosbag files have been updated in the Drive. Please download them again.
You have been the first to report this, so you'll have 1%/5 extra on your grade.
Good luck 😎

I'll reopen it meanwhile for visibility to others.


Pass in user defined parameters to deploy

Description
Currently in the zkapp-cli, the deploy function takes the following arguments:
 zkapp.deploy({ verificationKey, zkappKey: zkAppPrivateKey });

Is there a method to pass in user-defined arguments from their built smart contract?

Current state: If you want to do something fancy, roll-your-own deploy functionality -- and if you want standard, just important JSON input.
Something more fancy is lower priority

Example if you want to roll-your-own: https://github.com/o1-labs/snarkyjs/blob/cba3f5b5e4ac6f1fdca894d97ceb11d15e875608/src/examples/simple_zkapp_berkeley.ts


Please anyone can help me

I'm submitting a...

[ ] Bug report
[ ] Feature request
[ ] Documentation issue or request

In package

[ ] @ng-toolkit/init
[ ] @ng-toolkit/serverless
[ ] @ng-toolkit/universal
[ ] @ng-toolkit/pwa
[ ] @ng-toolkit/firebug

Current behavior

Expected behavior

Minimal reproduction of the problem with instructions

Example repository

What is the motivation / use case for changing the behavior?

Environment

Angular version: 7.3.7

- Node version: XX
- Platform:

Others:

⚠ Closed as invalid issue. Please, fill the issue template when opening new issues in order to help us track the bug.
Cheers!


Integration Test - IssueCommandFlow - 2022/08/11 06:38:36

Auto create for integration test.

#pri1


Bug: Reveal File in Spaces collapses all Space Folders

This feature does quite the opposite of what it should do ;)
it doesn't reveal a random note, it collapses to complete folder space.
please see the details in the YouTube video I recorded:
https://youtu.be/fSZ6ElxOB-Y
Random file. Expanded Tree in the Space:

Use the command palette to Reveal the File:

See the result, the collapsed space tree:

thanks in advance

+1 this has been broken for quite a while, would like to see this fixed


Fix changelog [skip ci]

Q
A

Is bugfix?
no

New feature?
no

Breaks BC?
no

Tests pass?
-

Fixed issues

This enh is for 2.0.8 version.

It does not matter any more. The list is being reordered automatically by RelaseController: https://github.com/yiisoft/yii2/blob/master/build/controllers/ReleaseController.php#L522

Thank you


Add Key Value output formatter

Here is an additional output formatter that simply creates a single line of comma separated key-value pairs.
The use-case is for ingesting the output of the measure command into InfluxDB, which is a data source for Grafana - basically so we can have nice dashboards showing the history of LOC, number of static methods etc from each CI build.

Hi, thanks for proposal.
I'd like to keep this package as simple as possible. The JSON already allow transforming to any format, so instead of hardcoding the output formatters here, anyone can do the conversion on the JSON output to any project they need.
Thanks for understanding

Not a problem but I figured key-value was a good generic second choice to be alongside JSON.
I was looking for a one-liner to use in CI without any additional dependencies apart from bash, to extract the data to key-values and figured it would be much easier to allow the lines package itself to do the work, rather than have some esoteric bash command.


about the parameters of train.py

Hello, thanks for sharing the wonderful code!
Could you please tell me how to configure the parameters of the train.py file to obtain the effect of the published model, or how many epochs of training are required.

Hi, thanks for the comment.
If you follow the instructions in the Readme, the default configuration should be sufficient.
Regarding the number of epochs, we trained until the validation results started to get worse. For COCO it was about 8 epochs but for smaller datasets (i.e. FlickrStyle10k), it was about 25 epochs.


Update git link for hero image

Struggled with content being imported and displaying properly. Also adjusted spacing and tags for accessibility. Struggled


Problem with month setter

Hello Amin,
watch this:
PrimeCalendar primeCalendar = new CivilCalendar();      primeCalendar.setMonth(12);
then if you check this, it's going to set 0 for month not 12!

Hello Parsa,
The behavior of PrimeCalendar library is similar to the java Calendar class. So, the month variable takes values in the range of 0 to 11.


Translate concepts/services-networking/add-entries-to-pod-etc-hosts-with-host-aliases.md in Korean

This is a Feature Request

What would you like to be added

Translate concepts/services-networking/add-entries-to-pod-etc-hosts-with-host-aliases.md in Korean
Why is this needed
No translation with concepts/services-networking/add-entries-to-pod-etc-hosts-with-host-aliases.md in Korean
Comments

Page to Update:
https://kubernetes.io/docs/concepts/services-networking/add-entries-to-pod-etc-hosts-with-host-aliases/

/language ko
/assign

안녕하세요.
@KimMJ 님께서 올려주신 이슈는 #17790 에 먼저 올라온걸로 확인이 되고 있습니다.
이에 다른 문서 선택을 요청드립니다.

@ysyukr
넵. 제가 확인을 잘못했네요.
다른 문서 찾아보겠습니다!


swift run start error

My environment is M1 Max Mac os 14.
However, when I run "swift run -c release FuzzilliCli --profile=v8 /Users/studiojihun/chrome/src/v8/src", an error occurs.
Error is
Building for production...
Build complete! (0.09s)
[Cli] Using the following arguments for the target engine: ["--expose-gc", "--omit-quit", "--allow-natives-syntax", "--fuzzing", "--jit-fuzzing", "--future", "--harmony", "--js-staging"]
[Cli] Enabled mutators: ["ExplorationMutator", "CodeGenMutator", "SpliceMutator", "ProbingMutator", "InputMutator", "InputMutator (type aware)", "OperationMutator", "CombineMutator"]
[Cli] No filesystem storage configured, found crashes will be discarded!
[REPRL] Script execution failed: Did not receive HELO message from child: No such process. Retrying in 1 second...
[REPRL] Script execution failed again: Did not receive HELO message from child: Operation timed out. Giving up
How should I solve this?

The v8 tested by Fuzzilli needs extra compile args v8_fuzzilli=true, Please make sure this  parameter exists when compiling
Below is an example
gn gen out.gn/fuzzilli_build --args='is_debug=false dcheck_always_on=true v8_static_library=true v8_enable_verify_heap=true v8_fuzzilli=true sanitizer_coverage_flags="trace-pc-guard" target_cpu="x64"'
ninja -C ./out.gn/fuzzilli_build d8


type-at-pos shows imported MyType

Seems like functionality changed a bit from version 0.68.0 to version 0.69.0.
with the below files: flow type-at-pos index.js 4 12
version 0.68.0 -> type MyType = {a: string, b: number}
version 0.69.0 -> imported MyType
Is this expected? What is weird is that under either version if i declare OtherType in the same file i'll get behavior like 0.68.0 type OtherType = {x: string}.
Is there a way to get more information out of an imported type alias besides the name and the fact that it was imported. It would be good to have this information on hover in editors.
Reproduction:
create-react-app someName
flow: 0.69.0 (seems like most version higher as well)
types.js
// @flow
type MyType = {
  a: string,
  b: number,
}
export type { MyType };

index.js
// @flow
import type { MyType } from './types';
type OtherType = { x: string };
const x: MyType = { a: 'a', b: 1 };
//                  ^ 'imported MyType'
const y: OtherType = { x: 'x' };

If more information is needed please let me know and i'll provide it.

Hi @XaxisJosh! Thanks for bringing this up. This was a deliberate change to remove clutter from imported types when doing type-at-pos, but I wasn't feeling very strongly about it.
Since you're bringing it up, I'll try to recover the extra info.

Hi!

This was a deliberate change to remove clutter from imported types when doing type-at-pos

I am curious when the clutter was a problem, or how did it manifest? Also would it be possible to put the imported Type behaviour in .flowconfig? I imagine it can be useful in some cases.
However, I agree that the old behaviour was more useful when using Flow as part of an IDE/editor setup.

Before a major rewrite of the type intelligence services (around the version that you mentioned above), Flow would occasionally produce very large types on type-at-pos because it expanded all type aliases. This had a couple drawbacks:

Printing out these types often caused slowdown.
It's impossible to print out recursive types, and so Flow would print any when such a type occurred.

Now, a lot changed since then. For the above reasons, Flow does not expand type aliases at type-at-pos (unless your hovering over a type alias name itself - there it'll do a one-off expansion of the type). As part of the same effort imported types were not expanded at all (the issue at hand here). I'm fairly confident that doing the one off expansion for imported types shouldn't cause any major regressions.
Also note that running type-at-pos in the terminal with the --expand-json-output (in more recent versions) already gives you the output you're looking for in the "expanded_type" field:
flow type-at-pos main.js 4 10 --pretty --expand-json-output
{
  "expanded_type":{
    "kind":"TypeAlias",
    "name":{
      "provenance":{"kind":"Imported","loc":"types.js:2:15,5:1"},
      "name":"MyType"
    },
    "typeParams":null,
    "body":{
      "kind":"Obj",
      "exact":false,
      "frozen":false,
      "props":[
        {
          "kind":"NamedProp",
          "prop":{
            "name":"a",
            "prop":{"kind":"field","type":{"kind":"Str"},"polarity":"Neutral","optional":false}
          }
        },
        {
          "kind":"NamedProp",
          "prop":{
            "name":"b",
            "prop":{"kind":"field","type":{"kind":"Num"},"polarity":"Neutral","optional":false}
          }
        }
      ]
    }
  },
  "type":"imported MyType",
  "reasons":[],
  "loc":{
    "source":"main.js",
    "type":"SourceFile",
    "start":{"line":4,"column":10,"offset":88},
    "end":{"line":4,"column":15,"offset":94}
  },
  "path":"main.js",
  "line":4,
  "endline":4,
  "start":10,
  "end":15
}

@goodmind I think we can close this (was fixed in 0.102)


Update jiralib.py

The  /rest/auth/1/session endpoint is deprecated for Jira Cloud, though this works for Server (at least the version we have 🤔).
https://developer.atlassian.com/cloud/jira/platform/deprecation-notice-basic-auth-and-cookie-based-auth/
In addition, when using basic auth with this integration, we're going to have to use our email as the username and a Jira API token as the password.
@zbazztian

Wait, are you saying most of the things work with cloud and that's the only thing you had to change?

Ok, #14 suggests that it actually does work. Thanks for trying this out, that's excellent news!

Yep that's the only thing that had to be changed, fortunately haha! 🙌🏼


feat: update to angular 15

updated to Angular 15
update to MDC components
updated the library version to v15 to follow angular versions

@llorenspujol can you review the PR

Merged! 🚀
I added some small fixes, nothing major.
I have set the Angular Material density to 0 for now. Yes, I agree that the inputs with the default density are to big 😆. I may revisit this after upgrading to v16, but it is not a priority for now.
Thanks @sheikalthaf ! 👏


update test and absolute path imports + new feature search dataset re…

Changelog:

use absolute imports for ts files within app (using notation @)
update tests to match the new CKAN endpoint results (only one test failing)
add new feature: searchDatasetRawResults to get the raw CKAN result JSON including things like facets and count

Thanks for merging. Could you also publish the new version on npm? Cheers.


Problems with uploading technology table

I tried to upload an csv technology table after I added to nodes but didn't change the old ones. The old technology file was downloaded and uploaded in the new one without any editing. Did didn't work.
After calling with Joris I added the nodes manually in the technology file. Uploading the file in te existing  LES didn't work but uploading in a new LES did. In the new LES I am able to do everything in the dashboard but I cannot use the edit LES button. The ID of the new LES is 500.

Assigning @grdw to look into this with priority.

I have already fixed this on master / beta. I'll deploy the fix to production.

Thanks @antw! I opened Jeroens LES (#500), but am still not able to edit it. Would you have time to look into this?

Will do.

This is fixed manually by @antw . Closing this issue.

It works on my computer too. Thank you guys.


Small typo in easyinstall.sh menu

Info

Which version of Pi are you using: Raspberry Pi Zero 2 W Rev 1.0
Which github revision of software: v23.11.01
Which board version: FULLSPEC
Which computer is the PiSCSI connected to: n/a
Which OS you are using (output of 'lsb_release -a'): Raspbian GNU/Linux 11 (bullseye)

Describe the issue
The menu items in easyinstall.sh are numbered from 1 to 17, but on the line to choose which menu item it says "(0-17)":
pi@rascsi:~/piscsi $ ./easyinstall.sh
    .~~.   .~~.
   '. \ ' ' / .'
    .╔═══════╗.
   : ║|¯¯¯¯¯|║ :
  ~ (║|_____|║) ~
 ( : ║ .  __ ║ : )
  ~ .╚╦═════╦╝. ~
   (  ¯¯¯¯¯¯¯  ) PiSCSI Assistant
    '~ .~~~. ~'
        '~'

Board Type: FULLSPEC | Compiler: clang++ | Compiler Cores: 1

Choose among the following options:
INSTALL/UPDATE PISCSI
  1) Install or update PiSCSI Service + Web Interface
  2) Install or update PiSCSI Service
  3) Install or update PiSCSI OLED Screen (requires hardware)
  4) Install or update PiSCSI Control Board UI (requires hardware)
NETWORK BRIDGE ASSISTANT
  5) Configure network bridge for Ethernet (DHCP)
  6) Configure network bridge for WiFi (static IP + NAT)
INSTALL COMPANION APPS
  7) Install AppleShare File Server (Netatalk)
  8) Install FTP File Server (vsftpd)
  9) Install SMB File Server (Samba)
 10) Install Web Proxy Server (Macproxy)
ADVANCED OPTIONS
 11) Compile and install PiSCSI stand-alone
 12) Configure the PiSCSI Web Interface stand-alone
 13) Enable or disable PiSCSI back-end authentication
 14) Enable or disable PiSCSI Web Interface authentication
EXPERIMENTAL FEATURES
 15) Share the images dir over AppleShare (requires Netatalk)
 16) Compile PiSCSI binaries
 17) Install Webmin to manage Netatalk and Samba
Enter your choice (0-17) or CTRL-C to exit:

It should say "Enter your choice (1-17)".

@uweseimet My github skills are a bit rusty, therefore I'll leave it up to the experts to create a PR with a fix for this.


Gitbook content page not redirecting to custom domain

Category
Bug
What is the current behavior?
Opening https://dhavalkapil.gitbooks.io/heap-exploitation/content/ does not redirect to my custom domain.
Note that https://www.gitbook.com/read/book/dhavalkapil/heap-exploitation does redirect as expected.
What is the desired behavior?
If someone has the earlier link of my gitbook (https://dhavalkapil.gitbooks.io/heap-exploitation/content/), I'll be having two separate copies of disqus comments(which are identified using URLs).

This is relevant for me as well - did you find a solution yet?


enable merge on ICEBERG via Athena

Background
See rationale below for why iceberg/open table. We can support it easily via existing destinations: Athena and Snowflake.

snowflake ICEBERG https://www.snowflake.com/blog/iceberg-tables-powering-open-standards-with-snowflake-innovations/ - still in private preview so we skip that
athena - supports ICEBERG and actually our destination is already using it for dlt tables

Why not spark:

too much work for this ticket.

Requirements

[x] add a new table hint table_format that will tell destination which table format to use

[x] for Athena: add global destination setting to set all tables to ICEBERG

[ ] enable merge support for iceberg tables on athena

Implementation Notes

Bulk loading into iceberg on athena can only be done from one table to another. So we need to retain our code for defining regular athena tables and additionally add iceberg tables to copy from the first tables.
Iceberg has nice "merge into ... on" sql directives that we can hopefully use for the merge write disposition
Do we need to run maintenance commands that clean up the underlying parquet files on iceberg or should we leave this to the user?

Rationale
Just few points on why iceberg is a game changer on data lakes:

it allows evolving the partitions
e.g. often data engineers design partitioning based on few use cases that they change over time, and Iceberg allows to just change those partition definitions, thanks to hidden partition (without rewriting the all dataset, that's the case for pure parquet)
it has build in procedures to optimize the dataset
are you aware of the small file problem in data-lakes? if so iceberg has a build in function to compact "objects" and improve performance and cost. Just a note on that: I wrote quite some compaction procedures on my life, and I believe that Iceberg make that way of compacting data standardised (same as DeltaLake)
using disposition like Merge > e.g. you can upsert by id
being able to delete data from a dataset - that's what I mentioned already, and that's why I choose Iceberg to ingest sensitive data from Kakfa for example.
this decision came after leading a GDPR project that allow to remove sensitive data on a data lake - again game changer that make data engineers life much easier.

last but not least: Iceberg is becoming one of the facto open table format in the data landscape:

Redshift now supports it (read only in preview - I net to test it)
Snowflake supports it in preview (read/write)
BiqQuery should support it (read only? - not sure)
Trino supports it (read/write)
The Athena supports it too - as it's an AWS fork of Trino (on engine v3)
Spark supports it - the most complete engine that support iceberg
Flink supports it - (read/write)
Dremio Support it - read for sure, not sure about writes
Clickhouse support it - only read
DuckDB - support  it - only read
Java client - read and write
python client - read only for now
rust client - coming soon
go client - coming soon

I'm interested to have merge support for the Athena destination.  For now I'll have to post process a merge within DBT or Airflow Athena operator.  So in essence will use dlt to build Athena staging tables (with replace write disposition).  I'd be happy to collaborate to help build this out.

@n0012 we had preliminary merge support working. @sh-rp was testing it. you can join our slack and ping me or Dave - maybe we can find out how you could help us...
what happens right now:

we create append job instead of merge job in athena.py
SQL engine on athena does not support temporary tables and a few other constructs we use for merge jobs so our default would need to be customized

(2) is the tricky part. we use just INSERT + DELETE to do merges but each destination has small differences in the syntax that needs to be handled

@rudolfix, what sort of timeline would you be looking at to add this merge support? I don't need a hard deadline or anything, just a vague idea if it's in the vicinity of weeks, 2-3 months or 6+ months

Adding here my use cases.

I ingest data from Google Big query (Google analytics dumps) in my final destination S3(parquet)/glue catalog(that I partitioned by event_date) I perform a delete/insert by event_date with a lookback window of 2 days. Doing so yesterday day is delete and insert again, in this case containing the entire events for yesterday. Said so I would like to perform a delete+insert by event_date. I pretty much have these setup in 80% of my case.
In order to do so I use aws-sdk-pandas with overwrite_partitions

import awswrangler as wr

source_df = ...# data from the last 7 days
wr.s3.to_parquet(
        df=source_df,
        path=f"s3://{DST_BUCKET}/{GLUE_TABLE_NAME}",
        dataset=True,
        schema_evolution=True,
        mode='overwrite_partitions',
        partition_cols=['my_date'],
        database='bronze',  # Athena/Glue database
        table=GLUE_TABLE_NAME  # Athena/Glue table
    )

worth to mention, I don't need to use iceberg here, because awswrangler overwrite the data for a partition performing a delete on s3 directly and just an append again. Using iceberg tables will be possible to do delete/insert

I merge my source_data by a specific key (e.g. id) in that sense I perform an upsert by id, keeping the latest data for an id based on my extracted batch of data.

Once merge disposition is implemented for athena iceberg table I could consider to use dlt instead - also a deal breaker for me is that I need also to have partition definition on my table to reduce data scans, but this is another issue


fix: Reset prompt groups query when changing filters

Change Type

[x] Bug fix (non-breaking change which fixes an issue)

Testing
Create at least 11 different prompts in the prompts library. Go to the next page. Use the keyword filter and then remove it. You should still be able to use the pagination feature.
Checklist

[x] My code adheres to this project's style guidelines
[x] I have performed a self-review of my own code
[x] My changes do not introduce new warnings
[x] Local unit tests pass with my changes

Issue #3290

this also fixes #3289

thanks!


[hw,dma,rtl] Clear STATUS when starting a new transfer

Previously, the FW needs to manually clear the done signal after a transfer, to allow a polling loop to get the fresh data. This PR changes the hardware to automatically clear the done bit on new transfer. This removes the requirement on the FW to clear the DONE bit after a transfer.
Furthermore, it clears also the error and sha2_digest_valid bit when starting a new transfer.
This PR is pulled over from https://github.com/lowRISC/opentitan-integrated/pull/660
/cc @alees24

Thanks for the review. I changed the SHA clearing mechanism to use the initial transfer bit for its clearing. I moved the clearing signal of SHA next to the clearing signal for the other registers to have them in the same place. I run all stress tests and they all pass:
### Test Results
|  Stage  |         Name         | Tests                |  Max Job Runtime  |  Simulated Time  |  Passing  |  Total  |  Pass Rate  |
|:-------:|:--------------------:|:---------------------|:-----------------:|:----------------:|:---------:|:-------:|:-----------:|
|   V2    | dma_handshake_stress | dma_handshake_stress |      30.517m      |    124.720ms     |     3     |    3    |  100.00 %   |
|   V2    |  dma_memory_stress   | dma_memory_stress    |      29.883m      |    159.712ms     |     3     |    3    |  100.00 %   |
|   V2    |                      | **TOTAL**            |                   |                  |     6     |    6    |  100.00 %   |
|         |    Unmapped tests    | dma_generic_stress   |      31.150m      |     38.041ms     |     5     |    5    |  100.00 %   |
|         |                      | dma_stress_all       |      30.150m      |    125.296ms     |     5     |    5    |  100.00 %   |
|         |                      | **TOTAL**            |                   |                  |    16     |   16    |  100.00 %   |


[bug] headings on full width image card aren't working

Describe the bug
We should be able to provide a heading below the full width image on rh-card, but it seems it doesn't work
Which Component?
rh-card
Steps to reproduce

Go to this page  and check the variant with full width image (last card)
Add any of the heading tags before
 and provide a slot="header"
Check the heading, it doesn't show up
Try providing h2 as part of the body
The styles for heading doesn't take any effect

The only way we could make this work was to wrap the whole body inside a div and applying styles to the div but thats not the recommended approach.
Expected behaviour
The heading is shown below the full width image card
Screenshots

Element HTML
<rh-card class="full">

Heading h2

Lorem ipsum dolor sit amet, consectetur adipiscing elit.
         Nullam eleifend elit sed est egestas, a sollicitudin mauris
         tincidunt. Pellentesque vel dapibus risus. Nullam aliquam
         felis orci, eget cursus mi lacinia quis. Vivamus at felis sem.

      <rh-cta priority="primary" slot="footer">
        Call to action
      </rh-cta>
</rh-card>

Operating System (OS)
macOS
Browser
Chrome, Safari, Firefox
More browser information
No response
Additional Info
No response

@diwanshi

Is this something like what you would expect?
& with the following html?
  <rh-card class="full">

Headline, sm

Lorem ipsum dolor sit amet, consectetur adipiscing elit.
       Nullam eleifend elit sed est egestas, a sollicitudin mauris
       tincidunt. Pellentesque vel dapibus risus. Nullam aliquam
       felis orci, eget cursus mi lacinia quis. Vivamus at felis sem.

    <rh-cta priority="primary" slot="footer">
      Call to action
    </rh-cta>
  </rh-card>

@brianferry yes thats exactly what we want

@diwanshi
I have updated the pull request to instead of setting up a new slot in the header it instead allows for multiple items to be slotted in the header.
The following html should work and is valid in the new update:
<rh-card class="full">

Heading h2

Lorem ipsum dolor sit amet, consectetur adipiscing elit.
         Nullam eleifend elit sed est egestas, a sollicitudin mauris
         tincidunt. Pellentesque vel dapibus risus. Nullam aliquam
         felis orci, eget cursus mi lacinia quis. Vivamus at felis sem.

      <rh-cta priority="primary" slot="footer">
        Call to action
      </rh-cta>
</rh-card>


color-hint => color_hint

In STAC we usually use snake_case for fields. Here, we did use color-hint instead of color_hint and I'd propose to change this so that tooling that relies on a certain set of characters don't get confused ;-)

Thanks, fixed in v1.1.0

Isn't this technically a breaking change?

Technically yes but at this pilot stage and limited if any implementation it felt odd to call this 2.0.0.
I'm fine with changing the version number to be semantically correct if you prefer.

Doesn't really affect me personally so I'm 🤷🏽, I was just curious if it had been considered. IMO if this is the policy, it'd be good to (a) document as such (e.g. on the readme) and (b) set some sort timeline for when we do hope to stabilize and commit to stricter versioning.

For sure, the README says the extension is a "pilot", though I don't how well yet the concept of "maturity" overlaps with semver. I'm not sure if anyone has implemented this extension in the wild - even us (Maxar) have not put it into production yet.
So I think the game plan is to see this actually get used to the point we can agree it's stable, then take it out of "pilot" in say 6 months or so.

For sure, the README says the extension is a "pilot", though I don't how well yet the concept of "maturity" overlaps with semver.

Yeah, I don't know how well its defined myself. IMO pilot seems to line up better with a v0.X.Y series but that ship has sailed :-).

I'm not sure if anyone has implemented this extension in the wild - even us (Maxar) have not put it into production yet.

We're using it on the PC.

So I think the game plan is to see this actually get used to the point we can agree it's stable, then take it out of "pilot" in say 6 months or so.

👍🏽


[bug] A bug that moves to the App Store when launched on an iPhone SE

Link to minimal reproducible example
https://web3modal.com/
Summary
I tried to connect metaMask from the page https://web3modal.com/ to test it, but even though the app is installed, it is moved to the metaMask app installation link, and there is no response after this. It happens on my iPhone is Iphone Se and the ios version is 16.4.1(a).
List of related npm package versions
"@wagmi/core": "^0.9.5",
"@web3modal/ethereum": "^2.1.1",
"@web3modal/html": "^2.1.1",
"web3": "^1.8.0",

This issue is on MetaMask / iOS side. They are currently using so called universal-links, while majority of other wallets use deep-links. Latter is preferable and MetaMask's next release should start supporting them, thus fixing the issue.


Z~!NEW-VIDEOs]* Oviya Helen Viral Leaked  Video 2024 LINK Viral on Social Media X Today (Trending)

33 sec ago, Oviya Helen, ZFULL Oviya Helen Leaks Video On Social Media X Twitter[2B75K]

Oviya Helen latest video has taken social media by storm, captivating viewers and sparking widespread discussions. This young digital creator has garnered significant attention for her engaging content, particularly with her recent Oviya Helen-themed video that has gone viral. Fans are eagerly searching for where to watch the full video, turning to platforms like Instagram, X (formerly Twitter), and Discord for access.
As the buzz around Oviya Helen grows, fans are actively sharing their reactions online. The viral success of her video highlights the powerful role of social media in propelling creators into the spotlight. With her unique style and creativity, Oviya Helen has quickly become a notable figure in the digital landscape, leaving many eager to learn about her future projects and content.
...
....
....
...
...
...
....
....
...
...
....
...
...
....
....
...
...
....
...
...
....
....
...
...
....
...
...
....
....
...
...
....
...
...
....
....
...
...
....
...
....
...
[

bcvbcv

bcvbcv


Feat cleanup reference dir

[ ] Closes #xxxx
[ ] Tests added
[ ] Performance benchmarks added
[ ] Performance regression benchmarks run
[ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
[ ] New functions/methods are listed in api.rst or it's sub rsts?

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (cdf1a91) 78.93% compared to head (a503d65) 78.85%.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #252      +/-   ##
===========================================
- Coverage    78.93%   78.85%   -0.08%
===========================================
  Files           51       51
  Lines         6694     6693       -1
===========================================
- Hits          5284     5278       -6
- Misses        1410     1415       +5

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Fix release pipeline

Codecov Report

Merging #190 (81d67a8) into main (9a78110) will decrease coverage by 0.11%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #190      +/-   ##
==========================================
- Coverage   36.02%   35.90%   -0.12%
==========================================
  Files          47       47
  Lines        3936     3938       +2
==========================================
- Hits         1418     1414       -4
- Misses       2392     2396       +4
- Partials      126      128       +2

Flag
Coverage Δ

unittests
35.90% <100.00%> (-0.12%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

pkg/runtime/golang.go
74.62% <100.00%> (+0.38%)
:arrow_up:

pkg/ghissue/buglink.go
90.32% <0.00%> (-9.68%)
:arrow_down:

pkg/utils/paths.go
5.55% <0.00%> (-5.56%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9a78110...81d67a8. Read the comment docs.


Traefik pods not able to renew certs with LetsEncrypt

Do you want to request a feature or report a bug?

Bug

What did you do?
We are using Traefik as our ingress, configured with Letsencrypt for certificate renewal.From time to time some of traefik pods aren't able to renew certificate(which causes some of our services to have Expired Certificates.

What did you expect to see?
After the certificate expired Traefik pods properly renew the certificates with LetsEncrypt.
What did you see instead?
Some of the pods aren't able to renew certificates.
Output of traefik version: (What version of Traefik are you using?)

Traefik v1.7.20

What is your environment & configuration (arguments, toml, provider, platform, ...)?
on prem hosted k8s cluster v1.16.9, AWS cloud, we are using Traefik's helm chart

values.yaml:
dashboard:
  enabled: true
  ingress:
    annotations:
      kubernetes.io/ingress.class: traefik-internal
  statistics:
    recentErrors: 100
rbac:
  enabled: true
replicas: 3
ssl:
  enabled: true
  enforced: true
  tlsMinVersion: VersionTLS12
proxyProtocol:
  enabled: true
  trustedIPs:
    - <IP_ADDRESS>/0
    - ::/0
forwardedHeaders:
  enabled: true
  trustedIPs:
    - <IP_ADDRESS>/0
    - ::/0
maxIdleConnsPerHost: -1
deployment:
  podAnnotations:
    # safe to evict because the only thing keeping the cluster auto-scaler from automatically evicting is local
    # storage for acme certs which is not used (Consul is used to persist the certs instead)
    "cluster-autoscaler.kubernetes.io/safe-to-evict": true
deploymentStrategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 50%
    maxSurge: 100%
kubernetes:
  ingressEndpoint:
    useDefaultPublishedService: true
acme:
  domains:
    enabled: true
  persistence:
    enabled: false
  email: email
  delayBeforeCheck: 240
  enabled: true
  challengeType: "dns-01"
  onHostRule: true
  staging: false
kvprovider:
  storeAcme: true
  consul:
    endpoint: "consul-server:8500"
    watch: true
resources:
  limits:
    cpu: 1
    memory: 2Gi
  requests:
    cpu: .5
    memory: 500Mi
autoscaling:
  minReplicas: 3
  maxReplicas: 15
  metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageUtilization: 60
  - type: Resource
    resource:
      name: memory
      targetAverageUtilization: 60
affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 85
        podAffinityTerm:
          labelSelector:
            matchLabels:
              app: traefik
          topologyKey: kubernetes.io/hostname
podDisruptionBudget:
  minAvailable: 2
metrics:
  prometheus:
    enabled: true
  serviceMonitor:
    enabled: true

If applicable, please paste the log output in DEBUG level (--log.level=DEBUG switch)
{"level":"info","msg":"Testing certificate renew...","time":"2020-09-16T02:06:27Z"}
{"level":"info","msg":"Renewing certificate from LE : {Main:*.our.domain SANs:[]}","time":"2020-09-16T02:06:27Z"}
{"level":"warning","msg":"A new release has been found: 2.2.11. Please consider updating.","time":"2020-09-16T02:06:27Z"}
{"level":"error","msg":"Error renewing certificate from LE: acme: Error -\u003e One or more domains had a problem:\n[*.*.our.domain] [*.*.our.domain] acme: error presenting token: route53: failed to determine hosted zone ID: zone our.domain. not found for domain _acme-challenge*.our.domain.\n","time":"2020-09-16T02:06:28Z"}

Hello @vladh91,
thanks for your feedback. According to the issue, it looks like you're running multiple replicas with Traefik v2 to solve ACME challenges. However, this is not supported as mentioned here https://github.com/traefik/traefik/issues/5426#issuecomment-539715461
Therefore, I'll close that issue.
Thanks.

Hey @SantoDE thanks for your response. Sorry but I probably didn't get what you mean by saying Traefik v2? Our traefik version is v1.7.20 if you mean that

Version:      v1.7.20
Codename:     maroilles
Go version:   go1.12.14
Built:        2019-12-10_05:02:10PM
OS/Arch:      linux/amd64```
This is the output of the docker image used in helm chart for traefik.

Ah, sorry! I overlooked that, but it's still the same. If you're not using a k/v store for leader election (which is a bit buggy), you'll have the exact same behavior.


Reproducing MolBERT results on QSAR tasks

Hi MolBERT team,
First of all thank you for releasing this repository and providing the scripts to reproduce your paper,
it is deeply appreciated!
I have an issue reproducing the QSAR results from Table 3 in the paper for MolBERT and MolBERT (finetune),
as detailed below:

I can exactly reproduce the Table 3 entries for RDKit and ECFC4 using scripts/run_qsar_test_molbert.py so that is reassuring
The MolBERT featurizer, however, yields lower AUROCs i.e. for BACE I get 0.835 vs 0.849 from the paper and for BBBP I get versus 0.744 vs 0.750 in the paper.
Similarly for MolBERT (finetune) using scripts/run_finetuning.py for BBBP I get 0.751 vs 0.762 reported in the paper

The pre-trained model I am using is the one provided in the README i.e. https://ndownloader.figshare.com/files/25611290
Could it somehow be that I am using the wrong weights, or the wrong weights were uploaded to figshare? This would effect the results in both 2. & 3. above so would make sense.
Finally, the parameters I have been using for the fine-tuning are the following:

freeze_level = 0 taken from the answer in #3
learning_rate = 3e-5 taken from the paper although I could only find the value for pre-training and not fine-tuning
batch_size=16

All other arguments are left to the defaults provided in the code. Should the above arguments reproduce results similar to the paper?
Thanks in advance!
Tom

@TWRogers ,
I also get poor predictions compared to the MolBert paper. Fine-tuned MolBert sometimes gets worse performance than MolBert without any fine-tuning. Maybe we need to get the details setting for finetuning MolBert.
Best regards,
Po-Yu

I am using the same setting as @TWRogers using, and also cannot get the reported performance.
Here are some results for the regression tasks from what I rerun the experiments,

Lipop: 0.710, 0.607, and 0.643 for three seeds, but  $0.561 \pm 0.03$ was reported in the paper;
ESOL: 0.551, 0.542, and 0.590 for three seeds, but $0.531 \pm 0.04$ was reported in the paper;
FreeSolv: 1.286, 0.940, and 1.680 for three seeds, but $0.948 \pm 0.33$ was reported in the paper.


Bump to version 2.1.1

Revert using Sentry option of rails_report_rescued_exceptions (https://github.com/alphagov/govuk_app_config/pull/140)

@kevindew does this mean we're done with https://trello.com/c/HeLolmLj/1668-investigate-lack-of-sentry-errors-despite-showing-an-error-page?

@benthorner Hopefully - I'll do a test deploy on integration to verify.


YES tool — Display alert in no-js environment

To help the user make the choice to either use a different browser or print the page, display an alert in a no-js env.
Additions

Alert message when javascript is not detected / unsupported

Changes

Changes print behavior. Now, JS will run to hide all certain elements when the print your plan button is clicked. Otherwise, all elements not tagged to hide on print will print in a no-js env.

Testing

When printing without JS, images, the red danger alert, and buttons should not appear in the print dialog box
When printing from the print your plan button, everything but the plan review section should be hidden.

Checklist

[ ] PR has an informative and human-readable title
[ ] Changes are limited to a single goal (no scope creep)
[ ] Code can be automatically merged (no conflicts)
[ ] Code follows the standards laid out in the CFPB development guidelines
[ ] Passes all existing automated tests
[ ] Any change in functionality is tested
[ ] New functions are documented (with a description, list of inputs, and expected output)
[ ] Placeholder code is flagged / future todos are captured in comments
[ ] Visually tested in supported browsers and devices (see checklist below :point_down:)
[ ] Project documentation has been updated
[ ] Reviewers requested with the Reviewers tool :arrow_right:

Testing checklist
Browsers

[ ] Chrome on desktop
[ ] Firefox
[ ] Safari on macOS
[ ] Edge
[ ] Internet Explorer 9, 10, and 11
[ ] Safari on iOS
[ ] Chrome on Android

Accessibility

[ ] Keyboard friendly
[ ] Screen reader friendly

Other

[ ] Is useable without CSS
[ ] Is useable without JS
[ ] Flexible from small to large screens
[ ] No linting errors or warnings
[ ] JavaScript tests are passing

Hmm, I still see alerts in the no-js print state, and this expandable button:

@anselmbradford Interesting, can you tell me more about what steps you are taking to see the alert at the top of the page? When I hit command+P, the print dialog window does not show the alert in the preview.

Oh right, the no js alert (notification) is not showing up, but I meant the other alerts in the doc, like these:

I wasn't sure if you wanted those or not.

Ah! Sorry, I definitely wasn't clear in my description. It probably makes sense to remove those as well, I'll double check with design. Thanks!

@anselmbradford I'm going to hide them, they don't make sense in a print environment. Thanks again for your feedback!


Script to create an x509 user config

This script also updates a cluster role mapping to give the user correct permissions

I was trying to test this, the first thing I notice is the the script doesn't have any output, so as an executer I don't know what has succeeded/failed.
So when I tried to debug by hand, and list all the CSRs, I don't find any, which tells me that something went wrong while executing the script but I don't know what is.
Maybe you could also include stderr when executing your commands, and check it the run resulted in any errors, and if that's the case, log the error & exit early ?

As I read through the script, the CSR doesn't exist since it's being deleted by the script, so that's normal.
Currently, it's erroring at f.write(ROLE_BINDING_PATCH_TEMPLATE.format(user=user)) with a KeyError, and I think that's because you've already formatted the string at the beginning.


[FLINK-18802][formats] Package uber jar including all dependencies for flink-avro

What is the purpose of the change
This pull request changes the pom.xml of flink-avro module in order to create a uber jar in packaging phase, including dependencies such as avro, jackson-core-asl, jackson-mapper-asl and joda-time.
Brief change log

Add another execution in maven-assembly-plugin to create a jar-with-dependencies
Remove the "provided" scope and "optional" tag of joda-time dependency to include it in the uber jar

Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable)

LGTM. We probably also want to have uber jars for other formats as well.
@wuchong Do you want to take a look?

Looks like CI failed. Fixed in 766339d

Thanks a lot @twalthr ~ What about upgrading Avro version to 1.9.0 so that we can get rid of joda-time, and packaging it into sql-jar as the default Avro version?

Hey @PatrickRen,
I think it would be fine upgrading the avro version for the sql jar. We could even think of upgrading it straight to the 1.10 version. Again I very much agree with @twalthr that we can do that for the sql jar only.
In case of the {{flink-avro}} module the situation is not that easy. Users might've generated their classes against specific version. Moreover they might want to use the version of avro that comes with hadoop they're using. Therefore they should have an easy way to replace the avro version.

I am closing this PR because of inactivity and because it is subsumed by https://github.com/apache/flink/pull/13373 . Feel free to reopen if you think otherwise.


shared_audits: add freetube to GITHUB_PRERELEASE_ALLOWLIST

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[ ] Have you written new tests for your changes? Here's an example.
[ ] Have you successfully run brew style with your changes locally?
[ ] Have you successfully run brew tests with your changes locally?
[ ] Have you successfully run brew man locally and committed any changes?

Required for https://github.com/Homebrew/homebrew-cask/pull/90294 to pass audit.

Thank you @dawidd6.


Performance issues with ValidationMode.Local

The latest change introduced in v2.15.0 has a huge performance issue when the validation mode is set to local. The change was to remove the time check in the DiscoveryDocumentIssuerSecurityTokenProvider class when the security tokens should be refreshed.
Now the configurationManager.GetConfigurationAsync() is  checking the AutomaticRefreshInterval and only returning a new OpenIdConnectConfiguration if it has been refreshed. However the RetrieveMetadata method is called before getting the Issuer, Audience and SecurityTokens. This is  blocking other incoming http requests as the ReaderWriterLockSlim is locking for each RetrieveMetadata call.
I have created a new pull request that basically reverts some of the code so that the RetrieveMetadata  method will only attempt to recreate the Issuer and SecurityTokens if the AutomaticRefreshInterval has lapsed.
For now, I have downgraded to v2.14.0.
Pull Request: https://github.com/IdentityServer/IdentityServer3.AccessTokenValidation/pull/155

This has also affected a production deployment for one of our APIs, causing timeouts when the API is under heavy load.

We have had intermittent timeouts and slow performance in our API for a while and running the profiler in Azure I found the following for a slow API call:
IdentityServerBearerTokenValidationMiddleware.Invoke | 4100.19 ms |
OTHER <<AsyncTaskMethodBuilder.Start>> | 4100.19 ms |
IdentityServerBearerTokenValidationMiddleware+<Invoke>d__8.MoveNext | 4100.19 ms |
OTHER <<AuthenticationMiddleware`1.Invoke>> | 4096.43 ms |
DiscoveryDocumentIssuerSecurityTokenProvider.get_SecurityTokens | 2870.17 ms |
DiscoveryDocumentIssuerSecurityTokenProvider.RetrieveMetadata | 2870.17 ms |
OTHER <<ReaderWriterLockSlim.TryEnterWriteLock>> | 2338.90 ms |
BLOCKED_TIME | 2338.59 ms |
CPU_TIME | 0.31 ms |

After finding this issue I also downgraded to 2.14 and it seems like it solved the timeout problems we had.

We did an update that's supposed to deal with the deadlock in Microsoft's config loader. Please test again the myget version and let us know.  https://www.myget.org/gallery/identity
Thanks

We believe we're seeing this as well....  did anyone test this latest 2.15.1?  @brockallen  I noticed this wasn't ever published to NuGet...
We went live with IdSrvr at the very end of June... and our first deployment was on 2.15.0.
Would you recommend us downgrading to 2.14 or utilizing 2.15.1?

Would a metric like this, where the AuthenticateRequest is taking 3+ seconds, be evident that the performance issues still exist in 2.15.1?

Thanks in advance!!

@kedwardsRenWeb That would be a good start!  We have not tested 2.15.1 yet as 2.14 has worked without issues and our next planned upgrade will move to .NET Core.

Well - if no-one gives the bug fix a try, we'll never find out.