Python DeltaTable does not support writes in multiple threads

Environment
Delta-rs version: 0.20.1
Binding: python
Environment:

Cloud provider:
OS: macOS
Other:

Bug
What happened:
Using the same DeltaTable across multiple threads in write_deltalake causes the error
RuntimeError: Already borrowed

This is not an issue if we use the table URI directly.
What you expected to happen:
Writes should work in a multithreaded environment.
How to reproduce it:
from concurrent.futures import ThreadPoolExecutor
from deltalake import DeltaTable, write_deltalake
import polars as pl

path = 'some local path'
table = pl.DataFrame({'a': [1,2,3]}).to_arrow()
write_deltalake(path, table)

dt = DeltaTable(path)

with ThreadPoolExecutor() as exe:
    list(exe.map(lambda _: write_deltalake(dt, table, mode='append'), range(2)))

produces
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/Users/peter.ke/.pyenv/versions/3.10.13/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator
    yield _result_or_cancel(fs.pop())
  File "/Users/peter.ke/.pyenv/versions/3.10.13/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel
    return fut.result(timeout)
  File "/Users/peter.ke/.pyenv/versions/3.10.13/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/Users/peter.ke/.pyenv/versions/3.10.13/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/Users/peter.ke/.pyenv/versions/3.10.13/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "<stdin>", line 2, in <lambda>
  File "/Users/peter.ke/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deltalake/writer.py", line 302, in write_deltalake
    table.update_incremental()
  File "/Users/peter.ke/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deltalake/table.py", line 1258, in update_incremental
    self._table.update_incremental()
RuntimeError: Already borrowed

More details:
Cause of error:

write_deltalake calls write_to_deltalake from rust to write, which borrows the RawDeltaTable immutably while releasing the GIL. The immutable borrow means theoretically writes in multiple threads should be fine
However, write_deltalake also calls table.update_incremental(), which borrows the RawDeltaTable mutably
Since GIL was released in write_to_deltalake, another thread running write_deltalake will try to mutably borrow the previously immutably borrowed RawDeltaTable and fails

From my brief investigation, I see two potential solutions:

The Rust API exposed to python could benefit from being refactored to be more immutable so it fits better in a multi-threaded environment. e.g. on the rust side, update_incremental can return a new DeltaTable rather than mutating the existing one- similarly for other internals like DeltaTableState or Snapshot.
This should be similar performance/memory wise, since cloning a Snapshot seems relatively cheap given that the bulk files uses arrow. However, this looks like a large-ish refactor because a lot of the internals expose mutable APIs.

Remove update_incremental from write_deltalake (or add a flag to disable it). We'd just need to ensure write_to_deltalake returns an updated RawDeltaTable so table._table can kept updated.

Actually the easiest solution might be to just clone the RawDeltaTable before we pass it into write_to_deltalake so it is not borrowed.

I believe you could get around it with multiprocessing (using spawn).

I believe you could get around it with multiprocessing (using spawn).

True, that does work. DeltaTable is not picklable though so it's a bit annoying. Since write_deltalake releases the GIL, I believe like the intention is to support multithreaded writes.

I think it's best to make the update_incremental call optional, seems unnecessary to always incur this overhead in the circumstances when the caller already know it has an up-to-date table instance.


error when using non-default dataset/stations dim names from cli

Fix error when providing non-default options via cli config file.
Current extract only using default config values

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Merged. Thanks @colonesej !


Not asking for Username & Password

Hi,
i just installed the RPi_Cam interface and it is functioning except it is not asking for the username and password when I access the HTML folder. It shows the full options, which I do not want people to access unless they have an account or know the default username and password I configured.
Can anyone help me get the username and password working?
I did a standard installation according to the instructions located here: http://elinux.org/RPi-Cam-Web-Interface#Simple_live-preview_only_page
IE basic Installation. The first time I told it no user / password. Then I re-ran the installation script and added a username and password, but when I access the site it doesn't ask for the username and password I entered.
Running Noobs, Latest Jessie version latest updates. I have an existing website hosted, running WordPress, that is why I installed it in the sub directory html. I didn't change the port.
Thanks!

Currently the username / password should be restricting access to the site. It doesn't control whether they can see the options or not. E.g. if you have set up username/password and try to access http://ipaddress  (defaults ti index.php) then it should ask for the credentials.
I just tried it with min.php and it did ask for credentials as expected.
Note that once you have logged in then a browser will maintain those credentials even if you close the browser tab. You need to close the browser to get the request again.
There is a request to separate out logins so that simple or settings views can be controlled via the login. I am starting to look at that but it is a significant change as currently the security is just a pure Apache function.
If the base security is not working for you then let me know and give as much detail as you can. Is the Wordpress running under Apache?

It has never asked me for a password, even after I changed the setting and added the username and password.
Yes, Wordpress is running under Apache2 and MySQL./ I am just learning this stuff so if you need any information please let me know what and I can get it for you.
I followed this guide to get Wordpress running under Apache2 on the Raspberry Pi 3.
https://www.raspberrypi.org/learning/lamp-web-server-with-wordpress/worksheet/

It is probably something to do with the Wordpress install confusing the installation of the RPi Cam.
Can you zip and post the /etc/apache2/sites-available/raspicam.conf file so I can check it out?

Sure, here it is.
raspi_conf.zip

The conf file looks OK.
I just retested re-installing with passwords on and off and it works OK for me.
Further thoughts.
Changes to config only take effect if apache2 service is restarted. install script does this but it might be worth doing again just to make sure (sudo service apache2 restart)
There should be a .htaccess file in the /var/www/html folder that points to the user / password security info. It should just have the following content. Note that . files are normally hidden but you can cat /var/www/html/.htaccess
AuthName "RPi Cam Web Interface Restricted Area"
AuthType Basic
AuthUserFile /usr/local/.htpasswd
Require valid-user
It s also possible that another apache2 conf file is also defining the html subfolder and this may be taking precedence and preventing the raspicam.conf file taking effect. Normally the installler disables the 000-default.conf from doing this but if there are others then that may be the cause. See what other conf files are  in the /etc/apache2/sites-available folder

I have already restarted Apache2 to see if that was the issue, no change. I also restarted my browser and flushed the cache to make sure that wasn't the issue.
The .htaccess file in that dir reads:
~:~ $ sudo cat /var/www/html/.htaccess
BEGIN WordPress

RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]

Here are the other files located in the /etc/apache2/sites-available folder:
./default-ssl.conf
./raspicam.conf
./000-default.conf

Actually I just noticed. When I login to my webpage hosted on the same RBPI it no longer works with WOrdpress after installing the RPi Cam Preview. It just shows a directory and the HTML folder as a list.

Do you know how to revert it to get the default Website back up and running and then just point to the min.php file? I don't mind the min.php view to not have a login, I just didn't want people logging into the site and navigating to the config section. But I would like to keep the Wordpress site up and running, which it no longer is after installing this.

Yeah I am unfortunately not an expert or very knowledgeable on HTML, but it looks like WP had also installed itself in /var/www/html and installing the RPi Cam Preview basically used the same file structure, so overwrote some of the WordPress files.
It looks like WordPress had it setup so when you typed the domainname.com it would redirect the root to /www/html/index.php which is the same that Rpi Cam Preview uses. Not sure how I will be able to get them back as independent sites and get the permissions working. I might have to trash the WordPress one as I don't see many options for restoring it and still use the cam preview.
Might be something to consider for future installation scripts, to check if WordPress is installed, though not many people run WordPress on a Pi, though I thought it was pretty nifty, not terribly slow, especially with the pi overclocked.

Thanks for the update.  Sounds like the fastest way to recover may be to reinstall Wordpress say in its html and then install the rpi into a different subfolder like camera.
The installer could do a bit more checking but it also has to allow overwriting itself so it would need to check for some specific file not installed by itself.

Hi
I am having the same problem with the login. I have never logged in and do not have Wordpress installed. Im working with a Pi0 and have the updated install of Jessie. No errors that I have found just no password.

I forgot my pass to the web interface (classic wright?:)).
How can I reset it? I can access the pi zero via ssh , or physically.

You could just run ./install.sh again and put in new password.
Alternatively you could just do the bit that sets up a password manually.
in ssh do
sudo htpasswd -b -B /usr/local/.htpasswd $user $webpasswd
where you substitute the user and password you want.


docs: Update christianmendoza.me in the showcase

This PR updates the thumbnail and adds the source link for my site's entry in the showcase.

Hello @christianmendoza,
I merged your update as 490601e.


[Bug]: Clarify if worker_sandboxing is supported

What happened?
The /README says
Please note that rules_ts does not work with --worker_sandboxing.
If that's true, then it should probably link to a bug about why not.
Version
HEAD
How to reproduce
No response
Any other information?
No response
Fund our work

[ ] Sponsor our open source work by donating a bug bounty

See https://github.com/aspect-build/rules_ts/issues/127#issuecomment-1312041592


One slow (disconnected) peer will slow down everyone else

Description
I was investigating broadcasting code flow (for the source of the problem of not delivered weak coins) and found that if a single peer is too slow and queue fills up it will slow down all delivery to other peers.
In the gossip module we will iterate over all peers and block until the message is added to the queue of each peer. The current queue size is 250 messages. If single peer got disconnected but the socket is not yet terminated it will block for response-timeout (which is 60s by default and in majority of the configurations, which is also should be lower) and after that fail with deadline exceeded error.
Steps To Reproduce
It can happen in a way that i described above, but i think it can be reproduced in a simpler way like: create two nodes, register gossip protocol, block in the handler instead of processing messages. Eventually, all queues must fill up and the peer on other side will be blocked on a send.
Solution
Setup a buffer in bytes. For example 2mb for each outgoing queue. If this queue fills up disconnect a peer preemptively.
We have a configuration for max inbound and max outbound peers. By having these 2 (3) knobs we can control precisely how much memory p2p stack will be consuming in total.
Expected Behavior
Single slow peer can't block everyone else.

cc @antonlerner @y0sher

Actually, this is not the reason why weak coin messages are not delivered on time. Otherwise, i will see this warning "connection: outbound send queue backlog".

We have a series of old PRs that will likely address this issue:

https://github.com/spacemeshos/go-spacemesh/pull/2435
https://github.com/spacemeshos/go-spacemesh/pull/2450
https://github.com/spacemeshos/go-spacemesh/pull/2463

Will get these merged ASAP

This was addressed.


Update vulkan dependencies

This updates all Vulkan dependencies (headers, validation layers, loader) to the same version which is <IP_ADDRESS>. An update was inevitable because the validation layer package is not available for the old version of Vulkan headers that we used to use.
The given version is the latest version available in Conan. It would also be possible to downgrade to some earlier version - maybe Vulkan 1.2, but without having any requirements I don't think that makes sense. Most Linux distributions come with some 1.3 version of Vulkan, so we need to support 1.3 anyway.
So there are the following things happening:

Update Vulkan Header to <IP_ADDRESS>
Require the Vulkan Validation Layers Conan package (same version) and remove the copy of the header from third_party/
Adjustment of the version of Volk (the loader)
Remove usage of deprecated macros from the vulkan tutorial code. (I made some tests. The code seems still to run - no crash.)

4. Remove usage of deprecated macros from the vulkan tutorial code. (I made some tests. The code seems still to run - no crash.)

I think that the point of the tutorial was one test in LinuxTracingIntegrationTest. Could you run it (needs sudo)?

Remove usage of deprecated macros from the vulkan tutorial code. (I made some tests. The code seems still to run - no crash.)

I think that the point of the tutorial was one test in LinuxTracingIntegrationTest. Could you run it (needs sudo)?

I tried but it tells me that it requires a Stadia machine.
I disable the check for Stadia and ran it anyway.  All what I can see is expect - like missing amdgpu tracepoints, etc. Here is the log:
Note: Google Test filter = LinuxTracingIntegrationTest.GpuJobs                                                                                [159/4703]
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from LinuxTracingIntegrationTest
[ RUN      ] LinuxTracingIntegrationTest.GpuJobs
[2022-10-28T11:09:57.093814] [LinuxTracingIntegrationTests/IntegrationTestUtils.cpp:31] Error: Stadia instance required for this test (but kernel release is "5.18.16-1rodete1-amd64")
[2022-10-28T11:09:57.094171] [          LinuxTracing/TracerImpl.cpp:89] Error: Invalid sample stack dump size: 0; reassigning to default: 256
[2022-10-28T11:09:57.094270] [LinuxTracingIntegrationTests/IntegrationTestPuppet.cpp:177] Puppet started
[2022-10-28T11:09:57.095193] [         LinuxTracing/TracerImpl.cpp:752] Error: Could not read cpuset
[2022-10-28T11:09:57.815458] [  LinuxTracing/LinuxTracingUtils.cpp:179] Error: Reading tracepoint id of amdgpu:amdgpu_cs_ioctl: Unable to open file "/sys/kernel/debug/tracing/events/amdgpu/amdgpu_cs_ioctl/id": No such file or directory
[2022-10-28T11:09:57.815581] [         LinuxTracing/TracerImpl.cpp:530] Error: Opening amdgpu:amdgpu_cs_ioctl tracepoint for cpu 0
[2022-10-28T11:09:57.815588] [         LinuxTracing/TracerImpl.cpp:821] There were errors opening GPU tracepoint events
[2022-10-28T11:09:58.171692] [LinuxTracingIntegrationTests/IntegrationTestPuppet.cpp:185] Puppet received command: VulkanTutorial
[2022-10-28T11:09:58.171761] [VulkanTutorial/OffscreenRenderingVulkanTutorial.cpp:33] InitVulkan
[2022-10-28T11:09:58.222994] [        LinuxTracing/TracerImpl.cpp:1494] PERF_RECORD_THROTTLE in ring buffer 'sampling_18' at timestamp<PHONE_NUMBER>852537
[2022-10-28T11:09:58.228624] [        LinuxTracing/TracerImpl.cpp:1494] PERF_RECORD_THROTTLE in ring buffer 'sampling_54' at timestamp<PHONE_NUMBER>368789
[2022-10-28T11:09:58.228654] [        LinuxTracing/TracerImpl.cpp:1494] PERF_RECORD_THROTTLE in ring buffer 'sampling_61' at timestamp<PHONE_NUMBER>573201
[2022-10-28T11:09:58.228749] [        LinuxTracing/TracerImpl.cpp:1494] PERF_RECORD_THROTTLE in ring buffer 'sampling_87' at timestamp<PHONE_NUMBER>299839
[2022-10-28T11:09:58.229656] [        LinuxTracing/TracerImpl.cpp:1498] PERF_RECORD_UNTHROTTLE in ring buffer 'sampling_61' at timestamp<PHONE_NUMBER>719251
[2022-10-28T11:09:58.242129] [        LinuxTracing/TracerImpl.cpp:1498] PERF_RECORD_UNTHROTTLE in ring buffer 'sampling_18' at timestamp<PHONE_NUMBER>674872
[2022-10-28T11:09:58.269625] [        LinuxTracing/TracerImpl.cpp:1498] PERF_RECORD_UNTHROTTLE in ring buffer 'sampling_87' at timestamp<PHONE_NUMBER>677123
[2022-10-28T11:09:58.457659] [        LinuxTracing/TracerImpl.cpp:1498] PERF_RECORD_UNTHROTTLE in ring buffer 'sampling_54' at timestamp<PHONE_NUMBER>716722
[2022-10-28T11:09:58.605163] [        LinuxTracing/TracerImpl.cpp:1494] PERF_RECORD_THROTTLE in ring buffer 'sampling_23' at timestamp<PHONE_NUMBER>063809
[2022-10-28T11:09:58.664845] [VulkanTutorial/OffscreenRenderingVulkanTutorial.cpp:238] image_format_=37
[2022-10-28T11:09:58.665833] [VulkanTutorial/OffscreenRenderingVulkanTutorial.cpp:256] memory_type_index=0
[2022-10-28T11:09:58.675832] [VulkanTutorial/OffscreenRenderingVulkanTutorial.cpp:59] MainLoop
[2022-10-28T11:09:58.971092] [        LinuxTracing/TracerImpl.cpp:1498] PERF_RECORD_UNTHROTTLE in ring buffer 'sampling_23' at timestamp<PHONE_NUMBER>738628
Trying if anonymous executable map at 0x7f9b0063e000-0x7f9b0063f000 belongs to "/etc/ld.so.cache"
Magic MS-DOS 2.0 value not found. Value read: 6c67
Magic MS-DOS 2.0 value not found. Value read: 6c67
No, anonymous executable map at 0x7f9b0063e000-0x7f9b0063f000 does NOT belong to "/etc/ld.so.cache": this is not a PE
Trying if anonymous executable map at 0x7f9b0063b000-0x7f9b0063c000 belongs to "/etc/ld.so.cache"
Magic MS-DOS 2.0 value not found. Value read: 6c67
Magic MS-DOS 2.0 value not found. Value read: 6c67
No, anonymous executable map at 0x7f9b0063b000-0x7f9b0063c000 does NOT belong to "/etc/ld.so.cache": this is not a PE
Trying if anonymous executable map at 0x7f9a65dfd000-0x7f9a667fd000 belongs to "/usr/local/google/home/hebecker/projects/orbit/build_clang11_release/bin/LinuxTracingIntegrationTests"
Magic MS-DOS 2.0 value not found. Value read: 457f
Magic MS-DOS 2.0 value not found. Value read: 457f
No, anonymous executable map at 0x7f9a65dfd000-0x7f9a667fd000 does NOT belong to "/usr/local/google/home/hebecker/projects/orbit/build_clang11_release/bin/LinuxTracingIntegrationTests": this is not a PE
Trying if anonymous executable map at 0x7f9b00639000-0x7f9b0063a000 belongs to "/etc/ld.so.cache"
Magic MS-DOS 2.0 value not found. Value read: 6c67
Magic MS-DOS 2.0 value not found. Value read: 6c67
No, anonymous executable map at 0x7f9b00639000-0x7f9b0063a000 does NOT belong to "/etc/ld.so.cache": this is not a PE
[2022-10-28T11:09:59.723516] [VulkanTutorial/OffscreenRenderingVulkanTutorial.cpp:76] CleanUp
[2022-10-28T11:09:59.892191] [         LinuxTracing/TracerImpl.cpp:923] Closing 576 file descriptors started
[2022-10-28T11:10:00.101677] [         LinuxTracing/TracerImpl.cpp:923] Closing 576 file descriptors took 209.478 ms
/usr/local/google/home/hebecker/projects/orbit/src/LinuxTracingIntegrationTests/LinuxTracingIntegrationTest.cpp:469: Failure
Value of: event.has_lost_perf_records_event()
  Actual: true
Expected: false

Remove usage of deprecated macros from the vulkan tutorial code. (I made some tests. The code seems still to run - no crash.)

I think that the point of the tutorial was one test in LinuxTracingIntegrationTest. Could you run it (needs sudo)?

I tried but it tells me that it requires a Stadia machine.

Yeah, I meant on a Stadia machine (after all, that's why that verification was in place). But I realize that that would mean that you are still able to build for it and I guess the answer is not anymore...?
In a way we could discuss whether it even makes sense to keep this test around then. Maybe it's a bit of a larger conversation. But I would think that the test would be useful on a machine with the right GPU and tracepoints.

Yeah I figured. Unfortunately it's not that easy to build the current state for Stadia.
But as you said, it's unclear right now if there is even a point.


fix:grafana cloud dashboard

This PR

fixes the service-level grafana cloud dashboard in case we run 2 clusters with the same name on different installations

Checklist

[x] Update changelog in CHANGELOG.md in an end-user friendly language.

Dashboard was uploaded and is not working fine


Make an assembly video

will speak with dieter about doing a really nice assembly video in the near future

Currently working on a video right now!


KYLIN-4337 Global domain dict for Spark build engine

issue link : https://issues.apache.org/jira/browse/KYLIN-4337

Codecov Report

Merging #1057 into master will increase coverage by 0.03%.
The diff coverage is 9.73%.

@@             Coverage Diff              @@
##             master    #1057      +/-   ##
============================================
+ Coverage        25%   25.04%   +0.03%
- Complexity     6180     6221      +41
============================================
  Files          1438     1439       +1
  Lines         87669    87773     +104
  Branches      12256    12272      +16
============================================
+ Hits          21926    21981      +55
- Misses        63593    63636      +43
- Partials       2150     2156       +6

Impacted Files
Coverage Δ
Complexity Δ

...be/model/CubeDescTiretreeGlobalDomainDictUtil.java
0% <0%> (ø)
0 <0> (?)

...apache/kylin/measure/bitmap/BitmapMeasureType.java
18.86% <0%> (ø)
4 <0> (ø)
:arrow_down:

...ache/kylin/engine/mr/common/AbstractHadoopJob.java
15.21% <0%> (-0.04%)
2 <0> (ø)

...org/apache/kylin/engine/spark/SparkExecutable.java
0% <0%> (ø)
0 <0> (ø)
:arrow_down:

...va/org/apache/kylin/cube/model/DictionaryDesc.java
65.38% <10%> (-34.62%)
8 <1> (+1)

...ain/java/org/apache/kylin/cube/model/CubeDesc.java
63.56% <20%> (-0.57%)
182 <1> (+1)

...org/apache/kylin/metadata/model/DataModelDesc.java
59.21% <25%> (-0.58%)
72 <1> (-1)

...c/main/java/org/apache/kylin/cube/CubeManager.java
36.52% <50%> (+0.02%)
44 <0> (ø)
:arrow_down:

...lin/cube/model/validation/rule/DictionaryRule.java
86.84% <60%> (-4.59%)
16 <1> (+1)

...ream/core/storage/columnar/ColumnarStoreCache.java
50% <0%> (-5.69%)
7% <0%> (-1%)

... and 8 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3777021...11a80bb. Read the comment docs.

Pull Request Test Coverage Report for Build 5516

19 of 113 (16.81%)  changed or added relevant lines in 9 files are covered.
6 unchanged lines in 4 files lost coverage.
Overall coverage decreased (-0.02%) to 27.503%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

core-cube/src/main/java/org/apache/kylin/cube/CubeManager.java
5
6
83.33%

core-cube/src/main/java/org/apache/kylin/cube/model/validation/rule/DictionaryRule.java
4
5
80.0%

engine-mr/src/main/java/org/apache/kylin/engine/mr/common/AbstractHadoopJob.java
0
1
0.0%

engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkExecutable.java
0
2
0.0%

core-metadata/src/main/java/org/apache/kylin/metadata/model/DataModelDesc.java
5
8
62.5%

core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapMeasureType.java
0
4
0.0%

core-cube/src/main/java/org/apache/kylin/cube/model/CubeDesc.java
3
10
30.0%

core-cube/src/main/java/org/apache/kylin/cube/model/DictionaryDesc.java
2
10
20.0%

core-cube/src/main/java/org/apache/kylin/cube/model/CubeDescTiretreeGlobalDomainDictUtil.java
0
67
0.0%

Files with Coverage Reduction
New Missed Lines
%

core-cube/src/main/java/org/apache/kylin/cube/model/CubeDesc.java
1
69.81%

stream-core/src/main/java/org/apache/kylin/stream/core/storage/columnar/ColumnarStoreCache.java
1
56.82%

core-job/src/main/java/org/apache/kylin/job/impl/threadpool/DefaultScheduler.java
2
80.23%

engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkExecutable.java
2
0.0%

Totals

Change from base Build 5512:
-0.02%

Covered Lines:
24140

Relevant Lines:
87773

💛  - Coveralls

Merged in PR https://github.com/apache/kylin/pull/1058


TTC boxes collapse weirdly in Hyper terminal.

Perhaps this should be warned or added in the ReadMe file. Not sure if it's a font issue. (?)

Woof, I wonder if this is a problem with blessed-contrib (the thing that I use to draw the boxes)
I'll try to install hyper term and see if their demos work there

Hadn't seen that lib, it's cool. Alright so I took a look at it and seems like the issue is with blessed-contrib and the issue has been submitted. And now the waiting begins!

Thanks! I'll close this one, i guess, since there's nothing I can do about it :(


Switch to netmiko

@fooelisa @dbarrosop @ktbyers
I think all of us agree that pexpect is already deprecated and proven to be an unreliable solution.
After a couple of months of using it, errors like #21 are not at all surprising anymore.
The XML document is not well-formed (https://github.com/fooelisa/pyiosxr/blob/master/pyIOSXR/iosxr.py#L36) also appears way too often, when the device actually does reply with a proper XML.
I am currently using this version and feels already like a massive improvement, although there's a major IOS-XR specific bug (reported, but without any interest for Cisco to fix it):
When issuing the CLI commands like:
<?xml version="1.0" encoding="UTF-8"?>
<Request MajorVersion="1" MinorVersion="0">
<CLI><Configuration>show commit changes diff</Configuration></CLI>
</Request>

The XML agent crashes and exists from XML mode, after throwing the error:
XML> ERROR: 0x44318c06 'XML-TTY' detected the 'warning' condition 'A Light Weight Messaging library communication function returned an error': No such device or address

All we can do in such case is to re-enter in XML mode... If the device allows, of course.
@ktbyers I am using send_command instead of send_command_expect which does not seem suitable in this particular scenario.

@mirceaulinic I reviewed the Netmiko parts of this.
In the close() method, you can just call self.device.disconnect(). Nothing wrong with the way you did it, however.
In Netmiko_1_0, send_command is going to become send_command_expect. There will be a separate send_command_timing() method. So we will need to fix this when Netmiko 1.0 is released (which won't be for a while).
What was the issue with using send_command_expect()? My concern is that send_command() does tend to be fragile i.e. when delays are meaningfully longer than expected things start to fail. In practice, send_command_expect() is more reliable (it will use the router prompt to determine that the command has completed). Of course, there could be some reason send_command_expect() doesn't work in this context...just didn't see it when looking through the code.

Hi @ktbyers thanks for looking at this.
The library consist basically on the following steps:

open SSH connection
send the command "xml" to enter in XML mode
send XML requests and read XML reply

After entering in the XML mode, the prompt becomes "XML>".
Although I tried to call using
self.device.send_command(command, expect_string='XML>', strip_prompt=False, strip_command=False, delay_factor=delay_factor)

It looks like it is waiting forever and never provides the output from the device. Is there anything I miss?

@ktbyers Also I noticed that huge replies are truncated. I am increasing now the value for max_loops to 1500.
However, my understanding is that send_command_expect should read till the line containing XML>. If that's the case indeed and will be able to make it work properly, I think that's the right method to implement.

@mirceaulinic Let me start my IOS-XR device and test it.

@mirceaulinic Probably should make the expect pattern be a raw string.
expect_string=r'XML>'

Can you capture the manual output of an SSH session up through the entering XML> session?
Also what error do you get when using send_command_expect()?
Yes, the huge replies are truncated is an example of why I thought the send_command_expect() would be better to use.

FYI, I am trying to get my IOS-XRv device to start, but it might be having issues...

@ktbyers My bad, send_command_expect works as expected, just that I need to pay more attention to specify the expect_string properly when switching between contexts.
Thank you very much for your input!
I am pretty content with the current behaviour - it is not perfect but it is definitely a progress. With these said, I will proceed to adapt the tests.

@mirceaulinic FYI, it looks like you forgot to import NetMikoAuthenticationException. You use it later on.
from netmiko.ssh_exception import NetMikoTimeoutException, NetMikoAuthenticationException

@mirceaulinic FYI, I have my IOS-XRv up and running. It is running 5.3.1 code. I can also test against this if you have a set of tests.

@ktbyers

FYI, it looks like you forgot to import NetMikoAuthenticationException

Thanks, missed that!
I just finished writing some tests, some feedback would be much appreciated.

Coverage decreased (-32.06%) to 63.983% when pulling 5bd78f27af424dddcc2614e61900e196b416a020 on mirceaulinic:CF-NETMIKO into e9994827c3e54a322e820c97911afcac87608df0 on fooelisa:master.

Coverage decreased (-32.06%) to 63.983% when pulling 0dd3bb35f4128c9355ca2b1e2f53dd5909c8ea59 on mirceaulinic:CF-NETMIKO into e9994827c3e54a322e820c97911afcac87608df0 on fooelisa:master.

Coverage decreased (-19.4%) to 76.695% when pulling b2c9a1e71d194023ef03ad5cd01fd208e2f1227d on mirceaulinic:CF-NETMIKO into e9994827c3e54a322e820c97911afcac87608df0 on fooelisa:master.

Coverage decreased (-12.9%) to 83.193% when pulling f5cc3d48a0b5ae5fcb6114917c31e5cfbc2d1536 on mirceaulinic:CF-NETMIKO into e9994827c3e54a322e820c97911afcac87608df0 on fooelisa:master.

Coverage decreased (-3.2%) to 92.827% when pulling aa404f932d42cc373fa9805c767e96878d951266 on mirceaulinic:CF-NETMIKO into e9994827c3e54a322e820c97911afcac87608df0 on fooelisa:master.

Coverage decreased (-3.2%) to 92.827% when pulling 131f655fb5c6f772429ac295dd2fe4c17a74e582 on mirceaulinic:CF-NETMIKO into e9994827c3e54a322e820c97911afcac87608df0 on fooelisa:master.

I finished writing the tests, I will wait for some more feedback from @ktbyers then I think we can merge this PR and release a new version.

Coverage decreased (-3.5%) to 92.562% when pulling d46e4208aacad7a7783068946b174cfdd9f14ece on mirceaulinic:CF-NETMIKO into e9994827c3e54a322e820c97911afcac87608df0 on fooelisa:master.

Coverage decreased (-3.5%) to 92.562% when pulling f244c1af49bfad796738895295774501c5891d98 on mirceaulinic:CF-NETMIKO into e9994827c3e54a322e820c97911afcac87608df0 on fooelisa:master.

Y'all IMO this pull request can be merged.
The code in here is already running in the production network for over a week and I can say it works just fine.
I tried to catch all the errors I've seen during the last couple of months, but I would not be surprised to see more :)
@ktbyers did you have any chance to test it a bit?

@mirceaulinic No, I didn't get a chance to test it.
I did look through the code earlier and I thought it was fine for the Netmiko parts. I will probably try to test in the next few weeks, but I am fine with PR being accepted.

@fooelisa could you please release a pyiosxr 0.20?


Figure out to start PS 5.7 with encryption for --test_mode

Based on:
https://dev.mysql.com/doc/refman/5.7/en/innodb-tablespace-encryption.html

It will be nice to start PS 5.7 with encryption enabled.
Run sysbench to have bunch of tables.
Alter those tables to use encryption.
Do whole backup cycle steps.

Seems to be worked now and this is fully implemented.


Fix an example.

Current coverage is 78.76%

Merging #120 into master will decrease coverage by -0.13% as of 8af263e

@@            master    #120   diff @@
======================================
  Files           77      77
  Stmts         2388    2388
  Branches       272     272
  Methods
======================================
- Hit           1884    1881     -3
- Partial         85      86     +1
- Missed         419     421     +2

Review entire Coverage Diff as of 8af263e

Powered by Codecov. Updated on successful CI builds.

LGTM


inconsistency in the crypt_and_hash sample application

when I decrypt a file, it writes: "File content not a multiple of the block size (16)." , for AES-CTR as well (which is not mandatory for CTR)
When I encrypt , we don't do the check for the filesize%block_size
and the error will\should be returned by the cipher itself, if it isn't a multiple of block size.
I think we should be consistent, either check the file size for both encryption and decryption(and for CTR mode not check the size), or not check at all, and rely on the cipher to return the proper error.

ARM Internal Ref: IOTSSL-1445


: ( https://www.typescriptlang.org/play - spam to history api

: (
https://www.typescriptlang.org/play
spam to history api
spam to history api
spam to history api
spam to history api
spam to history api
to browser history api

I'm upset he erased my history

You can turn this off keeping the URL as in the new playground settings page - I'm afraid some browsers don't respect the 'don't set history option;


RFC: device: convert device struct instances to const

This PR has an alternative prototype of the part of #24873 that allows struct device instances to be put into flash.  The intent is to isolate support for instance-specific device mutable state from other enhancements in that and preceding PRs.
It also differs in the approach to dealing with kernel object lookup (by trying to make the lookup methods const-correct) and in the relationship between objects (in that the new struct device_context is specifically intended to be containable within a subsystem-specific common state structure).  Changes in the helper macros that define device objects to support that containment are out of scope for this prototype.
The initial version supports only a flag indicating that initialization succeeded.

Existing code that passes a device pointer as an argument to a function that receives a void * parameter is going to be a huge problem.  The const qualifier must be cast away in the ISR infrastructure, as well as many thread entry functions and other callback user-data parameters.  At this time the necessary conversions don't appear to be automatable.

perhaps cherry-pick the obvious patches I have already done in #24873 about putting struct device to const instead of redoing them.
you'll need that one as well https://github.com/zephyrproject-rtos/zephyr/pull/24873/commits/37171961616717471ffc455a92fe9b3000153533

perhaps cherry-pick the obvious patches I have already done in #24873 about putting struct device to const instead of redoing them.

I looked at them and this told me it wasn't right:
                if (dev != NULL) {
-                       z_object_init(dev);
+                       z_object_init((void *)dev);
                }

We can't go removing const from pointers.  It's outlawed by MISRA 11.8:

A cast shall not remove any const or volatile qualification from the type pointed to by a pointer

We can't go removing const from pointers. It's outlawed by MISRA 11.8:

Did not know that, I should probably download someday the misra papers.
Anyway my point was that instead of chasing the same rabbit in parallel, let's use each-others works properly. At least, I am not going to waste my time on a task if I know somebody else is doing the same work. Unless, you want to take over this task (though it makes already my work a bit of a waste then)

I'm not interested in taking over your wider work on the device structure, but I am interested in pursuing issues related to making struct device const.  I don't feel that this work conflicts with yours, but I do feel that trying to isolate those issues in the context of a set of draft patches that touch many files is too difficult.  So I'm trying to do the minimum necessary to isolate those specific issues.
I expect to have my results in the next day or two after a few more passes through shippable to identify those issues, then we can assess how to proceed.  This effort has already identified existing MISRA violations in current master that should probably be addressed before 2.3 release (#25247).

Dev-Review (May 14):

We should take a step back and decide if making device struct const is a direction we want to go, and thus preclude runtime creation of devices.  (@galak the 2 examples I can come up with are PCI/e devices and USB, scanning a bus at runtime)

Latest run passes all tests, so we'll leave it here.  Some of the commits should be submitted for master regardless of whether we continue with making definitions const.


airgeddon: 11.02 -> 11.10

Description of changes

Changes: https://github.com/v1s1t0r1sh3r3/airgeddon/compare/v11.02...v11.10
Added "asleap" which I didn't note was already in repos.

Things done

Built on platform(s)

[X] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[ ] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 202674 run on x86_64-linux 1

1 package built:

airgeddon


DeployDownloadOptions

Hi,
ist there an example how I can get progress information during downloading a package?
regards
Meex

We need to show examples of this in the documentation, but it's just done as a progress callback of download and extract:
$ionicDeploy.download({
  'onProgress': function(progress) {
    console.log(progress);
  }
});

thx a lot!


Add more asserts to try to track down RenderPadding intrinsics bug

See https://github.com/flutter/flutter/issues/19696

Should https://github.com/flutter/flutter/pull/32817/ be reverted along with this PR? That would increase the instances of this flake occurring and perhaps help with debugging. But, maybe it would happen too frequently that it would hurt the team's overall productivity from having to re-run the flaky tests.

Has anyone been able to reproduce the failures locally on Windows?

I'm not sure, I've only seen the failures in the context of the Windows Cirrus tests flaking.

It's crazy, but the tests keep timing out on Cirrus on all three platforms. I wonder what is going on...

(PR triage): @Hixie Do you still have plans to follow-up on this PR?

I need to track down the timeouts. Haven't had much time recently. The timeouts are very suspicious. If someone wants to take it over that's totally fine.

Dealing with timeouts in https://github.com/flutter/flutter/pull/33932

Turns out these asserts are just really really REALLY expensive (one test on my machine ends up taking about 18 hours to run instead of about 5 seconds). I'll have to try a different approach.


Go command bypass (Security flaw)

If users try to use the go/gå command to go to a meeting that they're not a member of, the command will offer them becoming a member of said meeting. This function does not check if they have status high enough to use the regular member command first.
This way, if you set the status of the member command to a status higher than the user has, the user can bypass this restriction by typing "go " and then accepting to join instead.
Either remove the "do you want to join"-function, or make it validate statusr equirements of the member command before offering to let the user join.

As discussed earlier on NiKom BBS, I will remove the feature to ask if the user wants to become member.


NTUEEPLUS-131. Career Refactor

What is this PR for?

move and create files in /in/career
What type of PR is it?
[Bug Fix | Improvement | Feature | Documentation | Hot Fix | Refactoring]
improvement
Todos

[ ] - no

What is the Jira issue?

https://ntueeplus.atlassian.net/browse/NTUEEPLUS-131
Screenshots (if appropriate)

@featherchen finish!


More auto-deploy changes.

Checklist

[x] Appropriate branch selected (all PRs must first be merged to dev before they can be merged to master)
[x] Any modified or new methods or classes have helpful JavaDoc and code is thoroughly commented
[x] The description lists all applicable issues this PR seeks to resolve
[x] The description lists any configuration setting(s) that differ from the default settings
[x] All tests and CI builds passing

Description
A few more changes so that auto-deployment works somewhat smoothly in practice. And a refactor to consolidate a bunch of job-related logic into its own util file.

Move all job-related fields and methods into new JobUtils class.
Introduce a new active field into MonitorableJob to correctly identify active jobs even when jobs are recurring or not yet started.
Change the way active jobs are found to use that new active field.
When queueing a new DeployJob, wait to create a new one so it doesn't show up in list of jobs for a split second.
Add new field autoDeployWithCriticalErrors to Project and update AutoDeployJob to use this field accordingly
Add a bunch of new completion messages to the AutoDeployJob
Only update feed versions in a deployment when the AutoDeployJob starts a DeployJob
Avoid overwriting deployment data in DeployJob by fetching needed data at point of update

Codecov Report

Merging #381 (db462d0) into auto-deploy (3083c5f) will increase coverage by 0.20%.
The diff coverage is 24.56%.

@@                Coverage Diff                @@
##             auto-deploy     #381      +/-   ##
=================================================
+ Coverage          28.94%   29.14%   +0.20%
- Complexity           729      740      +11
=================================================
  Files                183      184       +1
  Lines               9388     9405      +17
  Branches            1281     1281
=================================================
+ Hits                2717     2741      +24
+ Misses              6384     6377       -7
  Partials             287      287

Flag
Coverage Δ
Complexity Δ

unit_tests
29.14% <24.56%> (+0.20%)
740.00 <12.00> (+11.00)

Flags with carried forward coverage won't be shown. Click here to find out more.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3083c5f...db462d0. Read the comment docs.


Pattern query test with AND operator fails intermittently

Description:
Ballerina stream's pattern query test with AND operator fails intermittently. Therefore, it's disabled in [1]. Creating this to keep track of above, and re-enable tests once it's fixed.
Related PRs:
[1] https://github.com/ballerina-platform/ballerina-lang/pull/9842

We will remove the legacy streams implementation and this issue will be resolved. Till then, the test has been disabled.

Closing because https://github.com/ballerina-platform/ballerina-lang/issues/10033#issuecomment-473790696


Failing while installing on Windows

Installation of this package is failed and it gave me the next log:
><EMAIL_ADDRESS>install C:\Users\grish\AppData\Local\Temp\apm-install-dir-116125-1944-1sbp8sr\node_modules\proto-repl-charts\node_modules\c3\node_modules\d3\node_modules\jsdom\node_modules\contextify
> node-gyp rebuild

C:\Users\grish\AppData\Local\Temp\apm-install-dir-116125-1944-1sbp8sr\node_modules\proto-repl-charts\node_modules\c3\node_modules\d3\node_modules\jsdom\node_modules\contextify>if not defined npm_config_node_gyp (node "C:\Users\grish\AppData\Local\atom\app-1.5.3\resources\app\apm\node_modules\npm\bin\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.js" rebuild )  else (node  rebuild )

gypnpm ERR! Windows_NT 6.2.9200
npm ERR! argv "C:\\Users\\grish\\AppData\\Local\\atom\\app-1.5.3\\resources\\app\\apm\\bin\\node.exe" "C:\\Users\\grish\\AppData\\Local\\atom\\app-1.5.3\\resources\\app\\apm\\node_modules\\npm\\bin\\npm-cli.js" "--globalconfig" "C:\\Users\\grish\\.atom\\.apm\\.apmrc" "--userconfig" "C:\\Users\\grish\\.atom\\.apmrc" "install" "C:\\Users\\grish\\AppData\\Local\\Temp\\d-116125-1944-15dx1i2\\package.tgz" "--target=0.34.5" "--arch=ia32"
npm ERR! node v0.10.40
npm ERR! npm  v2.13.3
npm ERR! code ELIFECYCLE

npm ERR<EMAIL_ADDRESS>install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>install script 'node-gyp rebuild'.
npm ERR! This is most likely a problem with the contextify package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node-gyp rebuild
npm ERR! You can get their info via:
npm ERR!     npm owner ls contextify
npm ERR! There is likely additional logging output above.

My OS is Windows 10 and I already have the 'proto-repl' package in Atom.

The latest fix to switch to git referenced c3 doesn't resolve the issue for me on Windows. Now the install just hangs forever. I'm going to see if I can bundle the dependency next.

@jasongilman Yeah, I have the same situation now on my PC.

It was caused by a dependency on contextify by D3. Later versions don't do this. I forked c3, updated the d3 version and changed the dependency to do that. I was able to install proto repl charts on windows and get it to work.


[Bug] TVM runtime cannot find vm_load_executable for vicuna-7b

🐛 Bug

TVMError:
An error occurred during the execution of TVM.
For more information, please see: https://tvm.apache.org/docs/errors.html
Check failed: (fload_exec.defined()) is false: TVM runtime cannot find vm_load_executable
Stack trace:
File "/home/*/MLC_NEW/tvm-unity/jvm/mlc-llm/cpp/llm_chat.cc", line 246
To Reproduce
Steps to reproduce the behavior:
1.we use this https://huggingface.co/TheBloke/vicuna-7B-1.1-HF to prepare the lib for vicuna
2. and will get the error info as shown above.

Expected behavior

Environment

Platform (Android):
Operating system (Ubuntu:
Device (Samsung s23 ...)
How you installed MLC-LLM (source):
How you installed TVM-Unity (source):
Python version (3.9):
TVM Unity Hash Tag (python -c "import tvm; print('\n'.join(f'{k}: {v}' for k, v in tvm.support.libinfo().items()))", applicable if you compile models):
Any other relevant information:

Additional context

Please check your TVM installation. I don't think you are using TVM Unity. https://mlc.ai/mlc-llm/docs/install/tvm.html

I did follow the Option 2. Build from Source¶, and successfully verify it with the script Validate Installation¶..So use the https://huggingface.co/TheBloke/vicuna-7B-1.1-HF is okay?

To be clear, vm_load_executable is defined on this line in TVM: https://github.com/mlc-ai/relax/blob/1b87dce4aee9f82d657808e9b244e335fd3cb8f0/src/runtime/relax_vm/executable.cc#L63
It means if you are using a correct TVM distribution, you will find this method properly. Therefore, it's not about any particular model. Please do check again if there are multiple different TVM distros you have installed in your system 🙂

(tvm-build-venv) owen@owen-HP-Z4-G4-Workstation:~/MLC_NEW/tvm-unity/build$ git log
commit bd5ee5888f4e7e1e9f14ffcbc335186efb21880f (HEAD -> mlc, origin/mlc, origin/HEAD)
Author: tqchen<EMAIL_ADDRESS>Date:   Mon Jun 12 22:23:11 2023 -0400
[CherryPick][ARITH] Improve arith simplify to handle symbolic reshape pattern

This PR enhances arith simplify to handle symbolic reshape patterns.
Lift the CombineIters to callers of TryFuseIters so they can be used
in early return simplifications. Testcases are added.

Also updates a minor spelling issue in the testcase.

(tvm-build-venv) owen@owen-HP-Z4-G4-Workstation:~/MLC_NEW/tvm-unity/build$ python -c "import tvm; print(tvm._ffi.base._LIB)"
<CDLL '/home/owen/anaconda3/envs/tvm-build-venv/lib/python3.9/site-packages/tvm-0.12.dev1094+gbd5ee5888-py3.9-linux-x86_64.egg/tvm/libtvm.so', handle 5609a1d288a0 at 0x7f7ee0b1fbb0>
(base) owen@owen-HP-Z4-G4-Workstation:~$ python -c "import tvm; print(tvm._ffi.base._LIB)"
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'tvm'
(base) owen@owen-HP-Z4-G4-Workstation:~$
thanks for explanation, and i confirm that i use the lastest tvm distros in my tvm-build-venv? but i am not sure the android build use the tvm version.. how to verify the andoird build use the right version tvm...


Support GitHub Dependency submission API

GitHub's Dependency submission API allows you to submit dependencies for projects, such as the dependencies resolved when a project is built or compiled, see https://docs.github.com/en/rest/dependency-graph/dependency-submission
As ORT generally detects dependencies better than GitHub I propose we add a feature to ORT to allow users to upload found dependencies to GitHub's Dependency submission API. We could then use this feature in ORT for GitHub Action developed in https://github.com/oss-review-toolkit/ort/issues/3512

The notifier module would probably be a good place to implement this.

Hmm, I would have rather thought of a reporter that pushes results to the remote API rather than writing to a local file.

For consistency I recommend we do the same as we did for SW360 as both are about uploading dependency info ... or if we decide to do things differently for Dependency submission API to update our SW360 support

We could probably just re-use the model classes from here.


Npm not installing properly

Linux CentOS 7 x64
In packages.sh I don't think we need curl -sL https://rpm.nodesource.com/setup | bash - as the repo should be taken care of by yum. See https://www.digitalocean.com/community/tutorials/how-to-install-node-js-on-a-centos-7-server for more info.
Also need to explicitly install npm

Closing since this has already been updated to use the package manager.


Add CSS linting

CSS should be linted and autocorrected. Considering CSSComb (http://csscomb.com/)

we have to move this to 3.2.0 due to lack of development

we may also consider https://github.com/causes/scss-lint
reports interesting stuff like too much nesting for example


Keyboard support for sort menu order

Added keyboard support for sort menu order.
Built on top of #5785.
Fixes #5410.

more docs on jquery ui sortable

https://api.jqueryui.com/sortable/
https://stackoverflow.com/questions/4911124/moving-an-item-programmatically-with-jquery-sortable-while-still-triggering-even (looks like you could move the DOM element with javascript and run refresh on the instance)

@PaarthAgarwal  it might be worth raising a PR for just the messages role=status - this will be a small quick win and can be reviewed in isolation of this work.

Messages when we sort page order using keyboard:

Tried doing the same when we move using a mouse but couldn't get the right variable to display the initial position of the element.

this code is looking good in principle - I will validate locally and aim to get this in.


Cannot change maker image on ios (from null to something)

Is this a bug report?
Yes
Have you read the Installation Instructions?
Yes
Environment
Expo: 0.32.0
react-native: 0.57.0
react: 16.5.0
react-native-maps: 0.22.1
iOS with apple maps
Steps to Reproduce
See  https://snack.expo.io/@mfazekas/react-native-map-image-change-issue

This by default places a marker without image.
Press Image1 to change the image attribute of marker to some image. (Nothing happens)
Press toggle twice to remove/readd the marker, now the marker has the image

Expected Behavior
At 1.) i'd either expect the marker to change to the image or a warning to appear that it's not supported.
Actual Behavior
Changing the image attribute of a marker (from null to valid image or vica versa) has no effect on iOS.
Reproducible Demo
https://snack.expo.io/@mfazekas/react-native-map-image-change-issue

Also facing this issue. I want to change the marker image when the marker is selected.


How about having a grid view instead of a list view for listing out the repos that need help?

A lot of wasted space on left and right of the app due to this, and overly long pages.

+1 I'd love to see this. It would condense viewing, great with so many repos.

I would also like to see triage/untriage button alongside the project names. This would help to quickly manage subscriptions to different projects.

Hey guys, we've made some proposed changes to the UI to improve user experience. Would love to get your thoughts. :)
https://github.com/codetriage/codetriage/pull/352

Closing this as we got new design. Thanks for the suggestions everyone!


Add TransportDecorator to the parameter set

Allows addition of logic to (or complete replacement of) the http stack
Signed-off-by: Paul Meyer<EMAIL_ADDRESS>

Enables #6 by hooking up github.com/paulmey/ntlmssp like this:
params := winrm.DefaultParameters()
params.TransportDecorator = func(t *http.Transport) http.RoundTripper { return ntlmssp.Negotiator{t} }
c, err := winrm.NewClientWithParameters(..., params)

LGTM.
Can you, please, add your example to the README so that people know how to use this feature?

How about this?

Thanks @paulmey, this looks good!


DBT Run execution consumption more memory

Describe the bug
I'm running dbt (5 models) in GCP (k8s engine) with minimal record count as 500 Records with some business logic and load the records from one dataset of table to another dataset of table (GBQ) but it consume more memory approx. 2GB.
Execute the same models in laptop, it occupy 5GB of ram out of 8GB. Even same memory occupy second time execution without recompile (No changes in the DBT Models).
Steps To Reproduce
dbt run --models +dq__act --selector aca_weeks
Expected behavior
Need to run the command dbt run with minimal memory.
Screenshots and log output
System information
Which database are you using dbt with?

[ ] postgres
[ ] redshift
bigquery
[ ] snowflake
[ ] other (specify: ____________)

The output of dbt --version:
Running with dbt=0.18.1
The operating system you're using:
The output of python --version:
Python 3.7.6
Additional context
Add any other context about the problem here.

A bit of extra context in Slack: https://getdbt.slack.com/archives/CBSQTAPLG/p1628066783232500

Thanks for the link to the Slack thread @joellabes!
@chandrainnive As Joel shared in Slack, dbt does not store any BigQuery data in memory, it simply sends and receives compiled SQL. (The notable exception is if you're using seeds, but it doesn't sound like that's the case here.)
It doesn't surprise me to hear that it requires the same memory to run dbt each time. It does surprise me to hear that you're measuring dbt's memory consumption at 2-5 GB; I wouldn't expect it to be nearly that much. I'd also expect dbt to require roughly the same memory, whether you run it locally (on your laptop) or in GCP (k8s engine).
It's hard to know exactly what's going on from the details you've shared thus far. I have a few questions, in the hopes that one of these may shed light on something else going on:

How did you install dbt?
Have you tried using a newer version of dbt?
How are you measuring dbt's memory consumption?
Have you tried placing a limit on the memory allocated to each k8s container?

@jtcohen6
Answers for you questions..!
How did you install dbt?
In laptop : pip install dbt. (At the time dbt have a version 0.18.1)
In GCP : In requirement.txt file (dbt<=0.18.0)
Have you tried using a newer version of dbt?
In laptop : Not Yet.
**In GCP :**Not Yet.
How are you measuring dbt's memory consumption?
In laptop :  Task Manager --> open resource monitor--> Memory (Verified the python.exe memory consumption)
**In GCP :**measured the pod memory while executing the dbt task in airflow.
Have you tried placing a limit on the memory allocated to each k8s container?
**In GCP :**No

Thanks for the quick answers @chandrainnive! When I run dbt locally, I see its memory usage at ~130 MB, which is in line with the order of magnitude I'd expect.
High memory usage isn't one of the performance issues we've been tracking, so I'll do some asking around to see if anyone has an instinct as to a root cause or potential leak. In the meantime, I'd be interested to know if dbt runs successfully for you in a memory-constrained environment.

130 MB, Surprise!
Yes, For the single DBT Model there is no issue but it consume more memory.
As per business logic, have to execute 15 DAG's (15 DBT task) on parallel. At that time DAG is failed and ended with this error.
{process_utils.py:66} INFO - Process psutil.Process(pid=16, status='terminated', exitcode=1, started='15:13:10') (16) terminated with exit code 1

@chandrainnive Ah, now we're getting somewhere! Can I ask why you're running 15 different dbt tasks in parallel? We really discourage that and view it as an anti-pattern—if you accidentally run the same dbt models at the same time in separate invocations, bad things tend to happen.
dbt has a built-in mechanism for running independent models concurrently, configured via threads. I think you'd find that a single dbt invocation running many models in parallel will be much more memory-efficient than running each model in a separate invocation.

@jtcohen6  The 15 different dbt tasks are not interlinked between that's. Each DBT task have independent other tasks. (No relation to other dbt models,source and target tables.)
Currently we configured in the profiles.yml with threads=5.
Even i'm trying run the single dbt task in GCP VM machine (Linux)
root@dbt-testing:/home/csomu/dbttesting/DBT# dbt run --models sq_edfi__academicWeeks
csomu@dbt-testing:~$ sudo pmap 2521 | tail -n 1
total          3071468K
For the above DBT model execution takes 3GB of memory consumption.

Hmm ok, I'm really not sure why a single dbt invocation would be eating up 3 GB of memory :(
I'm not sure this will change anything, but you could you try installing a more recent version of dbt, and see if the memory consumption is still that high?

Hi @jtcohen6
Updated the DBT with latest version.
While executing the DBT model, getting the below error
root@dbt-testing:/home/csomu/dbtperformance# dbt run --model dq_edfi_academicWeeks
Running with dbt=0.20.1
Encountered an error:
the JSON object must be str, bytes or bytearray, not 'tuple'
While checking log, it have
File "/usr/local/lib/python3.6/dist-packages/dbt/hooks.py", line 19, in get_hook_dict
return json.loads(source)
File "/usr/lib/python3.6/json/init.py", line 348, in loads
'not {!r}'.format(s.class.name))
TypeError: the JSON object must be str, bytes or bytearray, not 'tuple'
Could you guide for fix this issue.

@chandrainnive It sounds like you may have a pre-hook or post-hook defined that's returning a tuple, rather than a string, array, or dictionary. That error message is not a known regression in v0.20.1, but it sounds like it might be, if it raised immediately after upgrading.
Is there anywhere where you set pre_hook or post_hook to the result of a macro or Jinja object?

@jtcohen6
After Upgraded the dbt with latest and trying to run the same models but it getting error as below.
Running with dbt=0.20.1
Encountered an error:
Compilation Error in model sq__actuals (models/data_lake/sq/sq__actuals.sql)
'NoneType' object is not iterable
Done the below steps
Upgrade the dbt by using pip upgrade dbt.
Added the config version as 2 in dbt_project.yml file.
Ran the dbt deps.
Executing the dbt model.
Need to alter the existing files dbt_project.yml , packages.yml , schema.yml and models/macro ?

Compilation Error in model sq__actuals (models/data_lake/sq/sq__actuals.sql)
NoneType' object is not iterable

Ah, sorry, that's not a particularly helpful error message. It sounds like the model sq__actuals may be looping over a variable set to None? Perhaps the result of a query that is not executed at parse time?

Ok, But the same code is working good in dbt version 0.18.0. I'm not sure....
why the code is not fit to version 0.20.1

@chandrainnive I'm not sure! We made a number of code changes to parsing in v0.19 + v0.20. It's possible that this represents a regression in dbt; it could also be faulty code that wasn't raising an error in older versions, when it should have.

@jtcohen6 - There is no faulty code. everything is working fine and fulfill the business validations in the dbt v0.18. Features  decrepitated may be a reason.
ok..! ok..!
there is any guidelines for reduce the memory consumption in the dbt model execution.

@jtcohen6:  Is there a way to Identify via release notes all of the changes between v0.18 and v0.20. This will aid in the effort the team has to make in refactoring the code.

Hi @jtcohen6
In my DBT model directory had a more than 100 models it includes data vault models too , with respective configuration in dbt_project.yml. (Laptop machine)
If i Just keep only one model and ran the model first time execution it takes 600MB memory consumption. Second & Third time model execution takes the memory consumption as 150MB only. If i kept all models under DBT directory, execute only one model at the time memory consumption is 3.0 GB and memory consumption is same for second and third time execution also.
Here i missing anything in the configuration. Any other thoughts or suggestions pl.


Add a kafka sink

Now, an increasing number of OPS systems have been integrating kafka  to provide the functionality of a messaging system. So I want to add a kafka sink for Heapster. Would you kindly offer me some advice? : )

@huangyuqi: You seemed to closed this issue. Was that intentional?

Sorry, @vishh , : ). I have found an existing issue about the kafka sink.: https://github.com/kubernetes/heapster/issues/172. So, i close this one.


Kbar implemented

https://refine.dev/docs/packages/command-palette/
https://kbar.vercel.app/

Works but :

would be great that it also changes selection in main windows + left sidebar

would be great to make it available on all pages of webapp: for now it's only avaiable on /actions page

I've created simplification issue
https://github.com/launchrctl/web/issues/86
should be one context for selected actions.
https://gyazo.com/64d00f6499e06ceb7c7fd51cbb895fe2

I've added context refactoring. please review.

[ ] would be great to make it available on all pages of webapp: for now it's only avaiable on /actions page

At the moment we can move from /actions and /actions/id page to /flow with highlighted form
basic feature works.
i suggest to merge it and resolve the rest in:
https://github.com/launchrctl/web/issues/86
we still have 2 contexts of active action.


NMRedata or jcamp files with assignments

Describe the bug
Document how to arrive at jcamp or NMRedata from raw files to submit to nmrXiv

Please find data in Bruker format, JCAMP-DX obtained with TopSpin and Mnova (most recent version and an older version) in: https://doi.org/10.22000/1875


fix: Remove extra logic to convert to project numbers now that this is handled in the backend

 8001: com.google.cloud.pubsublite.ProjectLookupUtils: Class com.google.cloud.pubsublite.ProjectLookupUtils removed
Need to add this one to the ignore file.

Fixed.


add lateral window

Hi again man,
I resolved the real time attack map with

now it looks cool, only one problem: I would like to add lateral pop-up window with the same style of the site itself,
do you know how could I accomplish that?
Regards,
Edoardo Mantovani, 2020

Happy new year!
One choice is try to implement a brand new window by imitating:
/src/_includes/collection_list.html: HTML code and Jekyll code of the "Archive" popup window
/src/_sass/components: Style file of "Archive"
There are some steps:

Add a new .html file in /src/_includes and a new .scss file in /src/_sass/components
Include the new .html file in /src/_layout/default.html, include the new .scss file in /src/_sass./_black_n_green.scss
Try to implement the popup window in your mind with a div element

If you need further help, please feel free to contact me.

Hi man,
Sorry for the late reply, but in those days I've tried to write a decent pop up..
unfortunately web isn't my "space" (I prefer C to Javascript & co).
How would you go about adapting the colors(& the size)  of the pop-up to those of the site?
(link: https://baseband-processor.github.io/WirelessLegacy.github.io/)
Regards,
Ah, Happy new year!

Hi man,
Sorry for the late reply, but in those days I've tried to write a decent pop up..
unfortunately web isn't my "space" (I prefer C to Javascript & co).
How would you go about adapting the colors(& the size)  of the pop-up to those of the site?
(link: https://baseband-processor.github.io/WirelessLegacy.github.io/)
Regards,
Ah, Happy new year!

Hi man again and sorry, I forgot to post the code:
(window.html)

<button class="open-button" onclick="openForm()">About the Author</button>

(_window.css)
.open-button {
  background-color: #555;
  color: white;
  padding: 16px 20px;
  border: none;
  cursor: pointer;
  opacity: 0.8;
  position: fixed;
  bottom: 23px;
  right: 28px;
  width: 280px;
}

/* The popup form - hidden by default */
.form-popup {
  display: none;
  position: fixed;
  bottom: 0;
  right: 15px;
  border: 3px solid #f1f1f1;
  z-index: 9;
}

/* Add styles to the form container */
.form-container {
  max-width: 300px;
  padding: 10px;
  background-color: white;
}

/* Full-width input fields */
.form-container input[type=text], .form-container input[type=password] {
  width: 100%;
  padding: 15px;
  margin: 5px 0 22px 0;
  border: none;
  background: #f1f1f1;
}

/* When the inputs get focus, do something */
.form-container input[type=text]:focus, .form-container input[type=password]:focus {
  background-color: #ddd;
  outline: none;
}

/* Set a style for the submit/login button */
.form-container .btn {
  background-color: #4CAF50;
  color: white;
  padding: 16px 20px;
  border: none;
  cursor: pointer;
  width: 100%;
  margin-bottom:10px;
  opacity: 0.8;
}

/* Add a red background color to the cancel button */
.form-container .cancel {
  background-color: red;
}

/* Add some hover effects to buttons */
.form-container .btn:hover, .open-button:hover {
  opacity: 1;
}

Hi man again and sorry, I forgot to post the code:
(window.html)

<button class="open-button" onclick="openForm()">About the Author</button>

(_window.css)
.open-button {
  background-color: #555;
  color: white;
  padding: 16px 20px;
  border: none;
  cursor: pointer;
  opacity: 0.8;
  position: fixed;
  bottom: 23px;
  right: 28px;
  width: 280px;
}

/* The popup form - hidden by default */
.form-popup {
  display: none;
  position: fixed;
  bottom: 0;
  right: 15px;
  border: 3px solid #f1f1f1;
  z-index: 9;
}

/* Add styles to the form container */
.form-container {
  max-width: 300px;
  padding: 10px;
  background-color: white;
}

/* Full-width input fields */
.form-container input[type=text], .form-container input[type=password] {
  width: 100%;
  padding: 15px;
  margin: 5px 0 22px 0;
  border: none;
  background: #f1f1f1;
}

/* When the inputs get focus, do something */
.form-container input[type=text]:focus, .form-container input[type=password]:focus {
  background-color: #ddd;
  outline: none;
}

/* Set a style for the submit/login button */
.form-container .btn {
  background-color: #4CAF50;
  color: white;
  padding: 16px 20px;
  border: none;
  cursor: pointer;
  width: 100%;
  margin-bottom:10px;
  opacity: 0.8;
}

/* Add a red background color to the cancel button */
.form-container .cancel {
  background-color: red;
}

/* Add some hover effects to buttons */
.form-container .btn:hover, .open-button:hover {
  opacity: 1;
}


[Bug] When dinky submitted flinksql, the entry class could not be found in Kubernetes Application mode.

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened
想询问一下这个应该如何正确编写配置 解决这个问题

这是我的dockerfile
FROM flink:1.16.2-scala_2.12
创建用户库目录
RUN mkdir -p $FLINK_HOME/usrlib
RUN echo $FLINK_HOME
修改时区为 Asia/Shanghai
RUN rm -f /etc/localtime && ln -sv /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && echo "Asia/Shanghai" > /etc/timezone
删除旧的 flink-K8S-1.0-SNAPSHOT.jar 文件
RUN if [ -d "$FLINK_HOME/lib" ]; then rm -rf $FLINK_HOME/lib/*; fi
#将dinky的相关jar包房入lib目录中
ADD extends $FLINK_HOME/lib
#将dinky的包放进去
COPY dinky-app-1.16-1.0.1-jar-with-dependencies.jar $FLINK_HOME/dinky-app-1.16-1.0.1-jar-with-dependencies.jar

What you expected to happen
我不确定是哪里出现了问题
How to reproduce
我提交上去就是显示找不到入口类
Anything else
No response
Version
1.0.1
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

This error is meaningless, please provide a more complete log, thank you

This error is meaningless, please provide a more complete log, thank you

这是在dinky中出现的错误

This error is meaningless, please provide a more complete log, thank you

这是在dinky中出现的错误
Please provide k8s log

Please provide k8s log

This error is not a class not found, please observe the error carefully, this is just a generic error that the task has failed, it doesn't make any sense for you to provide this log!

This error is not a class not found, please observe the error carefully, this is just a generic error that the task has failed, it doesn't make any sense for you to provide this log!
这个是什么原因呀,是需要把dinky-app-1.15-1.0.0-SNAPSHOT-jar-with-dependencies.jar 这个包也放到lib目录下吗

我都已经放到容器中了

并且在注册中心已经指定


richer results for play

Codecov Report

Merging #36 (3b87420) into main (3e83cec) will increase coverage by 0.07%.
The diff coverage is 96.42%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #36      +/-   ##
==========================================
+ Coverage   92.18%   92.26%   +0.07%
==========================================
  Files           7        7
  Lines         717      724       +7
==========================================
+ Hits          661      668       +7
  Misses         39       39
  Partials       17       17

Impacted Files
Coverage Δ

play.go
82.44% <96.42%> (+0.99%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3e83cec...3b87420. Read the comment docs.


Add osx and aarch64 builds

Checklist

[x] Used a personal fork of the feedstock to propose changes
[x] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[ ] Ensured the license file is being packaged.

@conda-forge-admin, please rerender

Hi @peci1 - this is ready to be merged. Thanks :)


Multi-scene support

Add the ability to have multiple separate scene hierarchies loaded at once. This is useful for tools/editors that need to have preview boxes and similar. Currently similar functionality can be achieved by using layers for rendering/physics filtering but a multi-scene approach would be more intuitive to use and would provide better and less error-prone design.
The implementation would likely involve:

Frontend - Extend the SceneManager to support multiple scene roots. Each SceneObject should be aware in which scene they are in using some sceneID. We will likely want to prevent scene objects being moved across scenes (as this will likely cause implementation complications with the backend systems and user can just clone the objects nearly as easily if required).
Backend - Systems that need modification that come to mind are Renderer, Physics and perhaps Audio.

Renderer - Renderer internally already works using the concept of scenes through RendererScene. currently there is only one ever allocated but it should be simple enough to create multiple (See RenderBeast). Renderer::notify* methods should then be modified so they work on objects in the appropriate scene using the sceneID. sceneID should probably be part of the SceneActor class since all renderer related objects implement it, and it can be taken from the relevant SceneObject. Then when rendering just iterate over all scenes and things should hopefully just work.
Physics - PhysX internally support scenes, but similarly to the renderer we only currently ever create one. This needs more investigating but a likely approach is similar to the renderer - propagate sceneID with relevant physics objects (likely once again through SceneActor) and then just add objects to different scenes based on the ID. Collision filtering and reporting might need tweaks so they also work on per-scene basis. Scene queries will likely need an additional sceneID parameter as well.
Audio - Potentially no modifications required, or limit the audio to just one scene.

Hi, Is that mean, that we could have ways for example, to manage the preload/unload from a transition from one scene to the other? (as for background loading of scenes, for seamless levels).
I'm not sure reading at your description, as it also seem to load scenes in the background to give live previews of them (I used to create a screenshot to have a preview, but it was static)

I mean preview windows like in the Inspector in an editor, where you can see a 3D model, or a material in separate window, purely for preview purposes.
Loading scenes in the background should already be supported using the async loading feature, although I haven't properly tested that yet so it might need some tweaks. I will be enabling async scene loading in the editor fairly soon, for purposes of testing this feature as well as making editor handle scene loading without freezing the GUI.

This would be useful for multi-monitor games where each monitor can be a different, totally unrelated, scene. E.g. a map view on one screen, part editor (e.g. mech lab from various Mechwarrior titles, vehicle assembly building from kerbal space program) on a second, and direct character control on a third.

I also very much need this feature. So far I'm only using the rendering part so I don't need the physics or audio part of this issue to be fixed. I'm trying to workaround the limitations of a single scene by using layers, but for example, lights, seem to not have a method for setting the layer they are on.

Hey - Just thought I would chime in on the physics side of things.
May be worth looking into shifting the scene origin. Since Banshee already uses PhysX we could use the built in function for shifting the scene origin. :)
https://docs.nvidia.com/gameworks/content/gameworkslibrary/physx/guide/Manual/OriginShift.html

Origin shifting in PhysX can be expensive and "recommend to use it only in the case where distance related precision issues may arise in areas far from the origin.", not to mention the other issues it would cause in specifying offsets everywhere, and doesn't solve the issue of multiple scenes, as origin shifting adjusts the whole scene.
I think it would be ideal to have all APIs associated with the scene, instead of more global like functions where you specify a scene id, this would help create a more clear ownership relationship for scene related systems, it potentially requires more rearchitecting of things but worth the additional cost.

Physics system was already refactored to support multi-scene, with most of the global API moved to PhysicsScene and physics objects can take different scene ownership.
What is left is the renderer, which also mostly supports multi-scene, and then it's just a matter of exposing all this to the user through scene objects and the scene manager.


2680 api errors for pc

Added a server error component if the API fails
Added a loading spinner component for when the pc results form opens up
Prevent forever spinner after 1 min

Looks great!


error when trying to export character

Which version? Should be fixed after v133

v138, also the latest commit (v139) doesn't fix it yet.

v138, also the latest commit (v139) doesn't fix it yet.

Can you take a full screenshot including the command you use and the full error message?

v138, also the latest commit (v139) doesn't fix it yet.

Can you take a full screenshot including the command you use and the full error message?

The last line said Exported (and it's true). The error message is harmless and removed in b6b4776.


updating the README.md

Fixes Issue

Changes proposed

Check List (Check all the applicable boxes)

[ x] My code follows the code style of this project.
[ x] My change requires changes to the documentation.
[ x] I have updated the documentation accordingly.
[ ] All new and existing tests passed.
[ ] This PR does not contain plagiarized content.
[ x] The title of my pull request is a short description of the requested changes.

Screenshots

Note to reviewers

@nickaldwin what Issue does this Pull Request relate to?

Hi @SaraJaoude  apologize  didn't make an issue first  This  issue highlighted the need for improved documentation on how to download Node.js on Debian. In this pull request, I have added detailed steps outlining the process

Thank you for the clarification but as I am sure you understand we have a Contributing Guide in place to manage this project - I would suggest you might want to have a read of this.
The changes you suggest are too specific to one operating system and a specific version number. At the moment we are keeping the documentation generic and do not wish to duplicate any official documentation, so I will close this Pull Request.


No 'Access-Control-Allow-Origin' header is present on the requested resource.

I'm using a REST API that triggers Lambda functions but I keep getting a CORS policy error: No 'Access-Control-Allow-Origin' header is present on the requested resource.
Even when I have specified 'Access-Control-Allow-Origin' and I'm calling the API from the Origin I defined on 'Access-Control-Allow-Origin' (i.e. http://localhost:3000)
how do I fix this error?
region: eu-west-2
app Id: d6ozmsl14fg99
AWS Amplify, API(REST),  Lambda
// Enable CORS for all methods
app.use(function(req, res, next) {
res.header("Access-Control-Allow-Origin", "http://localhost:3000")
res.header("Access-Control-Allow-Headers", "Authorization")
next()
});
//Calling the API
const testAPI = (event) => {
event.preventDefault();
const apiName = "TestAPI";
const path = "/item/testingApi";
const myInit = {
body: {
emails<EMAIL_ADDRESS>/* REQUIRED /,
subject: "The Subject" / REQUIRED /,
body: "The Body \n The Body Also" / REQUIRED /,
replyTo<EMAIL_ADDRESS>/ OPTIONAL */,
},
};
API.post(apiName, path, myInit)
.then( async (response) => {
console.log("success", response);
alert("success: "+ response);
})
.catch((error) => {
console.log("error", error);
});
}

Hi @NobbuMajankwe, are you using the CLI for this? I'm going to transfer this to the CLI repo.
In the meantime, do you mind trying with the ...Allow-Origin/...Allow-Headers set to *? Can you try setting the headers allong the lines of:
 exports.handler = async (event) => {
   // TODO implement
   const { name } = JSON.parse(event.body)
   const body = JSON.stringify(`Hello, ${name}!`)
   const response = {
     statusCode: 200,
     //  Uncomment below to enable CORS requests
     headers: {
       'Access-Control-Allow-Origin': '*',
       'Access-Control-Allow-Headers': '*',
     },
     body,
   }
   return response
 }

https://docs.amplify.aws/guides/api-rest/node-api/q/platform/js/#3-updating-the-function-code.


feat(fields): add cursorHeight property for form builder text field

Connection with issue(s)

Close #719
Solution description
Simply added the delegation to TextField like cursorWidth. However I was not able to test this with the example project. There is some contribution documentation missing on how to build the package locally to use it in the example project.
To Do

[x] Read contributing guide
[ ] Check the original issue to confirm it is fully satisfied
[x] Add solution description to help guide reviewers
[ ] Add unit test to verify new or fixed behaviour
[ ] If apply, add documentation to code properties and package readme

Hi!
Thanks for contribution
To build example, only  need open the example project, get dependencies and run it

Hi! Thanks for contribution To build example, only need open the example project, get dependencies and run it

But just getting the dependency will only fetch the one published? Not the one I changed locally?


[FLINK-17894][table] RowGenerator in datagen connector should be serializable

What is the purpose of the change
RowGenerator will in StreamGraph and be submitted to runtime, so it should be serializable
Verifying this change
DataGenTableSourceFactoryTest
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no

LGTM. Thanks for your fix.

Thanks @zjuwangg for the review, merging...


Add API version endpoint

Clients need to verify if their implementation is compatible, so they need to check our API version.
It would be good to require upgrading version if our schema is changed.

#2112 might be worth considering in this context?


Does not work with non-parallel tests + Fastlane

Non-parallel test runs incorrectly report captured flakes as failures, probably because Fastlane itself fails the run rather than Xcode.
Current workaround is to run tests in parallel since Fastlane doesn't see flakes as failures that way;

This is the line in Fastlane that's detecting the flakes as failures. https://github.com/fastlane/fastlane/blob/6b7251d7346f3e2e1014dacb0640ba43dcb94f0a/scan/lib/scan/runner.rb#L109

Obviously it could still break with parallel tests, but I really encourage you to use parallel tests asap for your iOS automation anyway if you're affected by this!


Document subclassing of computer for novel operations

This is mentioned. Better docs can wait for another day.


Design overhaul

I was thinking something along the lines of

Removed the list styling, added dividers and gave the links the same color you used on the logo.
I can't quite put my fingers on it, but something is off. Let me know what you think.

This looks really good! I think maybe we should make the font size smaller too.

This is in the store now. Always looking for improvements though.


Mention official feature in README.md

Now that k8s supports liveness checks using grpc, we need to mention this in the README.

I have no idea why I can approve this PR


Bad ReqMgr2 spec parameter type error message

Sharad tried to clone a workflow and got an error, which fell into my hands as a error message like:

400 Bad Request

Invalid spec parameter value: int() argument must be a string or a number, not 'NoneType'

which clearly doesn't help much :-) We should add the parameter name into this error message
Whole traceback is:
INFO:cherrypy.access:[05/Nov/2018:16:25:55] vocms0161.cern.ch 2001:1458:d00:13::9c "GET /reqmgr2/data/request?name=amaltaro_RVCMSSW_6_0_0_pre11SingleGammaPt10_120809_164018_8972 HTTP/1.1" 200 OK [data: 6055 in 2635 out 193091 us ] [auth
: OK "/DC=ch/DC=cern/OU=computers/CN=pdmvserv/vocms081.cern.ch" "" ] [ref: "" "Python-urllib/2.7" ]
[05/Nov/2018:16:25:55]  Error: Traceback (most recent call last):
  File "/data/srv/beHG1810f/sw/slc7_amd64_gcc630/cms/reqmgr2/1.1.17.pre3-comp/lib/python2.7/site-packages/WMCore/ReqMgr/Service/Request.py", line 192, in validate
    self._validateRequestBase(param, safe, validate_request_create_args)
  File "/data/srv/beHG1810f/sw/slc7_amd64_gcc630/cms/reqmgr2/1.1.17.pre3-comp/lib/python2.7/site-packages/WMCore/ReqMgr/Service/Request.py", line 87, in _validateRequestBase
    workload, r_args = valFunc(args, self.config, self.reqmgr_db_service, param)
  File "/data/srv/beHG1810f/sw/slc7_amd64_gcc630/cms/reqmgr2/1.1.17.pre3-comp/lib/python2.7/site-packages/WMCore/ReqMgr/Utils/Validation.py", line 104, in validate_request_create_args
    request_args)
  File "/data/srv/beHG1810f/sw/slc7_amd64_gcc630/cms/reqmgr2/1.1.17.pre3-comp/lib/python2.7/site-packages/WMCore/WMSpec/StdSpecs/StdBase.py", line 906, in factoryWorkloadConstruction
    workload = self.__call__(workloadName=workloadName, arguments=arguments)
  File "/data/srv/beHG1810f/sw/slc7_amd64_gcc630/cms/reqmgr2/1.1.17.pre3-comp/lib/python2.7/site-packages/WMCore/WMSpec/StdSpecs/TaskChain.py", line 277, in __call__
    self.setupTask(task, taskConf)
  File "/data/srv/beHG1810f/sw/slc7_amd64_gcc630/cms/reqmgr2/1.1.17.pre3-comp/lib/python2.7/site-packages/WMCore/WMSpec/StdSpecs/TaskChain.py", line 151, in __call__
    self.func(self.obj, task, taskConf)
  File "/data/srv/beHG1810f/sw/slc7_amd64_gcc630/cms/reqmgr2/1.1.17.pre3-comp/lib/python2.7/site-packages/WMCore/WMSpec/StdSpecs/TaskChain.py", line 503, in setupTask
    taskConf=taskConf)
  File "/data/srv/beHG1810f/sw/slc7_amd64_gcc630/cms/reqmgr2/1.1.17.pre3-comp/lib/python2.7/site-packages/WMCore/WMSpec/StdSpecs/StdBase.py", line 395, in setupProcessingTask
    procTask.setProcessingVersion(procVer)
  File "/data/srv/beHG1810f/sw/slc7_amd64_gcc630/cms/reqmgr2/1.1.17.pre3-comp/lib/python2.7/site-packages/WMCore/WMSpec/WMTask.py", line 1349, in setProcessingVersion
    self.data.parameters.processingVersion = int(taskProcVer)
TypeError: int() argument must be a string or a number, not 'NoneType'

[05/Nov/2018:16:25:55]  SERVER REST ERROR WMCore.ReqMgr.DataStructs.RequestError.InvalidSpecParameterValue 18b05fd78d8b32ea62ae4c9941ef507b (Invalid spec parameter value: int() argument must be a string or a number, not 'NoneType')
INFO:cherrypy.error:[05/Nov/2018:16:25:55]  SERVER REST ERROR WMCore.ReqMgr.DataStructs.RequestError.InvalidSpecParameterValue 18b05fd78d8b32ea62ae4c9941ef507b (Invalid spec parameter value: int() argument must be a string or a number, not 'NoneType')
[05/Nov/2018:16:25:55]    Traceback (most recent call last):
INFO:cherrypy.error:[05/Nov/2018:16:25:55]    Traceback (most recent call last):
[05/Nov/2018:16:25:55]      File "/data/srv/beHG1810f/sw/slc7_amd64_gcc630/cms/reqmgr2/1.1.17.pre3-comp/lib/python2.7/site-packages/WMCore/REST/Server.py", line 711, in default
INFO:cherrypy.error:[05/Nov/2018:16:25:55]      File "/data/srv/beHG1810f/sw/slc7_amd64_gcc630/cms/reqmgr2/1.1.17.pre3-comp/lib/python2.7/site-packages/WMCore/REST/Server.py", line 711, in default
[05/Nov/2018:16:25:55]        return self._call(RESTArgs(list(args), kwargs))
INFO:cherrypy.error:[05/Nov/2018:16:25:55]        return self._call(RESTArgs(list(args), kwargs))
[05/Nov/2018:16:25:55]      File "/data/srv/beHG1810f/sw/slc7_amd64_gcc630/cms/reqmgr2/1.1.17.pre3-comp/lib/python2.7/site-packages/WMCore/REST/Server.py", line 783, in _call
INFO:cherrypy.error:[05/Nov/2018:16:25:55]      File "/data/srv/beHG1810f/sw/slc7_amd64_gcc630/cms/reqmgr2/1.1.17.pre3-comp/lib/python2.7/site-packages/WMCore/REST/Server.py", line 783, in _call
[05/Nov/2018:16:25:55]        v(apiobj, request.method, api, param, safe)
INFO:cherrypy.error:[05/Nov/2018:16:25:55]        v(apiobj, request.method, api, param, safe)
[05/Nov/2018:16:25:55]      File "/data/srv/beHG1810f/sw/slc7_amd64_gcc630/cms/reqmgr2/1.1.17.pre3-comp/lib/python2.7/site-packages/WMCore/ReqMgr/Service/Request.py", line 206, in validate
INFO:cherrypy.error:[05/Nov/2018:16:25:55]      File "/data/srv/beHG1810f/sw/slc7_amd64_gcc630/cms/reqmgr2/1.1.17.pre3-comp/lib/python2.7/site-packages/WMCore/ReqMgr/Service/Request.py", line 206, in validate
[05/Nov/2018:16:25:55]        raise InvalidSpecParameterValue(msg)
INFO:cherrypy.error:[05/Nov/2018:16:25:55]        raise InvalidSpecParameterValue(msg)
[05/Nov/2018:16:25:55]    InvalidSpecParameterValue: InvalidSpecParameterValue 18b05fd78d8b32ea62ae4c9941ef507b [HTTP 400, APP 1102, MSG "Invalid spec parameter value: int() argument must be a string or a number, not 'NoneType'", INFO None, ERR None]
INFO:cherrypy.error:[05/Nov/2018:16:25:55]    InvalidSpecParameterValue: InvalidSpecParameterValue 18b05fd78d8b32ea62ae4c9941ef507b [HTTP 400, APP 1102, MSG "Invalid spec parameter value: int() argument must be a string or a number, not 'NoneType'", INFO None, ERR None]
[05/Nov/2018:16:25:55] vocms0161.cern.ch 2001:1458:d00:12::71 "POST /reqmgr2/data/request HTTP/1.1" 400 Bad Request [data: 12141 in 800 out 8571686 us ] [auth: OK "/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=sagarwal/CN=821361/CN=Sharad Agarwal" "" ] [ref: "" "" ]

Hello,
PdmV would also be very grateful if the error message would give some hints on what we do wrong. This morning alone we got this

Invalid spec parameter value: int() argument must be a string or a number, not \'NoneType\'

five times when submitting ~48 RelVals :( Not to mention that this occasionally pops up when submitting Data or MC.

Hi @justinasr , I might be able to get it from the logs, but in case you have the exact dictionary posted to ReqMgr2, could you please share it?

No, unfortunately I don't have the exact dictionary that failed (it is not logged on failure), I can only give you supposedly identical one, i.e. same code builds a dict using same data from DB. Would that work? I can also add logging and wait for the next occurrence.
We always "fix" this by just retrying - it never fails twice, so even the exact dictionary might not be sufficient to reproduce this.

We always "fix" this by just retrying - it never fails twice, so even the exact dictionary might not be sufficient to reproduce this.

Hmm, that's not very encouraging :-(
I found some useful information in the logs and the traceback is very much unexpected. I don't see why ProcessingVersion would be None...
For future reference, once I (or someone else) starts investigating this issue, here is the reqmgr2 log around that error:
[18/Feb/2021:08:25:03]  Updating request "None" with these user-provided args: {u'Group': u'PPD', u'Requestor': u'pdmvserv', u'ProcessingString': u'113X_mcRun4_realistic_v3_2026D49noPU', u'Multicore': 1,
u'SizePerEvent': 1.0, u'Memory': 3000, u'PrepID': u'CMSSW_11_3_0_pre3_G4VECGEOM__UPSG_Std_2026D49noPU-MinBias_14TeV-00001', u'DQMUploadUrl': u'https://cmsweb.cern.ch/dqm/relval', u'Task1': {u'KeepOutput':
 True, u'Campaign': u'CMSSW_11_3_0_pre3_G4VECGEOM__UPSG_Std_2026D49noPU-1613632162', u'GlobalTag': u'113X_mcRun4_realistic_v3', u'Seeding': u'AutomaticSeeding', u'SplittingAlgo': u'EventBased', u'Processi
ngString': u'113X_mcRun4_realistic_v3_2026D49noPU', u'ScramArch': u'slc7_amd64_gcc900', u'RequestNumEvents': 90000, u'ConfigCacheID': u'5588c9b190957287619aa60c3721dee1', u'Memory': 12000, u'Multicore': 8
, u'TaskName': u'GenSimFull', u'AcquisitionEra': u'CMSSW_11_3_0_pre3_G4VECGEOM', u'PrimaryDataset': u'RelValMinBias_14TeV', u'EventsPerJob': 100, u'EventsPerLumi': 100, u'CMSSWVersion': u'CMSSW_11_3_0_pre
3_G4VECGEOM'}, u'Task2': {u'KeepOutput': True, u'GlobalTag': u'113X_mcRun4_realistic_v3', u'InputFromOutputModule': u'FEVTDEBUGoutput', u'Memory': 12000, u'ProcessingString': u'113X_mcRun4_realistic_v3_20
26D49noPU', u'SplittingAlgo': u'LumiBased', u'InputTask': u'GenSimFull', u'ScramArch': u'slc7_amd64_gcc900', u'ConfigCacheID': u'5588c9b190957287619aa60c3721ee9d', u'LumisPerJob': 10, u'Multicore': 8, u'T
askName': u'DigiTrigger_2026D49', u'AcquisitionEra': u'CMSSW_11_3_0_pre3_G4VECGEOM', u'Campaign': u'CMSSW_11_3_0_pre3_G4VECGEOM__UPSG_Std_2026D49noPU-1613632162', u'CMSSWVersion': u'CMSSW_11_3_0_pre3_G4VECGEOM'}, u'SubRequestType': u'RelVal', u'RequestType': u'TaskChain', u'TimePerEvent': 1.0, u'AcquisitionEra': u'CMSSW_11_3_0_pre3_G4VECGEOM', u'CMSSWVersion': u'CMSSW_11_3_0_pre3_G4VECGEOM', u'EnableHarvesting': True, u'RequestPriority': 500000, u'ProcessingVersion': 1, u'Task3': {u'KeepOutput': True, u'GlobalTag': u'113X_mcRun4_realistic_v3', u'InputFromOutputModule': u'FEVTDEBUGHLToutput', u'Memory': 12
000, u'ProcessingString': u'113X_mcRun4_realistic_v3_2026D49noPU', u'SplittingAlgo': u'LumiBased', u'InputTask': u'DigiTrigger_2026D49', u'ScramArch': u'slc7_amd64_gcc900', u'ConfigCacheID': u'5588c9b1909
57287619aa60c37221fc4', u'LumisPerJob': 10, u'Multicore': 8, u'TaskName': u'RecoGlobal_2026D49', u'AcquisitionEra': u'CMSSW_11_3_0_pre3_G4VECGEOM', u'Campaign': u'CMSSW_11_3_0_pre3_G4VECGEOM__UPSG_Std_202
6D49noPU-1613632162', u'CMSSWVersion': u'CMSSW_11_3_0_pre3_G4VECGEOM'}, u'DQMConfigCacheID': u'5588c9b190957287619aa60c372241da', u'RequestString': u'RVCMSSW_11_3_0_pre3_G4VECGEOMMinBias_14TeV__2026D49noP
U', u'CouchURL': u'https://cmsweb.cern.ch/couchdb', u'GlobalTag': u'113X_mcRun4_realistic_v3', u'Campaign': u'CMSSW_11_3_0_pre3_G4VECGEOM__UPSG_Std_2026D49noPU-1613632162', u'DbsUrl': u'https://cmsweb.cer
n.ch/dbs/prod/global/DBSReader', u'ScramArch': u'slc7_amd64_gcc900', u'ConfigCacheUrl': u'https://cmsweb.cern.ch/couchdb', u'TaskChain': 3}
...
[18/Feb/2021:08:25:04]  Error: Traceback (most recent call last):
  File "/data/srv/HG2102e/sw/slc7_amd64_gcc630/cms/reqmgr2/1.4.5.pre4/lib/python2.7/site-packages/WMCore/ReqMgr/Service/Request.py", line 194, in validate
    self._validateRequestBase(param, safe, validate_request_create_args)
  File "/data/srv/HG2102e/sw/slc7_amd64_gcc630/cms/reqmgr2/1.4.5.pre4/lib/python2.7/site-packages/WMCore/ReqMgr/Service/Request.py", line 89, in _validateRequestBase
    workload, r_args = valFunc(args, self.config, self.reqmgr_db_service, param)
  File "/data/srv/HG2102e/sw/slc7_amd64_gcc630/cms/reqmgr2/1.4.5.pre4/lib/python2.7/site-packages/WMCore/ReqMgr/Utils/Validation.py", line 113, in validate_request_create_args
    request_args)
  File "/data/srv/HG2102e/sw/slc7_amd64_gcc630/cms/reqmgr2/1.4.5.pre4/lib/python2.7/site-packages/WMCore/WMSpec/StdSpecs/StdBase.py", line 916, in factoryWorkloadConstruction
    workload = self.__call__(workloadName=workloadName, arguments=arguments)
  File "/data/srv/HG2102e/sw/slc7_amd64_gcc630/cms/reqmgr2/1.4.5.pre4/lib/python2.7/site-packages/WMCore/WMSpec/StdSpecs/TaskChain.py", line 278, in __call__
    self.setupTask(task, taskConf)
  File "/data/srv/HG2102e/sw/slc7_amd64_gcc630/cms/reqmgr2/1.4.5.pre4/lib/python2.7/site-packages/WMCore/WMSpec/StdSpecs/TaskChain.py", line 152, in __call__
    self.func(self.obj, task, taskConf)
  File "/data/srv/HG2102e/sw/slc7_amd64_gcc630/cms/reqmgr2/1.4.5.pre4/lib/python2.7/site-packages/WMCore/WMSpec/StdSpecs/TaskChain.py", line 503, in setupTask
    taskConf=taskConf)
  File "/data/srv/HG2102e/sw/slc7_amd64_gcc630/cms/reqmgr2/1.4.5.pre4/lib/python2.7/site-packages/WMCore/WMSpec/StdSpecs/StdBase.py", line 396, in setupProcessingTask
    procTask.setProcessingVersion(procVer)
  File "/data/srv/HG2102e/sw/slc7_amd64_gcc630/cms/reqmgr2/1.4.5.pre4/lib/python2.7/site-packages/WMCore/WMSpec/WMTask.py", line 1329, in setProcessingVersion
    self.data.parameters.processingVersion = int(taskProcVer)
TypeError: int() argument must be a string or a number, not 'NoneType'

[18/Feb/2021:08:25:04]  SERVER REST ERROR WMCore.ReqMgr.DataStructs.RequestError.InvalidSpecParameterValue 0d0b7c0cfa5ae43a4019300f65d399f2 (Invalid spec parameter value: int() argument must be a string or a number, not 'NoneType')
[18/Feb/2021:08:25:04]    Traceback (most recent call last):
[18/Feb/2021:08:25:04]      File "/data/srv/HG2102e/sw/slc7_amd64_gcc630/cms/reqmgr2/1.4.5.pre4/lib/python2.7/site-packages/WMCore/REST/Server.py", line 726, in default
[18/Feb/2021:08:25:04]        return self._call(RESTArgs(list(args), kwargs))
[18/Feb/2021:08:25:04]      File "/data/srv/HG2102e/sw/slc7_amd64_gcc630/cms/reqmgr2/1.4.5.pre4/lib/python2.7/site-packages/WMCore/REST/Server.py", line 803, in _call
[18/Feb/2021:08:25:04]        v(apiobj, request.method, api, param, safe)
[18/Feb/2021:08:25:04]      File "/data/srv/HG2102e/sw/slc7_amd64_gcc630/cms/reqmgr2/1.4.5.pre4/lib/python2.7/site-packages/WMCore/ReqMgr/Service/Request.py", line 208, in validate
[18/Feb/2021:08:25:04]        raise InvalidSpecParameterValue(msg)
[18/Feb/2021:08:25:04]    InvalidSpecParameterValue: InvalidSpecParameterValue 0d0b7c0cfa5ae43a4019300f65d399f2 [HTTP 400, APP 1102, MSG "Invalid spec parameter value: int() argument must be a string or a number, not 'NoneType'", INFO None, ERR None]
[18/Feb/2021:08:25:04] reqmgr2-68dbcbbf74-g8sn8 <IP_ADDRESS> "POST /reqmgr2/data/request HTTP/1.1" 400 Bad Request [data: 3735 in 799 out 881188 us ] [auth: OK "/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=pdmvserv/CN=821111/CN=Robot: PdmV Service" "" ] [ref: "" "" ]

I looked around the functions that are in the traceback and indeed don't see why it would sometimes have None.
Only one guess/observation (mandatory "I'm not an expert, but...", "As far as I understand", etc.), don't hesitate to ignore:
Method setupProcessingTask in StdBase.py has a line procVer = taskConf.get("ProcessingVersion") or self.processingVersion, so the or should protect from None in taskConf and use self.processingVersion where self is the whole TaskChain (?). This part looks ok and I would wonder why self.processingVersion is None.
When TaskChain is creating and setting up tasks, setupTask method is wrapped in ParameterStorage decorator that does some things with TaskChain's attributes, one of them being processingVersion, i.e. self.processingVersion (processing version of the whole TaskChain?). I am still clueless and not pointing any fingers, but I would maybe look at what exactly is happening when TaskChain's parameters are stored, altered, restored and reset.