Unable to import psdr_cuda

Hello and thanks for sharing your work!
I cannot find any package or module named psdr_cuda after compiling the system. I can only find one pyd file in psdr-cuda\build\lib\Release(or psdr-cuda\build\lib) named psdr_cuda.cp39-win_amd64.pyd. I wonder if this is the right result after compiling? Did I make any mistake while compiling the system?

This seems to be correct. Have you add psdr-cuda\build\lib\Release to your python path?
Also, make sure enoki build is also in your python path

Yeah, I did. And I also tried:
import sys
import os
sys.path.append('D:/study/CG/psdr-cuda/build/lib')
os.environ['path'] += ';D:/study/CG/psdr-cuda/build/lib'
import psdr_cuda
to import psdr_cuda, but it didn't work.

The error is:
ImportError: DLL load failed while importing psdr_cuda:  The specified module could not be found.

I just make an update, can you pull and rerun?

I pull and rerun the system, but errors still occur.
I got some warning while compiling. Did these warnings matter?
D:\study\CG\psdr-cuda\src\core\miniz.cpp(2430,143): warning C4334: '<<' : result of 32-bit shift implicitly converted to 64 bits [D:\study\CG\psdr-cuda\build\psdr_cuda.vcxproj]
D:\study\CG\psdr-cuda\include\psdr/core/tinyexr.h(2449,39): warning C4018: '<' : signed/unsigned mismatch (D:\study\CG\psdr-cuda\src\core\bitmap_loader.cpp) [D:\study\CG\psdr-cuda\build\psdr_cuda.vcxproj]
D:\study\CG\psdr-cuda\include\psdr/core/tinyexr.h(2712,35):warning C4018: '<' : signed/unsigned mismatch (D:\study\CG\psdr-cuda\src\core\bitmap_loader.cpp) [D:\study\CG\psdr-cuda\build\psdr_cuda.vcxproj]
D:\study\CG\psdr-cuda\src\optix\ptx.cpp(94,1): warning C4273: “psdr::getPtxString”: inconsistent dll linkage [D:\study\CG\psdr-cuda\build\psdr_cuda.vcxproj]
D:\study\CG\psdr-cuda\include\psdr/scene/optix.h(271,59): warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data (D:\study\CG\psdr-cuda\src\scene\scene_optix.cpp) [D:\study\CG\psdr-cuda\build\psdr_cuda.vcxproj]

warnings are fine. Are you able to import enoki?

gonna sleep, remember to double-check if enoki is in your PATH and PYTHON PATH.
Here is a guide: https://docs.oracle.com/en/database/oracle/machine-learning/oml4r/1.5.1/oread/creating-and-modifying-environment-variables-on-windows.html

I fix the problem by change the interpreter from python of Windows to a virtual environment created by Anaconda! (But I don't know why it helps) Thanks for help!

one possible reason is that you are using python3.9, while the default enoki precompiled pkg is python3.8


unimportConfig - dead?

The xsl files unimport-extract-disable.xsl are not present. Assuming this is either incomplete, or no longer intended to be used so some comments should be made in the task-defs file.

Hi,
I believe your concerns are coming from here: https://github.com/ibm-datapower/datapower-configuration-manager/blob/881cc35a659c17ec83fe2e76b72ae4bde9c08506/src/dcm-taskdefs.ant.xml#L3466
This code has existed since the original commit in 2014. I suspect this snippet was pulled from internal DataPower somewhere with the intent of saving it for potential future use. The macrodef unimportConfig is not used anywhere, so I am not concerned with it created bugs or problems with the present code. For the time being, I am going to let it be unless it causes a problem.
I have made a note of it in a recent commit.
-Nick

Unimport was intended as "undo" for an import operation, giving you the
ability to remove a service from a domain.  I haven't actually used it in
the real world, though, and I don't know that anyone else has used it, so I
wouldn't suggest using it unless you are prepared to be a pioneer.
On Fri, Sep 15, 2017 at 9:48 AM, Nick Mathison<EMAIL_ADDRESS>wrote:

Closed #20
https://github.com/ibm-datapower/datapower-configuration-manager/issues/20
.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/ibm-datapower/datapower-configuration-manager/issues/20#event-1251343893,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADuMmvS0I7GDGa06YbGH8w1LTLsXhIVBks5sipw9gaJpZM4I8b_P
.

I can see this command having value in the context of UCD if a deployment goes awry following an import step. This is a quick method to revert back to an earlier state, however you would then have incorrect versioning inventory. Typically UCD users would use a snapshot system to revert back to a success state. This option is far more safe.


We have a weird way of causing released images to rebuild.

We have to put this in our dockerfiles to trigger rebuilds on releases to embed version numbers.
# The version has been updated in yarn-project-prod.
# Adding COMMIT_TAG here to rebuild versioned image.
ARG COMMIT_TAG=""

Release please is meant to handle versioning by actually updating code in the repo with a commit, that should result in things being rebuilt, and no need for the hacky "update version" script that we currently use.
Let's make release-please do its thing, and remove these lines from dockerfiles, remove the update version script, and remove the check for these lines from check_rebuild.
I might be missing context. Happy to discuss here.

We added this in as a quick way to rebuild the "final" images when the version number changes. We had to do this because, while yarn-project-prod did get rebuilt, the images downstream of it did not (because none of the code in their rebuild patterns changed).
I think could add .release-please-manifest.json as a rebuild pattern to aztec-sandbox and cli images or add the base image to the rebuild patterns (maybe by adding a build argument to specify the base image to be used by the FROM directive?)


Add slicing options

Each ReChorus instance should know how to train incrementally based on slicing strategy.

Slicing does not contribute much to the project, besides performance improvements. It's only a trivial supplement for sharding, without connnection with clustering strategy. So we put it aside for now.


Use HTTPS for GoogleAPIs repository

Context

See https://github.com/gradle/gradle-private/issues/2027
Gradle Core Team Checklist

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation
[ ] Recognize contributor in release notes

Hi Team,
We are using plugin : 'Java' in our gradle project.
As this vulnerability is for 'Javascript' and 'Coffeescript' plugin, is it necessary for us to upgrade gradle version?
Kindly suggest.

If you are using the API in your build script to add the repository then there is a potential. A deeper analysis would be require for finding out if you are indeed vulnerable when using the API. All this change does is default to HTTPS when adding the repository. If you don't use the API, there is nothing to worry about on that specific issue.


Question: Support for running hie remotely

Occasionally with long running sessions and medium-large size codebases hie grows to use a significant fraction of memory making it hard to work with for any hardware with less than 8GB of RAM.
Since hie already users server-client architecture, is there any support to run hie on a remote machine with potentially more resources?
Additionally, given the convenience of hie-launcher, is this something that could be added to it by launching through SSH (tunneling or not) and making sure editors find the right socket or host:port to talk to?
Somewhat related to https://github.com/haskell/haskell-ide-engine/issues/196

In principle there is no reason this would not work, but there would be plumbing details to sort out.  For example the IDE itself would also need to be aware of the remote server, when opening files, and the protocol would have to run against some kind of handle instead of stdin/stdout.
You may want to take a look at https://github.com/sourcegraph/lsp-adapter, it might be possible to reuse some of what is happening there.

Reading a little on LSP, it does seem like file management isn't a core feature of the protocol.
Some servers implement a file extension.
Another alternative would be to leave file management up to the user (sshfs, NFS, ...) at the cost of some additional latency.
I'll rename the question to a feature request.


Addition of a nunjucks block wrapping the modules calls in outerLayoutBase.html in apostrophe-templates

Added a Nunjucks block to allow overriding Apostrophe's create and mirror calls. The goal here is to allow overriding the modules calls to easily plug in Webpack.

Hmm, you need those calls for Apostrophe's interface to work, and they don't push assets to the browser, they invoke things you already pushed.
So I'm not sure why you would override these even if you got the JS into browserland differently.
Tell me more about your use case.

Hi Tom,
May I complete the request from @Acombes
Currently, in production, every scripts are loaded asynchronously once the page is rendered. This allows progressive enhancement once content, css and media are loaded. (This is for performance purpose, and justified by the kind of website we deploy)
Currently  we override the way apostrophe scripts are loaded (by re-implementing the helper {{ apos.assets.scripts(data.when) }}). We need too, to override the way those scripts are called (globalCalls and reqCalls) in order to doing this async too.
For this purpose, we are forced to override all outerLayoutBase.html (which is not great). By wrapping the script tags (see my comment in the PR source) we will be able to re-implement only this part of the the script loading logic.

I see... well there is no reason to oppose allowing for flexibility!
Please rename the block "jsCalls" and wrap it around the script tag too, as
Jean suggested, and I will merge it.
On Wed, Aug 16, 2017 at 4:11 AM, Jean Ponchon<EMAIL_ADDRESS>wrote:

Hi Tom,
May I complete the request from @Acombes https://github.com/acombes
Currently, in production, every scripts are loaded asynchronously once the
page is rendered. This allows progressive enhancement once content, css and
media are loaded. (This is for performance purpose, and justified by the
kind of website we deploy)
Currently we override the way apostrophe scripts are loaded (by
re-implementing the helper {{ apos.assets.scripts(data.when) }}). We need
too, to override the way those scripts are called (globalCalls and
reqCalls) in order to doing this async too.
For this purpose, we are forced to override all outerLayoutBase.html
(which is not great). By wrapping the script tags (see my comment in the PR
source) we will be able to re-implement only this part of the the script
loading logic.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/punkave/apostrophe/pull/976#issuecomment-322698429,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAB9fXM3dgn3abEd1LSP-e1tDNaRjUR7ks5sYqROgaJpZM4O2Z1X
.

--
*THOMAS BOUTELL, *SUPPORT LEAD
P'UNK AVENUE |<PHONE_NUMBER>  |  punkave.com

Alright, thanks, I'm taking care of it.

Thanks for your contribution!


Peak detection and marching cubes?

Hi @guiwitz ,
Would you be interested in having the peak_local_max as well as the marching_cubes algorithms within this project? I don't think I've seen it implemented often elsewhere and I think they are quite useful to transition more easily between image layers > points layer as  well as Labels layers > Surface layers.
From a menu perspective, I would put them into the following places:
Layers
  - Layer Type
    - Detect local maxima (returns points layer)
    - Marching cubes (returns surface layer)

With the structure you have already provided I don't think it should be a lot of effort to implement - let me know what you think :)

Hi @jo-mueller, these two functions would definitely fit very well in the project, and would be great way to show case different outputs of different types that images and labels. A PR would be very wellcome!


Management: add version information

Is your feature request related to a problem? Please describe.
it would be helpful to see the current version of the management server in self-hosted scenarios
Describe the solution you'd like
Show a version number in the footer.
Describe alternatives you've considered
No idea
Additional context

To find the version fo your current Netbird Management instance, all you need to do is run docker ps get the name for the container likely netbird-management-1 then exec this command int he container docker exec -it netbird-management-1 /go/bin/netbird-mgmt --version. Response should be something like netbird-mgmt version 0.35.1

Thank you. I was aware of how to get the information via the command line. I wanted to see this information in the management dashboard i.e. as a line in the footer to see this without connecting via SSH to the server.
Happy New Year.


Error on search

I get the following error when trying to search:
nil
"2014-06-04 23:16:41.083 osascript[82464:1107] Cannot find executable for CFBundle 0x7fc9e9d3d760 </Library/ScriptingAdditions/AeroFSFinderExtension.osax> (not loaded)
osascript: OpenScripting.framework - scripting addition "/Library/ScriptingAdditions/AeroFSFinderExtension.osax" declares no loadable handlers.
306:307: syntax error: Expected “,” but found unknown token. (-2741)
" - nil
iTunes Search
nil
"2014-06-04 23:16:41.154 osascript[82466:1107] Cannot find executable for CFBundle 0x7fde5bd14c90 </Library/ScriptingAdditions/AeroFSFinderExtension.osax> (not loaded)
osascript: OpenScripting.framework - scripting addition "/Library/ScriptingAdditions/AeroFSFinderExtension.osax" declares no loadable handlers.
306:307: syntax error: Expected “,” but found unknown token. (-2741)
" - nil
iTunes Search
nil
"2014-06-04 23:16:41.221 osascript[82468:1107] Cannot find executable for CFBundle 0x7fdba9e1e5d0 </Library/ScriptingAdditions/AeroFSFinderExtension.osax> (not loaded)
osascript: OpenScripting.framework - scripting addition "/Library/ScriptingAdditions/AeroFSFinderExtension.osax" declares no loadable handlers.
306:307: syntax error: Expected “,” but found unknown token. (-2741)
" - nil
iTunes Search
nil
"2014-06-04 23:16:41.293 osascript[82470:1107] Cannot find executable for CFBundle 0x7fe5706410b0 </Library/ScriptingAdditions/AeroFSFinderExtension.osax> (not loaded)
osascript: OpenScripting.framework - scripting addition "/Library/ScriptingAdditions/AeroFSFinderExtension.osax" declares no loadable handlers.
306:307: syntax error: Expected “,” but found unknown token. (-2741)
" - nil

Removing /Library/ScriptingAdditions/AeroFSFinderExtension.osax now fixes this on OS X Yosemite.

Wow, excellent. Amazing detective work and commitment. I'm sorry I never could help more.


No contextual typing from return types for boolean literals

Fixes #48363.
Also fixes issue mentioned here because boolean literals contextually typed by boolean are now widened.

@typescript-bot test this
@typescript-bot user test this inline
@typescript-bot run dt
@typescript-bot perf test faster

All tests are clean and performance is unaffected. I think this one is good to merge.


Graphql BOM depends on old snapshot of  graphql-java-extended-scalars

Expected Behavior
My expectation is that the micronaut-graphql-bom would depend on graphql-java-extended-scalars:22.0
Actual Behaviour
Currently it depends on 2023-01-24T02-11-56-babda5f which is a master branch snapshot.
Steps To Reproduce
See https://mvnrepository.com/artifact/io.micronaut.graphql/micronaut-graphql-bom/4.5.0
Environment Information
No response
Example Application
No response
Version
4.5.0

@johndevs I fixed it here: https://github.com/micronaut-projects/micronaut-graphql/pull/572
FYI: This repo just aggregating all micronaut projects BOMs. If you have a problem or suggestion about one of them - need to create issue in project repo. In you case this is https://github.com/micronaut-projects/micronaut-graphql


Additions to the recompiler.

Added all the Registers I could find.
Also added the codes: Movzx_r32_rm8, Nopd, Setne_rm8, Sub_64_imm8

I'd also like to say I tested all these operations, and none of them threw an error or gave an unexpected behavior.

I fixed these issues, however, I'm working on a part 2 for this, so hold until next commit.

@ultimaweapon Have fun! I would keep going but... I'm stuck on Imul_r32_rm32_imm8


Proxy and the homeserver disagree about one-time key counts

There seems to be an issue where, if a user claims one-time keys of our user, the proxy won't get the correct count after we replenish our one-time keys.
The /keys/upload requests is happening right after the /sync and tells us that there are 51, even though the request only uploaded a single additional one. The next sync resets our count again to 49 and we upload another one-time key because of that.
This continues happening until we have 53 one-time keys in total on the homeserver. Eventually the /sync response will tell us that we have more than 50 keys uploaded and we stop uploading.
This is mostly to do with the fact that we don't currently eagerly push OTK updates to clients.

Fixed as of https://github.com/matrix-org/sliding-sync/commit/6c4f7d37225ed9d2b3a18268ac2fe8fedecdc7a4 - we now eagerly push OTK updates.


Arch Linux wallet configuration

A tab with directions for Arch Linux configuration for wallet

Bump. Can someone merge this?


Startup probe and Readiness probe failed.

I have installed the operator and trying to create cluster with the example config given. Its creating the pods but pods are not in ready state as startup probe is failing.  Startup probe failed: dial tcp 1.x.x.x:9200: connect: connection refused
I tried to exec one of the pod and executed curl commands, below is the result.
sh-4.2$ curl -XGET https://localhost:9200 -u 'admin:admin' --insecure
OpenSearch Security not initialized.sh-4.2$ ^C
sh-4.2$ ^C
sh-4.2$ ^C
sh-4.2$ curl -XGET http://localhost:9200 -u 'admin:admin' --insecure
curl: (52) Empty reply from server
sh-4.2$
Can anyone help me to understand what is going wrong.
apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  name: my-cluster
  namespace: opensearch-demo
spec:
  general:
    version: 1.3.2
    httpPort: 9200
    vendor: opensearch
    serviceName: my-cluster
  dashboards:
    version: 1.3.0
    enable: true
    replicas: 2
    resources:
      requests:
         memory: "1Gi"
         cpu: "500m"
      limits:
         memory: "1Gi"
         cpu: "500m"
  confMgmt:
    smartScaler: true
  nodePools:
    - component: masters
      replicas: 3
      diskSize: "1Gi"
      NodeSelector:
      resources:
         requests:
            memory: "2Gi"
            cpu: "500m"
         limits:
            memory: "2Gi"
            cpu: "500m"
      roles:
        - "master"
        - "data"
    - component: nodes
      replicas: 3
      diskSize: "1Gi"
      NodeSelector:
      resources:
         requests:
            memory: "2Gi"
            cpu: "500m"
         limits:
            memory: "2Gi"
            cpu: "500m"
      roles:
        - "data"
    - component: coordinators
      replicas: 3
      diskSize: "1Gi"
      NodeSelector:
      resources:
         requests:
            memory: "2Gi"
            cpu: "500m"
         limits:
            memory: "2Gi"
            cpu: "500m"
      roles:
        - "ingest"

Below is the opensearch.yml in the pod
cluster.name: docker-cluster

#### Bind to all interfaces because we don't know what IP address Docker will assign to us. ###
network.host: <IP_ADDRESS>

###  minimum_master_nodes need to be explicitly set when bound on a public IP ###
###  set to 1 to allow single node clusters ###
### discovery.zen.minimum_master_nodes: 1 ###

### Setting network.host to a non-loopback address enables the annoying bootstrap checks. "Single-node" mode disables them again. ###
### discovery.type: single-node ###

######## Start OpenSearch Security Demo Configuration ########
### WARNING: revise all the lines below before you go into production
plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - CN=kirk,OU=client,O=client,L=test, C=de

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".plugins-ml-model", ".plugins-ml-task", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*",
".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opensearch-notifications-*", ".opensearch-notebooks", ".opensearch-observability", ".opendistro-asynchronous-
search-response*", ".replication-metadata-store"]
node.max_local_storage_nodes: 3
######## End OpenSearch Security Demo Configuration ########

Hi @NoorKumar, can you check in kubernetes if the pod <cluster-name>-securityconfig-update is completed and what its log says? You could also check the log of the opensearch pods themselves if they report any problems. Note that starting up a cluster can take a few minutes.
Also: Did you test with the newest version (2.0)?

Closing, as there was no further response from the issue reporter.


⚠ Bofrost.be has degraded performance

In f4e6aff, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 1238 ms

Resolved: Bofrost.be performance has improved in 99b8703 after 38 minutes.


Tooltips are not working properly

The tooltips seem to be having issues. For example, when hovering inside a chart with multiple timeseries, the tooltip is shown and the series is listed in the tooltip, but no values are displayed. They are empty, although the display of the panel is set to show the tooltip of all series.
Only if the cursor touches one of the graph lines a value is displayed for that graph only in the tooltip, which is not the best when on wants to see values from all graphs alongside each other.
Any chance to fix this? Maybe I could try to fix it, but can someone tell me where/what I am looking for, as I have not worked with grafana's internals.
thanks.

Do you have a screenshot and examples?

Sure:
On multi-series charts the values are missing:

Notice here, I point on the Pressure chart and it works, but it doesn't on the Humidity above:

Grafana is 5.2.1. Not sure I should do anything; i tried changing the display settings of the charts, but the values are not displayed.
thank you.

I'm seeing the same sort of issues. The graphs are visually correct, but no numbers. The table also has the correct data. I noticed that the type of the value field I'm aggregating is a string. Maybe that has something to do with this?

I added in a parseInt for the timeseries JSON generator to convert strings to integers and that solved my issues. I have no idea what will happen for other data types, but that seems to be the root cause!


Use event.timeStamp instead of Date.now()

Pull Request Test Coverage Report for Build 235

0 of 0 (NaN%) changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build 233:
0.0%

Covered Lines:
103

Relevant Lines:
103

💛  - Coveralls

I'd like to use event.timeStamp instead of creating a timestamp with Date.now(). It seems cleaner to me to use an already created timestamp than to create our own.
Also, it looks like Chrome improved event timestamp accuracy a couple years ago.


[stable/mongodb-replicaset] Update MongoDB Exporter

What this PR does / why we need it:
Update the MongoDB exporter to version 0.11.0. Remove the the values for the --mongodb.socket-timeout and --mongodb.sync-timeout flag.
Another option would be to add the socket and connection timeout to the connection string and change the value names to sth. like metrics.connectTimeoutMS and metrics.socketTimeoutMS.
Checklist

[x] DCO signed
[x] Chart Version bumped
[x] Variables are documented in the README.md
[x] Title of the PR starts with chart name (e.g. [stable/mychartname])

/assign @steven-sheehy

Hi, is there any chance to get this merged?

/ok-to-test


[SUPPORT] Unstable Flink-Hudi connector on scale

Tips before filing an issue

Have you gone through our FAQs? Yes

Join the mailing list to engage in conversations and get faster support at<EMAIL_ADDRESS>

If you have triaged this as a bug, then file an issue directly.

 We are observing a lot of issues with few tables. The table emits around 10K upserts per second (a huge number of updates).
We are using EMR 6.11.0 with Hudi version 13.0 and Flink version 1.16.0
Initially, we tried Copy On Write (COW) but then saw lot of issues with checkpointing in Flink. The main culprit was the error -
Checkpoint expired before completing.
Increasing resources, decreasing checkpoint interval, increasing checkpoint timeout nothing helped.
We then moved to Merge On Read table. We were still seeing issues like -
java.lang.IllegalStateException: Receive an unexpected event for instant<PHONE_NUMBER>1658265 from task 7
Open hudi issue like this suggest that it's multiple writer issue. We set the writers to 1 to resolve this and then increasing checkpoint timeout from 15 minutes to 60 minutes helped.
The checkpoint was taking around 30 minutes.
However, the main problem we faced here was with whatever config we change for the Flink pipeline, we had to rebootstrap the table. This was not the case with CoW table. Bootstrapping is expensive for us and takes quite some time. If we do not bootstrap, then we see the error of IllegalStateException again or FileAlreadyExistsException (for log files)
Our main questions are -

Is there any way where we can circumvent bootstrapping for flink config changes like changing checkpoint interval etc for the MOR option.
For such a pipeline, how do we parallelize this with multiple writers without running into aforementioned IllegalStateException/FileAlreadyExistsException problem.
What better configs should we set to increase the efficiency of the ingestion?

To Reproduce
Steps to reproduce the behavior:

On EMR, setup an ingestion from MySQL Debezium changelogs emitted to Kafka and finally consumed by Flink to create a Hudi table
The scale of changelogs should be around 10K upserts/sec
Hudi/Flink configs

MOR table
Single writer
Checkpoint configuration -

hudi.clean.policy=KEEP_LATEST_BY_HOURS
--hudi.clean.retain_hours=168
All other configs are just default configurations

Environment Description

Hudi version : 13.0

Flink version : 1.16.0

Hive version :

Hadoop version :

Storage (HDFS/S3/GCS..) : S3

Running on Docker? (yes/no) : no

Stacktrace
Stack trace of the aforementioned errors -

FileAlreadyExistsException

org.apache.hudi.exception.HoodieRollbackException: Failed to rollback s3://slack-dw/hudi/channels_members commits<PHONE_NUMBER>1658265
...
Caused by: org.apache.hudi.exception.HoodieException: Error occurs when executing flatMap
...
Caused by: org.apache.hudi.exception.HoodieRollbackException: Failed to rollback for instant [==>20230912181658265__deltacommit__INFLIGHT]
...
Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: File already exists:s3://slack-dw/hudi/channels_members/.108ba95b-d395-4b98-826d-9e8e0b26d8e1-1_20230912004700323.log.1247_1-0-1
...
2023-09-13 19:50:06,654 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Trying to recover from a global failure.
org.apache.flink.util.FlinkException: Global failure triggered by OperatorCoordinator for 'stream_write: channels_members' (operator cd7a24f114e59daa8b573b2d08871155).
	at org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder$LazyInitializedCoordinatorContext.failJob(OperatorCoordinatorHolder.java:617) ~[flink-dist-1.16.0.jar:1.16.0]

IllegalStateException

org.apache.flink.util.FlinkException: Global failure triggered by OperatorCoordinator for 'stream_write: channels_members' (operator cd7a24f114e59daa8b573b2d08871155).
	at org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder$LazyInitializedCoordinatorContext.failJob(OperatorCoordinatorHolder.java:617)
	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$start$0(StreamWriteOperatorCoordinator.java:190)
	at org.apache.hudi.sink.utils.NonThrownExecutor.handleException(NonThrownExecutor.java:142)
	at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:133)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.hudi.exception.HoodieException: Executor executes action [handle write metadata event for instant ] error
	... 6 more
Caused by: java.lang.IllegalStateException: Receive an unexpected event for instant<PHONE_NUMBER>1658265 from task 7
	at org.apache.hudi.common.util.ValidationUtils.checkState(ValidationUtils.java:67)
	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.handleWriteMetaEvent(StreamWriteOperatorCoordinator.java:447)
	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$handleEventFromOperator$4(StreamWriteOperatorCoordinator.java:288)
	at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:130)
	... 3 more

Sorry for the issue @fenil25 , the release 0.13.0 is a very buggy release, I'm wondering if you can try rellease 0.12.3 or 0.13.1 instead.

@fenil25 Were you able to try with the 0.12.3 or 0.13.1. Did you still faced this issue?


[Windows 10] Excessive CPU usage

I have checked that there are no issues with similar or the same content
I'm seeing other CPU usage issues, for Mac especially. Also checked closed issues, there's #1777 that's already closed but seems a little related... I guess
I have checked the FAQ (https://github.com/MarshallOfSound/Google-Play-Music-Desktop-Player-UNOFFICIAL-/wiki/FAQ) and the answer I am looking for is not there
I have double checked and can reproduce the issue
every time

OS:
Windows 10 x64
GPMDP Version: 4.2.0
16 gigs of ram, i7 @3.4Ghz (4c,8t)
Screen resolution: 3200x1800 DPI: roughly 262 from my quick maths (14" screen)
Touchscreen: yes
Issue Descriptions:
I'm experiencing pretty big CPU hit when using the player (~10-20%). Noting that I have a 4-core 8-thread i7 clocked at 3.4GHz, that's a big chunk of CPU being utilised for music playback. Since I can't find other reports related, I'm thinking this may be related to UI drawing and my resolution which is 3200x1800. I'm happy to do troubleshooting if that would help, I'm loving the player.
~1% CPU when UI is closed using the X button after starting playback
~2-4% CPU when UI is minimized

~8% when having the window open and not moving the cursor at all

~20% when moving the cursor over interactive elements in the window

~20% when moving the cursor over empty, non-interactive space in the window

(looks like a different thread is hogging the resources here compared to the interactive elements one)
This has been going on for over 6 months and over different app versions
Steps to Reproduce:
Launch the app. Play a song. Minimize the player to see reduced CPU usage. Restore it and move cursor around inside the window to see abnormal CPU usage

CPU usage on other platforms has the same cause as this, GPM is poorly optimized and I can do nothing about it, see those mac issues you mentioned for some tricks you could use on windows to get the usage down 👍

I'm having similar issues, although not quite to the same scale.
No bump when moving over blank areas
~18% when moving over interactive areas
~50% bump when moving over the "related artists" section
And the biggest one: When playing music in Google Music in Chrome, my total CPU usage goes up by ~10 % points. When it's paused it goes back down.
When playing music in GMPDP, total CPU usage goes up by ~20%, again going back down when paused.
Not sure if anything can be done about this, or if it's even a bug, but thought it at least worth mentioning here, since it doesn't seem worthy of a new issue.


Pasted key: needs to guess the type of the key being pasted

Because we use this later to display and show other bits of information.

Currently, the UI hardcodes the type of the key to RSA-4096, which is wrong.
But also, not that important!   That information is purely for cosmetic purposes, so it could be removed


Deploy with Portainer issue

Recently I started receiving the error described here https://github.com/karfly/chatgpt_telegram_bot/issues/318 (empty messages)
So I decided to redeploy it.
I try to deploy using Portainer and get an error:
Traceback (most recent call last):
File "bot/bot.py", line 33, in
import config
File "/code/bot/config.py", line 8, in
with open(config_dir / "config.yml", 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/code/config/config.yml'
So the part "/code/" seems wrong.
I think I have to rewrite the path to the config files somewhere in bot.py
Any suggestions?

Did you rename the files in config/config.example.yml to config.yml?


[sobjectizer] Update to <IP_ADDRESS>

SObjectizer updated to the latest version.

Thanks for the update!


Request: add support for building arbitrary binaries (bins/examples/tests)

Currently, it's not possible to pass --example to serve and test out components in isolation. It would be great if this were possible to test out different configurations of an app, or to view the examples for a given library.

I think this is very similar to #287. We currently don't support many cargo options, and in my opinion we should just add a lot in one big sweep.
That means we would add:

Target selection: --lib, --bin, --example, --package
Feature selection: --no-default-features, --features, --all-features
Others: --locked, --offline, --frozen

There are some others (--test, --bench, and selectors to build all bins, libs, ...), but I don't think they would be useful sense in the context of Trunk as it's rather unusual to bundle a test or benchmark and building all bins would be difficult as Trunk (currently) expects a single target.

I'd really like --locked as it helps with caching for CI build systems.


AttributeError: 'PoseHigherResolutionNet' object has no attribute 'module'

when I run the "dist_train.py", I got a error:
Traceback (most recent call last):
File "tools/dist_train.py", line 322, in
main()
File "tools/dist_train.py", line 123, in main
tb_log_dir
File "tools/dist_train.py", line 306, in main_worker
'best_state_dict': model.module.state_dict(),
File "/root/miniconda3/envs/hh/lib/python3.6/site-packages/torch/nn/modules/module.py", line 535, in getattr
type(self).name, name))
AttributeError: 'PoseHigherResolutionNet' object has no attribute 'module'
My configuration environment is CUDA9.0+pytorch1.0.1+tensorboardX1.6, I guess this problem may have something to do with the pytorch version. I haven't solved this problem yet. Has anyone solved this problem?

I have successfully solved the proble.

Hello, I have the same problem. How did you solve this problem (Python 1.2.0, torch vision 0.2.1, cuda10.0)?


Step(s) to populate data missing

The deployment guide does not contain steps to populate data. The most visible result is broken images, as the storage account that should contain them is empty.
Can this be included?

Not having the Azure storage accounts populated with a data is a big miss, At least include a storage account there so we can manually update

Hi!
I've updated the doc and added a script (/Deploy/Deploy-Pictures-Azure.ps1) that copies images to Azure Storage.
There is also the option to run an azure storage emulator in AKS.


Update for 10.0.0 release

Bump for 10.0.0
Checklist

[x] I've checked that all new and existing tests pass
[x] I've updated the documentation if necessary
[x] I've added an entry in the CHANGELOG to credit myself

Description

Use the latest version of cocoapods (1.15.2)
Bump version to 10.0.0

@Liquidsoul FYI: Just did similar update to latest CocoaPods in https://github.com/AliSoftware/OHHTTPStubs/pull/368 than the one you included in this PR


test: playwright initial setup and basic test

Initial setup for Playwright and a basic test.
Why Playwright? I think this library could do with some in browser tests, although the tests with React Testing Library are pretty good, some things are still not quite the same ie. cursor movement.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (1a3d1bd) 100.00% compared to head (dcd9126) 100.00%.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff            @@
##              main      #327   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           15        15
  Lines          411       411
  Branches       159       159
=========================================
  Hits           411       411

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


RuntimeError: expected mat1 and mat2 to have the same dtype, but got: c10::Half != float

Dear authors,
Thanks for your nice work! I have already cloned this repo and downloaded the 'Platypus2-7B' model. However, I meet with the following error when running the 'fine-turning.sh':
Traceback (most recent call last):   File "finetune.py", line 245, in <module>     fire.Fire(train)   File "/home/anaconda3/envs/platypus/lib/python3.8/site-packages/fire/core.py", line 141, in Fire     component_trace = _Fire(component, args, parsed_flag_args, context, name)   File "/home/anaconda3/envs/platypus/lib/python3.8/site-packages/fire/core.py", line 475, in _Fire     component, remaining_args = _CallAndUpdateTrace(   File "/home/anaconda3/envs/platypus/lib/python3.8/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace     component = fn(*varargs, **kwargs)   File "finetune.py", line 172, in train     trainer.train(resume_from_checkpoint=resume_from_checkpoint)   File "/home/anaconda3/envs/platypus/lib/python3.8/site-packages/transformers/trainer.py", line 1539, in train     return inner_training_loop(   File "/home/anaconda3/envs/platypus/lib/python3.8/site-packages/transformers/trainer.py", line 1869, in _inner_training_loop     tr_loss_step = self.training_step(model, inputs)   File "/home/anaconda3/envs/platypus/lib/python3.8/site-packages/transformers/trainer.py", line 2777, in training_step     self.accelerator.backward(loss)   File "/home/anaconda3/envs/platypus/lib/python3.8/site-packages/accelerate/accelerator.py", line 1851, in backward     self.scaler.scale(loss).backward(**kwargs)   File "/home/anaconda3/envs/platypus/lib/python3.8/site-packages/torch/_tensor.py", line 492, in backward     torch.autograd.backward(   File "/home/anaconda3/envs/platypus/lib/python3.8/site-packages/torch/autograd/__init__.py", line 251, in backward     Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass   File "/home/anaconda3/envs/platypus/lib/python3.8/site-packages/torch/autograd/function.py", line 288, in apply     return user_fn(self, *args)   File "/home/anaconda3/envs/platypus/lib/python3.8/site-packages/torch/utils/checkpoint.py", line 288, in backward     torch.autograd.backward(outputs_with_grad, args_with_grad)   File "/home/anaconda3/envs/platypus/lib/python3.8/site-packages/torch/autograd/__init__.py", line 251, in backward     Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass   File "/home/anaconda3/envs/platypus/lib/python3.8/site-packages/torch/autograd/function.py", line 288, in apply     return user_fn(self, *args)   File "/home/anaconda3/envs/platypus/lib/python3.8/site-packages/bitsandbytes/autograd/_functions.py", line 480, in backward     grad_A = torch.matmul(grad_output, CB).view(ctx.grad_shape).to(ctx.dtype_A) RuntimeError: expected mat1 and mat2 to have the same dtype, but got: c10::Half != float

Yes, I get the same error. Did you find any solution?

Yes, I get the same error. Did you find any solution?
with Platypus2-7B

I change gpu，A800 is OK

I change model.py line33 "load_in_8bit=True" as "load_in_4bit".

Yes, I get the same error. Did you find any solution?


python312Packages.tencentcloud-sdk-python: 3.0.1118 -> 3.0.1119

Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 299947 run on x86_64-linux 1

4 packages built:

python311Packages.tencentcloud-sdk-python
python311Packages.tencentcloud-sdk-python.dist
python312Packages.tencentcloud-sdk-python
python312Packages.tencentcloud-sdk-python.dist


MGet with go redis failing

Actual behavior The unit test at https://github.com/PhilAndrew/mockgenredisv9/blob/main/mock_v9/redis_cache_v2_test.go#L36 should pass as it takes and supplies 3 parameters. Instead it says wrong number of arguments in DoAndReturn func for *mock_v9.MockCmdable.MGet: got 3, want 2. This suggests there is a bug in mockgen where it is not compatible with Redis v9 for the MGet method https://github.com/redis/go-redis
I tried modifying the number of parameters but that didn't work.
Expected behavior I expect the unit test at this github repo to pass. https://github.com/PhilAndrew/mockgenredisv9
To Reproduce Steps to reproduce the behavior
Git clone https://github.com/PhilAndrew/mockgenredisv9 and run go test -cover ./...
Additional Information

gomock mode (reflect or source):I generated source
gomock version or git ref:Latest version v0.3.0
golang version:1.20

Triage Notes for the Maintainers

Hi Phil, I got the solution from your problem


failed to cherry pick PR 4196 from ty-dc, to branch release-v0.9

commits a810abd2711d2fa46361ca45c1c4d539774cf74b of ty-dc conflict when merging to branch release-v0.9, please manually cherry pick it by yourself.
PR https://github.com/spidernet-io/spiderpool/pull/4196 ,  action https://github.com/spidernet-io/spiderpool/actions/runs/11514732841
CONFLICT (modify/delete): docs/usage/dra.md deleted in HEAD and modified in a810abd2 (Merge pull request #4196 from ty-dc/fix/ipam-image).  Version a810abd2 (Merge pull request #4196 from ty-dc/fix/ipam-image) of docs/usage/dra.md left in tree.
CONFLICT (modify/delete): docs/usage/dra_zh_CN.md deleted in HEAD and modified in a810abd2 (Merge pull request #4196 from ty-dc/fix/ipam-image).  Version a810abd2 (Merge pull request #4196 from ty-dc/fix/ipam-image) of docs/usage/dra_zh_CN.md left in tree.
Auto-merging docs/usage/readme-zh_CN.md
Auto-merging docs/usage/spider-multus-config-zh_CN.md
Auto-merging docs/usage/spider-multus-config.md
Auto-merging docs/usage/underlay_cni_service-zh_CN.md
error: could not apply a810abd2... Merge pull request #4196 from ty-dc/fix/ipam-image
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git cherry-pick --continue".
hint: You can instead skip this commit with "git cherry-pick --skip".
hint: To abort and get back to the state before "git cherry-pick",
hint: run "git cherry-pick --abort".
hint: Disable this message with "git config advice.mergeConflict false"

fix #4210


Fix broken link pointing to rekord_schema

This is a tiny change to fix the broken link which pointed to rekord_schema.md and not rekord_schema.json.

nice catch, thanks @bistaastha


XIRQ is firing continuously

Hello,
I am trying to use external Interrupts.
In my ProcessorTop-Template (neorv32_SystemTop_axi4lite) I changed the number of XIRQ-Channels to 4, I left trigger-type and polarity at 0xFFFFFFFF, which should result in interrupts on a rising edge. I connected the 4 input channels to 4 high-active buttons.
In the firmware I used
neorv32_rte_setup();
neorv32_xirq_setup();
neorv32_xirq_install(0, xirq_handler);
neorv32_xirq_global_enable();

and
neorv32_cpu_eint();

to setup and enable interrupts
Whenever I push a button, the corresponding handler gets executed continuously. I tried to clear the pending interrupt inside the handler, but that doesn't change  anything.
I also tried the XIRQ-Demo on my and a another minimal design (neorv32_ProcessorTop_MinimalBoot), where I connected gpio_o(31 downto 0) to xirq_i with the same result, the interrupt handler is firing continuously:
...
XIRQ interrupt from channel 3<\r><\n>
XIRQ interrupt from channel 3<\r><\n>
XIRQ interrupt from channel 3<\r><\n>
XIRQ interrupt from channel 3<\r><\n>
XIRQ interrupt from channel 3<\r><\n>
...

I have an older design (v<IP_ADDRESS>) here, and there it works as expected...

Hi,

Whenever I push a button

Just to ensure, you debounce the button?

Hi Stephan, i checked on my design. Same for me the NEORV32 craches with an RTE:
# <RTE> Illegal instruction @ PC=0x80000000, INST=0x0042 </RTE>
# <RTE> Illegal instruction @ PC=0x80000004, INST=0x0000 </RTE>

@AWenzel83: If you like to double check, add neorv32_rte_setup(); to your c program and dump from the UART.
BR,
Andeas

One additional hint from my site, i use riscv32-unknown-elf.gcc-10.2.0.rv32i.ilp32.newlib.tar.gz as compiler.

@AWenzel83
Hey there!

trigger-type and polarity at 0xFFFFFFFF, which should result in interrupts on a rising edge. I connected the 4 input channels to 4 high-active buttons.

I have the same setup here but I cannot reproduce your issue. The external interrupts fire as expected. Are you using the latest version of the core? Could you share the code for your xirq_handler function?

@akaeba

i checked on my design. Same for me the NEORV32 craches with an RTE:

Oh, that looks bad... But I think this might be caused by a compiler issue. How did you compile your executable? Could you also share the code that causes this exception?

i use riscv32-unknown-elf.gcc-10.2.0.rv32i.ilp32.newlib.tar.gz as compiler

This version is outdated, but it should still work... However, could you try updating to a newer version? For example this one: https://github.com/stnolting/riscv-gcc-prebuilt/releases/tag/rv32i-4.0.0

Hi Stephan,

This version is outdated, but it should still work... However, could you try updating to a newer version? For example this one: https://github.com/stnolting/riscv-gcc-prebuilt/releases/tag/rv32i-4.0.0

With version v4.0.0 is all fine :) Final question for me: has my MIF convertor a bug, or depends it on the GCC. But now for me issue solved. Thanks for supporting :)

With version v4.0.0 is all fine :)

Great to hear!

Final question for me: has my MIF convertor a bug, or depends it on the GCC

I'm not sure... But this

# <RTE> Illegal instruction @ PC=0x80000000, INST=0x0042 </RTE>

looks like a broken executable. The data memory starts at address 0x80000000 and the CPU should not try executing code from there as this is where the .data segment is located. 🤔

Sorry for the late reply, I haven't been in the office in the last week.

If you like to double check, add neorv32_rte_setup(); to your c program and dump from the UART.

I had that in the original version, posted above. Its not dumping anything to the UART.

Just to ensure, you debounce the button?

Yes, I do. But the problem also exists in the XIRQ-Example, where the interrupts are triggered in software.

Are you using the latest version of the core?

not quite, I'm on Version <IP_ADDRESS>. Since I didn't want to change my code for the updated TWI- and SPI-Interfaces, but I will try the newest version, next.

Could you share the code for your xirq_handler function?

sure, its pretty simple:
void xirq_handler(void)
{
  neorv32_uart0_printf("Interrupt\n");
  NEORV32_GPIO.OUTPUT_LO ^= 0x400;
}

Do you have a running simulation from your core configuration? Especially the cpu_irq_o from https://github.com/stnolting/neorv32/blob/278ab39682ead6c76164f426e4838bd33be11cc8/rtl/core/neorv32_xirq.vhd would be interesting.

I haven't, but I will look into that.

With version v4.0.0 is all fine :)

I tried that one, too. I used my own toolchain built with crosstool-ng, before. Unfortunately that didn't resolve my problem.

Sorry for the late reply, I haven't been in the office in the last week.

No problem!

I'm on Version <IP_ADDRESS>. Since I didn't want to change my code for the updated TWI- and SPI-Interfaces, but I will try the newest version, next.

The modifications of the SPI and TWI modules should be "quite" backwards-compatible (with some minor exceptions)... 😅

I have setup a minimal program to check the XIRQ again:
#include <neorv32.h>

#define BAUD_RATE 19200

void xirq_handler(void)
{
  neorv32_uart0_printf("Interrupt\n");
  NEORV32_GPIO.OUTPUT_LO ^= 0x400;
}

int main() {

  neorv32_rte_setup();

  neorv32_uart0_setup(BAUD_RATE, PARITY_NONE, FLOW_CONTROL_NONE);

  neorv32_gpio_port_set(0);

  neorv32_xirq_setup();
  neorv32_xirq_install(0, xirq_handler);
  neorv32_xirq_install(1, xirq_handler);
  neorv32_xirq_install(2, xirq_handler);
  neorv32_xirq_install(3, xirq_handler);
  neorv32_xirq_global_enable();

  neorv32_cpu_eint();

  while(1);

  return 0;
}

Compiled with make MARCH=rv32i clean_all exe and executed on my default setup using the following XIRQ configs:
    -- External Interrupts Controller (XIRQ) --
    XIRQ_NUM_CH                  => 4,
    XIRQ_TRIGGER_TYPE            => (others => '1'),
    XIRQ_TRIGGER_POLARITY        => (others => '1'),

I have connected 4 high-active push buttons (no debouncing) and everything works as expected. The interrupts fire once (or twice, when pushed "too slow" due to bouncing...) when I hit any of the buttons:
Awaiting neorv32_exe.bin... OK
CMD:> e
Booting from 0x00000000...

I tried again, this time with the current release Version 1.7.8 and what should I say, the exact same design works like expected...
I don't really know what changed between these Versions, I guess I broke something on my side...
Thank you for the help


command and instructions to inject sourcemaps

@keybase/react-hackers

:+1: How would we do this on linux?

We'd have to make the command find where your package is. This is just for our own debugging so we'd have to see if it's necessary. Likely not u less it's a Linux specific js bug


[CLI]: Offline run folders appear without required information and can't be synced

Describe the bug

Offline runs don't always log the information needed for them to be synced, so the information they generate is effectively lost. This happens for around 50% of the offline runs on my cluster, so it's a large issue. So far, I haven't found any factor that correlates with a run working or not. When I run the same code but as an online run, everything works as expected.
More precisely, when the bug occurs, the offline-run-... folder created in the wandb directory is missing files—see in additional files for details. When I try to log it with wandb sync, on one such folder, I get
(py310IMLESSL) [tme3@narval3 wandb]$ wandb sync offline-run-20230517_061457-bz3nyzr9
Find logs at: /tmp/debug-cli.tme3.log
Skipping directory: /lustre06/project/6061877/tme3/IMLE-SSL/wandb/offline-run-20230517_061457-bz3nyzr9

The relevant logs are apparently empty. I also don't find mention of the run in debug-cli.tme3.log, while the UIDs of runs that did work correctly do appear there.
I would very much appreciate help solving this! Please let me know if there is other information that would be useful.
Additional Files
To fully explain what I can see of the issue, here are two runs with identical arguments. I've attached files giving the SLURM submission script, job output file, and zipped run directory respectly, first for a run that did work, and next for one that didn't. Note that the submission scripts and output files are effectively the same—up to compute node, time, and wandb UID details.
--- Job that did work correctly ---
good_job_results.txt
good_job_submission_script.txt
offline-run-20230517_070615-a25xgj3m.zip
--- Job that didn't work correctly ---
bad_job_submission_script.txt
bad_job_results.txt
offline-run-20230517_061457-bz3nyzr9.zip
The major thing to note about the offline run directory that was created was that it contained only the images I wanted to log, and none of the other files—ie. its top-level should contain the following five entries—files  logs  run-a25xgj3m.wandb  tmp  wandb—but instead looks has just one—files.
(Also: due to space constraints from GitHub, I had to unzip and delete some of the images so the zip files would fit.)
Environment
WandB version: 0.15.1
OS: Linux with Lustre filesystem—probably modified since it's the Narval cluster of ComputeCanada
Python version: 3.10.11
Versions of relevant libraries:
Additional Context
No response

Up till cancelling, you'd see the same outputs in all offline runs, regardless of if they could be synced. In this case, after seeing the job had been running for a while without the folder being created correctly, I cancelled it to avoid wasting compute.

Hi @tristanengst thanks for the additional information, could you please share a minimal example of your Slurm job script file? are these runs part of Sweep and if so are you submitting jobs in a preemptible queue?

Hi! I've abbreviated and attached one of the SLURM submission scripts from my comment above, and I've replicated it below (or am I misinterpreting and you're asking for a different file?). These aren't part of sweeps and the cluster itself doesn't have preemtion.
The key is in the quotation marks; the first apparent invocation of python is just echoing the script and its arguments to the job result file. Maybe it's somehow being executed?
#!/bin/bash

# The SBATCH directives must appear before any executable line in this script.

#SBATCH --time=3:00:00
#SBATCH --account=rrg-keli
#SBATCH --mem=100GB
#SBATCH --nodes=1
#SBATCH --cpus-per-task=12
#SBATCH --gres=gpu:a100:1

#SBATCH --job-name=imle-vit_base_v0-imagenet-bs32-epochs50-ipe20-lr1e-8_ramp_1e-6_ramp_5e-7_1e-7-ns128-nshotall-nway1-seed0-bz3nyzr9-May15-FinetuneLR-GPUS1-TIME3:00:00
#SBATCH --output=job_results/%x_%a.txt

# Below sets the email notification, swap to your email to receive notifications
#SBATCH<EMAIL_ADDRESS>#SBATCH --mail-type=BEGIN
#SBATCH --mail-type=END
#SBATCH --mail-type=FAIL
#SBATCH --mail-type=REQUEUE
#SBATCH --mail-type=ALL

export OMP_NUM_THREADS=1
# Print some info for context.
pwd
hostname
date

echo "Starting job number $SLURM_JOB_ID"
source ~/.bashrc
conda activate py310IMLESSL
export PYTHONUNBUFFERED=1

echo "echo 'start: '`date`
mkdir -p $SLURM_TMPDIR/imagenet
echo 'Extracting /home/tme3/scratch/Data/imagenet/train.tar...'
tar -xf /home/tme3/scratch/Data/imagenet/train.tar -C $SLURM_TMPDIR/imagenet
echo 'done: '`date`
mkdir -p $SLURM_TMPDIR/imagenet
echo 'Extracting /home/tme3/projects/rrg-keli/data/imagenet/val.tar...'
tar -xf /home/tme3/projects/rrg-keli/data/imagenet/val.tar -C $SLURM_TMPDIR/imagenet
echo 'done: '`date`

python TrainSSL.py  --wandb offline ... other arguments"

echo 'start: '`date`
mkdir -p $SLURM_TMPDIR/imagenet
echo 'Extracting /home/tme3/scratch/Data/imagenet/train.tar...'
tar -xf /home/tme3/scratch/Data/imagenet/train.tar -C $SLURM_TMPDIR/imagenet
echo 'done: '`date`
mkdir -p $SLURM_TMPDIR/imagenet
echo 'Extracting /home/tme3/projects/rrg-keli/data/imagenet/val.tar...'
tar -xf /home/tme3/projects/rrg-keli/data/imagenet/val.tar -C $SLURM_TMPDIR/imagenet
echo 'done: '`date`

python TrainSSL.py  --wandb offline ... other arguments

# Print completion time.
date

Hi @tristanengst thanks so much for sharing the SLURM job script. I was wondering if that issue occurs for you only when you run two consecutive experiments in the way you provided above. Does it happen if you only had a single run? If not, could you please make sure that you add in the end of your TrainSSL.py script a wandb.finish() call?

Also, would it be possible in the meantime to spoof a correct run by creating the necessarily files manually? Maybe it'd be too much effort?

Hi @tristanengst thanks for the detailed additional information. Could you please pip install wandb --upgrade to our most recent version, and check if the issue still occurs there?
Do you pass any settings in your wandb.init call, could you please share here a code snippet? Also, can you try to disable the symlinks as follows, would this help? wandb.init(settings=wandb.Settings(symlink=False))

Here's my wandb.init(), with symlink=False as suggested. args is an argparse Namespace:
wandb.init(anonymous="allow", id=args.uid, config=args,
        mode=args.wandb, project="3MRL", entity="apex-lab",
        name=os.path.basename(model_folder(args)),
        resume="allow" if args.continue_run else "never",
        settings=wandb.Settings(code_dir=os.path.dirname(__file__), symlink=False))

With conda-installed wandb=0.15.1, this appears to fix the issue! I will test the newer WandB version soon, though I suspect I will have to install it via conda, as pip packages are in my experience less likely to work on the cluster in question.

Hi @tristanengst thanks a lot for the confirmation, glad to hear this is now resolved for you! Just for your information, we have released a new version v0.15.4, and you could pip or conda upgrade to it. I will close this ticket for now, as it seems resolved, but please feel free to reopen it if you experience any issue after upgrade and we will be happy to keep investigating.

Unfortunately it happened again; I'm continuing to investigate with wandb=0.15.4.

Sorry to hear this issue occurred again @tristanengst please let me know if that happens with the most recent version too, and I will report/escalate to our SDK team.

Hi @tristanengst just checking in here to see if you have seen the same issue again after upgrading to v0.15.4? Thank you!

So far I have two successes and zero failures with v0.15.4. I'll follow up if there are subsequent failures as I'll be using the cluster fairly heavily this week.

Perfect, glad to hear that so far seems to work fine. Please let us know later this week if everything went fine.

Hi @tristanengst just checking in here to see if the issue popped up again? thanks!

Hi @tristanengst since we haven't heard back from you and last time the issue seemed resolved, I will go ahead and close this ticket for now. If you're still experiencing any issue please let us know and we will be happy to keep investigating.


Specify in CONTRIBUTING.MD the need to add new datasets to doctree and Catalog JSON schema

Description
To allow for both contributions, I think we should specify the following in Contributing.md:

Contributors need to add their dataset to the the doctree: https://github.com/quantumblacklabs/kedro/blob/master/docs/source/15_api_docs/kedro.extras.datasets.rst
Contributors need to add their dataset to catalog json schema: https://github.com/quantumblacklabs/kedro/blob/master/static/jsonschema/kedro-catalog-0.17.json

The idea for this stems from the fact that I have often observed missing datasets in either one.
On top of this, the JSON schema needs to brought up-to-date. Multiple datasets are missing and in my opinion, the datasets here should be alphabetically ordered like in the doctree.
Implementation
If no one opposes to my ideas, I would:

Add necessary instructions in contributing.md
Reorder JSON schema.
Add missing datasets to JSON schema.

Hi @lucasjamar thank you for your suggestion. Our contribution guide is deliberately more high-level than that, as we don't want to be prescriptive about every single thing that needs to change for a given bugfix/feature. Every PR is reviewed and approved by 2 people and missing files should be caught at that stage.
As for the catalog schema however, it is indeed something that needs a bit more love - I believe there's already a ticket on our backlog for this, but if you (or anyone else) has some time to pick it up, by all means please do, as it's unfortunately a low priority for us at the moment.

Hi @lorenabalan ,
Thanks for the input. This is a good point.
In which case I will leave the contributing.md as is.
As for the JSON schema, I will reorder the datasets and add anything missing. I will do this for Kedro 0.18 only based on the dev branch :)

Excellent, thanks a lot! ❤


Add processor group to process aggregated data from CSV to JSON.

When ingesting data sources as per cQube Edu that require

no metrics to be processed [Done]
filters as required in the visualization [To Do]

This issue is not covered in v5.0 and needs to be analysed.

It is implemented


TodoMVC style example?

Is there an end-to-end example that showcases skatejs? Something along the lines of todomvc? If not, would you accept a pull request?
I'm evaluating frameworks that would help me implement my relatively small web app following the principles of web components. Polymer is just too big for this particular use case. I would like to give skatejs a try. The documentation and the talk at SydJS are pretty detailed, but it would be best to look at an end-to-end example to figure out what it involves.

My question might not be relevant to the topic of this thread, but I wonder how it handles the IE8/9 users as I know Atlasssian has used this library in production. Thanks.

@shenrunfu We support IE9 but no longer support IE8. If you want IE8 support you need to polyfill mutation events yourself (it's doable) but I can't guarantee that Skate's code will run in IE8's non ES5 compat engine.

We've submitted an issue to the TodoMVC repo for adding Skate.
https://github.com/tastejs/todomvc/issues/1306

As an FYI for anybody looking here, we said that a PR was welcome :) https://github.com/tastejs/todomvc/issues/1306#issuecomment-105055798

Forked TodoMVC. Will start work in about a week. cc @alexreardon

TodoMVC contribution guidelines:
https://github.com/tastejs/todomvc/blob/master/contributing.md
Base app template
https://github.com/tastejs/todomvc-app-template

Branch created with our work on this one:
https://github.com/skatejs/todomvc

Forgot to say: feel free to ping me if you need any help or have any questions.

This is making good progress. The application itself is finished, just fleshing out the tests

Skate has undergone some significant changes in the last 1-2 weeks. It would be worth reevaluating our current implementation and whether it could be done differently in light of the changes. /cc @treshugart

It definitely could be done simpler now. I have some unpublished changes that I'm working on against master, at the moment. I say we don't PR back to TodoMVC until 0.14 or 1.0.

For posterity:

https://github.com/skatejs/todomvc/tree/skatejs/examples/skatejs
https://github.com/skatejs/todomvc/tree/skatejs/examples/skatejs-dom-diff

They've both been called out in the README and we'll hopefully be submitting them once we get a chance to run them through the automated tests and fix any outstanding items.
I'll close this now as those examples are pretty close to what they should be and are now browsable by anyone.
Let me know if you have any questions.


fix: increase UT memory limit

#minor

Coverage remained the same at 51.141% when pulling a34de6d7b660dc762b529824690fb6eb03bfb566 on donglei/increase-ut-memory into 683520a7c511fceace08ae15e2f438aab53d688d on main.


fix bug in readme example code

Coverage remained the same at 100.0% when pulling d80d205bef16ce2869007b4093abe802260f9aa4 on jackeyGao:master into a6531fd6a23c3b9f380cea185410e4dd466f59a8 on teambition:master.


Improving tab navigation

On the month edges - instead of leaving the picker and going to the next natural tab index, jump back to the first day of the month (or the last when shift tabbing).

Good finding, adapted the code to use self.days.


Implementation of Class Balancing

Did you implement the median class balancing technique as said in the paper?

No, mean was used. It is trivial to edit it in. You may create a PR too. :)
The function definition corresponding to the class probabilities can be found here.


[FLINK-28985][planner]sql like statement supports views

What is the purpose of the change
At present, to create a table based on table like, you can only use the table type table, not the view type.

create table <tableName> like <viewName>;

Only like table type can be used before. This is similar to create table as < querysql >, but some scenarios use views more flexibly and can reuse a single view in multiple places.

Brief change log

create table like statement support create table  like

Verifying this change
test function
This change is already covered by existing tests, such as org.apache.flink.table.planner.operations.SqlToOperationConverterTest#testCreateTableLikeViewWithFullPath.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): ( no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no )
The runtime per-record code paths (performance sensitive): (no )
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: ( no)
The S3 file system connector: (no )

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable )

Hello, I hope you can review the code when you are free. Thank you very much!  @gyfora @mbalassi @JingsongLi

@MartijnVisser


Refactor some stuff

Resolves #65
Resolves #63
Resolves #62
Resolves #60

Setting this to a draft to get https://github.com/TheBevyFlock/bevy-template/pull/79 merged first.


Hide window thumbnail title bar

Description of the new feature / enhancement
I would also like to hide or have a minimized dropdown window title bar so that it doesn't say "[Window Title] - Cropped".
Scenario when this would be used?
I like to crop the song title and artist of spotify so that I can have it as an overlay (I can't use game bar on my work computer to achieve this). It would just look cleaner without that pop-out style title bar.
Supporting information
No response

/dup #28336


Add an account field for account types

We need a way of setting whether an account is a developer account or not so that we can top up developer accounts with credits each month.

The current plan is to have an account field like account_type and allow it to be one of a list of possible string values. 'normal' and 'developer' might work as the possible field values for now, though 'developer' might not be the right word to use, since that should cover cases like internal, hackathon and entrepeneur accounts. Thoughts on a better name?

Depends on #1132, otherwise ready for review.

Hmm, wondering if this field should've been added to the billing account model instead of the user account model. For the reason this is being added, adding it to the billing account model might make more sense. Having an account type field does sound like something general enough that we might need it for other reasons later though.

Ready for review.

:+1:

Closing this in favour of #1137


[api] Fixes toDebugString() IllegalStateException if NDArray is closed

Description
Brief description of what this PR is about

If this change is a backward incompatible change, why must this change be made?
Interesting edge cases to note here

Codecov Report
Base: 72.08% // Head: 74.31% // Increases project coverage by +2.22% :tada:

Coverage data is based on head (5677f11) compared to base (bb5073f).
Patch coverage: 74.95% of modified lines in pull request are covered.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #2347      +/-   ##
============================================
+ Coverage     72.08%   74.31%   +2.22%
- Complexity     5126     6749    +1623
============================================
  Files           473      662     +189
  Lines         21970    29319    +7349
  Branches       2351     3038     +687
============================================
+ Hits          15838    21787    +5949
- Misses         4925     6048    +1123
- Partials       1207     1484     +277

Impacted Files
Coverage Δ

api/src/main/java/ai/djl/modality/cv/Image.java
69.23% <ø> (-4.11%)
:arrow_down:

...rc/main/java/ai/djl/modality/cv/MultiBoxPrior.java
76.00% <ø> (ø)

...rc/main/java/ai/djl/modality/cv/output/Joints.java
71.42% <ø> (ø)

.../main/java/ai/djl/modality/cv/output/Landmark.java
100.00% <ø> (ø)

...main/java/ai/djl/modality/cv/output/Rectangle.java
72.41% <0.00%> (ø)

...i/djl/modality/cv/translator/BigGANTranslator.java
21.42% <0.00%> (-5.24%)
:arrow_down:

.../modality/cv/translator/ImageFeatureExtractor.java
0.00% <0.00%> (ø)

.../ai/djl/modality/cv/translator/YoloTranslator.java
27.77% <0.00%> (+18.95%)
:arrow_up:

...ain/java/ai/djl/modality/cv/util/NDImageUtils.java
67.10% <0.00%> (+7.89%)
:arrow_up:

api/src/main/java/ai/djl/modality/nlp/Decoder.java
63.63% <ø> (ø)

... and 607 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Fix potential position embedding issue in decoders

Resolve #158

Thanks for the quick update. This is a critical fix that can potentially impact many modules examples. Can you add more unit tests (if applicable) to make sure it works as intended, and run relevant examples (e.g., seq2seq_attn, transformer) to see if the results can be reproduced?

Thanks for the quick update. This is a critical fix that can potentially impact many modules examples. Can you add more unit tests (if applicable) to make sure it works as intended, and run relevant examples (e.g., seq2seq_attn, transformer) to see if the results can be reproduced?

Tested seq2seq_attn and transformer on GPU. Results can be reproduced.

Thanks for the quick update. This is a critical fix that can potentially impact many modules examples. Can you add more unit tests (if applicable) to make sure it works as intended, and run relevant examples (e.g., seq2seq_attn, transformer) to see if the results can be reproduced?

Unit tests in transformer_decoders_test.py have already tested such behavior.

Wait... I understand the need to add back next_inputs, but why can't we provide a default implementation in the base class? It's still the same in every derived class, just as before.

I agree. Let's remove the duplicate code

I will remove the duplicate code.