two instance affect each other

When using two instances of pcm-memory one affects the other , then I know that someone else on the machine also used this tool in parallel and you have to filter out extreme large values from the result :) The results you get temporarily for some counters are close to max(int64) or -max(int64) .

Hi, running several pcm instances is not supported. But we should probably have a mechanism to detect the other instance and fail to start if the other instance is running.


Make subscribe for TopicsConsumer using builder

Following PIP-12, we should turn to use builder for creating TopicsConsumer.

PR #1269 is for this


"Save Changes" button not appearing

The "Save Changes" button does not show up for some users on version >= 0.04a
I am unable to reproduce this issue locally.
Screenshot provided by affected user: https://i.imgur.com/j1KmnXQ.jpg

@MerijnHendriks I would appreciate if you could take a look at this. The issue seems to have originated from the migration to .NET 5.0

@MerijnHendriks I would appreciate if you could take a look at this. The issue seems to have originated from the migration to .NET 5.0

yeah got the same problem too

yeah got the same problem too


[RFC][language][vm] add Container::Locals

An attempt to remodel Locals. Not intended to be checked in.

:umbrella: The latest upstream changes (presumably cdd1c933cb8378620d422006b84346e53b223ef2) made this pull request unmergeable. Please resolve the merge conflicts.


[chat]: update rm, add wandb and fix bugs

📌 Checklist before creating the PR

[x] I have created an issue for this PR for traceability
[x] The title follows the standard format: [doc/gemini/tensor/...]: A concise description
[ ] I have added relevant tags if possible for us to better distinguish different PRs

🚨 Issue number

Link this PR to your issue with words like fixed to automatically close the linked issue upon merge
e.g. fixed #1234, closed #1234, resolved #1234

Fixed #4125 (partially), close #4681, close #4749, close #4698, close #4685.
📝 What does this PR do?

Summarize your work here.
if you have any plots/diagrams/screenshots/tables, please attach them here.

[x] Update forward() function of reward model and critic model.
[x] Add wandb in train_sft, train_rm and train_prompts
[x] Fix wrong IGNORE_INDEX
[x] Check padding_size in generation
[x] Use very small learning rate and bfloat16 in integrated tests to make them more stable
[x] Update gemini plugin

💥 Checklist before requesting a review

[x] I have linked my PR to an issue (instruction)
[ ] My issue clearly describes the problem/feature/proposal, with diagrams/charts/table/code if possible
[x] I have performed a self-review of my code
[x] I have added thorough tests.
[ ] I have added docstrings for all the functions/methods I implemented

⭐ Do you enjoy contributing to Colossal-AI?

[ ] 🌝 Yes, I do.
[ ] 🌚 No, I don't.

Tell us more if you don't enjoy contributing to Colossal-AI.

thanks for fixing tensorboard issue!


Remove kind/resource type mentions from metadata.name in resources

The issue resolved by this Pull Request:
Resolves RHOAIENG-1680
Description of your changes:
This PR removes kind/resource type mentions from metadata.name in resources
Testing instructions
Just normal Sanity checks
Checklist

[X] The commits are squashed in a cohesive manner and have meaningful messages.
[X] Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
[X] The developer has manually tested the changes and verified that the changes work

A new image has been built to help with testing out this PR: quay.io/opendatahub/data-science-pipelines-operator:pr-557
An OCP cluster where you are logged in as cluster admin is required.
To use this image run the following:
cd $(mktemp -d)
git clone git@github.com:opendatahub-io/data-science-pipelines-operator.git
cd data-science-pipelines-operator/
git fetch origin pull/557/head
git checkout -b pullrequest 11d6114a7fa26b92f480e3d4536b90e25cbcf1da
make deploy IMG="quay.io/opendatahub/data-science-pipelines-operator:pr-557"

More instructions here on how to deploy and test a Data Science Pipelines Application.

/retest

Aren't there RoleBindings that would need to be changed as well? (edit: or in this case, ClusterRoleBindings...)

I couldn't find any ClusterRoleBindings in this case. Do you have a reference point if any?

Nope, I'm not very familiar with them.
@gmfrasca do you know if there are ClusterRoleBindings somewhere that would also need to be changed?

Nope, I'm not very familiar with them.
@gmfrasca do you know if there are ClusterRoleBindings somewhere that would also need to be changed?

If you look at the commit history you can find CRBs in this commit:
https://github.com/opendatahub-io/data-science-pipelines-operator/commit/2fec33481ad87da7de6aee6a5ba56125e31d75ec
And not in the next one: https://github.com/opendatahub-io/data-science-pipelines-operator/commit/027e8abfc98c41acfc1f284a2c3a1caa396baa44
Not sure if they were moved to deleted.

Change to PR detected. A new PR build was completed.
A new image has been built to help with testing out this PR: quay.io/opendatahub/data-science-pipelines-operator:pr-557


Add Solutions for LeetCode Problems 21-30

📥 Pull Request
Description
This pull request adds solutions for LeetCode problems 21 to 30, enhancing our repository's collection of coding challenges. These solutions include various data structures and algorithms, providing valuable examples for users looking to improve their problem-solving skills.
Closes #297
Screenshot

Type of change

[x] Documentation update

Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my code
[x] I have commented my code, particularly in hard-to-understand areas
[x] Any dependent changes have been merged and published in downstream modules

@ajay-dhangar please review my pr soon.


ability to automatically choose interfaces by internal node locality

Is your feature request related to a problem? Please describe.
Some HPC systems have multiple network cards per node.  If all of them are visible, then Mercury will choose the first one advertised by OFI by default unless the user specifies an interface in the Mercury address string.  If all processes use the first interface then the network resources will be underutilized.  If they are selected explicitly then the Mercury user has the burden of trying to figure out which NIC would be most appropriate for each Mercury instance to use.
The Crusher system at the OLCF (https://docs.olcf.ornl.gov/systems/crusher_quick_start_guide.html) is a good example of a system were this could be important.  Each compute node has 4 Slingshot interfaces.  As shown in the node diagram on that page, the NICs are directly attached to GPUs rather than CPUs.  All of them are accessible indirectly from any CPU core, but you can minimize the number of internal bus hops by routing traffic to the NIC/GPU on the local NUMA domain.  Crusher supports a variety of process mappings; it would be nice if the optimal network interface were selected automatically regardless of where a process is launched.
Describe the solution you'd like
MPICH (https://github.com/pmodels/mpich/blob/main/src/mpid/ch4/netmod/ofi/ofi_nic.c#L174) and OpenMPI (https://github.com/open-mpi/ompi/blob/master/opal/mca/common/ofi/common_ofi.c#L596) both include logic that consults hwloc to see which, if any, out of a set of potential interfaces has the best locality from a given process.  We should integrate the same capability within Mercury, so that if multiple Mercury instances are executed on the same node without requesting an explicit interface, then they will naturally divide work among interfaces according to architecture locality.  hwloc is the most platform-agnostic solution.
Describe alternatives you've considered
Most of the alternatives would be platform-specific (for example, selecting NICs on crusher according to NUMA domain number) and thus difficult to maintain.  Mercury could also select interfaces randomly to increase utilization, but it would not gain the latency and bus utilization benefits of purposefully accounting for locality.
Additional context
Selecting interfaces based on locality will only produce an optimal solution if a process is pinned to a set of cores so that it won't migrate somewhere with different locality.  Processes are often pinned on HPC systems already for other reasons, though, and even if the process can hypothetically migrate, the initial locality is still a better guess than random or first-advertised.

Hi Phil, For a node with multiple interfaces, would you need to know two things? 1. Of all available interfaces, which can reach my desired destination? 2. Of the interfaces in item (1), which is “closest” to me (assuming I am a process pinned to specific cores)? Hwloc can help with item (2), but did Brice every add support for item (2) (i.e., netloc)? Scott

Hi @scottatchley ,
For the time being I'm personally just worried about 2), assuming that all of the NICs are on the same fully connected fabric for the hosts they might communicate with (this would be good enough for communication among Crusher compute nodes, for example).

would be particularly interesting for a router/forwarder/bridge use case, but to my knowledge there isn't anything integrated in Mercury to help manage that right now.  It would probably take help from something atop Mercury to sort out?

OFI provides a fi_nic() interface to pull NIC attributes, including PCI attributes like BDF. With this information from a fi_getinfo() query and hwloc topology info, one can find the NIC closest to the CPU a process is pinned to and open an endpoint with that NIC. This is what the OFI MTL in Open MPI does today.

Thanks @rajachan for pointing out the OFI MTL code, that was definitely helpful.


Bug: assertSee() can not assert title tag.

Describe the bug
assertSee() can not assert title tag.
$result->assertSee('<title>CodeIgniter Tutorial</title>'); -> Pass
$result->assertSee('CodeIgniter Tutorial', 'title'); -> Fail
CodeIgniter 4 version
4.0.4
Affected module(s)
Tests
Expected behavior, and steps to reproduce if appropriate
assertSee('CodeIgniter Tutorial', 'title'); should pass when the page has the title tag.
Context

OS: [macOS 10.15.7]
Web server [n/a]
PHP version [7.4.13]

I'm not familiar with DomXPath it uses under the hood, but it seems when the 1st arg of assertSee has tags, it considers the whole DOM. However, removing the tags, the code will search only within the <body/>. I dunno if that is a bug for you or is desirable.

I'm not familiar with DomXPath it uses under the hood, but it seems when the 1st arg of assertSee has tags, it considers the whole DOM. However, removing the tags, the code will search only within the <body/>. I dunno if that is a bug for you or is desirable.

I don't know why it searches only within the <body/>.

I don't know why it searches only within the <body/>.

The first example that passes uses simple mb_strpos() for search. But the second one that fails uses the DOMXPath class, which has buggy rules that force a search only inside the body tag.

The first example that passes uses simple mb_strpos() for search. But the second one that fails uses the DOMXPath class, which has buggy rules that force a search only inside the body tag.

@paulbalandan is right here.
When using assertSee, the see method of DOMParser is called. If no element selector is passed to assertSee, this searches the whole DOM. If an element is passed, the DOMParser->doXPath function is used for searching. In this function body implicitly always gets added to the search string.
I feel this is more a consistency issue than a bug. Taking the UG literally, it
 asserts that text/HTML is on the page

so I would think the <title> tag in the <head> should never be found. This is also what the PHPDoc states:
Assert that the desired text can be found in the result body.

However, DOMParser->see and doXPath do not explicitly state that only the body is searched, but rather the whole DOM.
My personal opinion:
I do not see, why doXPath should only search the DOM of body. Btw. doXPath even behaves differently when using different kinds of selectors (e.g., passing a tag with an id will only search the body, but passing an id without a tag will search the whole DOM).
Options that I see:
1 modify the DOMParser->see() method to always use doXPath
2 modify doXPath to search the whole DOM anytime and update the assertSee UG
3 modify DOMParser and doXPath by adding an additional optional parameter bool $onlyBody = false/true to enable switching

@paulbalandan is right here.
When using assertSee, the see method of DOMParser is called. If no element selector is passed to assertSee, this searches the whole DOM. If an element is passed, the DOMParser->doXPath function is used for searching. In this function body implicitly always gets added to the search string.
I feel this is more a consistency issue than a bug. Taking the UG literally, it
 asserts that text/HTML is on the page

so I would think the <title> tag in the <head> should never be found. This is also what the PHPDoc states:
Assert that the desired text can be found in the result body.

However, DOMParser->see and doXPath do not explicitly state that only the body is searched, but rather the whole DOM.
My personal opinion:
I do not see, why doXPath should only search the DOM of body. Btw. doXPath even behaves differently when using different kinds of selectors (e.g., passing a tag with an id will only search the body, but passing an id without a tag will search the whole DOM).
Options that I see:
1 modify the DOMParser->see() method to always use doXPath
2 modify doXPath to search the whole DOM anytime and update the assertSee UG
3 modify DOMParser and doXPath by adding an additional optional parameter bool $onlyBody = false/true to enable switching

The first example that passes uses simple mb_strpos() for search. But the second one that fails uses the DOMXPath class, which has buggy rules that force a search only inside the body tag.

:D three minutes too late.
Can you also update the assertSee PHPDoc? It now searches the whole result, not only the result body.

The first example that passes uses simple mb_strpos() for search. But the second one that fails uses the DOMXPath class, which has buggy rules that force a search only inside the body tag.

:D three minutes too late.
Can you also update the assertSee PHPDoc? It now searches the whole result, not only the result body.

😅
I'm not a linguistic expert but

Assert that the desired text can be found in the result body.

doesn't sound to me like the suggestion of a body tag... but I might be wrong. English is not my first language, so can I hear one more opinion on this one before I push a change?

😅
I'm not a linguistic expert but

Assert that the desired text can be found in the result body.

doesn't sound to me like the suggestion of a body tag... but I might be wrong. English is not my first language, so can I hear one more opinion on this one before I push a change?

😅
I'm not a linguistic expert but

Assert that the desired text can be found in the result body.

doesn't sound to me like the suggestion of a body tag... but I might be wrong. English is not my first language, so can I hear one more opinion on this one before I push a change?

No worries, neither am I. Just the way I read those docs. I'm in for other opinions :)

😅
I'm not a linguistic expert but

Assert that the desired text can be found in the result body.

doesn't sound to me like the suggestion of a body tag... but I might be wrong. English is not my first language, so can I hear one more opinion on this one before I push a change?

No worries, neither am I. Just the way I read those docs. I'm in for other opinions :)

My English is not good. :-)
The User Guide says:

You can perform tests to see if specific elements/text/etc exist with the body of the response with the following assertions.

I read it as the body of the response object, or the HTTP response body (not in the response header).
Not in the body tag.

My English is not good. :-)
The User Guide says:

You can perform tests to see if specific elements/text/etc exist with the body of the response with the following assertions.

I read it as the body of the response object, or the HTTP response body (not in the response header).
Not in the body tag.

Okay, just leave it then :-)

Okay, just leave it then :-)

Hi, do you mind share how you get $result?
thanks


Panel loads and appends every time.

If it is going to load each time it should at least replace rather append.
Ideally it would load once, then periodically update and or force update with a button.

Now it loads if it has been more than 10 minutes since last load.
This should be configurable but preferences and how to handle them hasn't been fleshed out.


[Bug]:  Dashboard Container Not Rendering Correctly

Is there an existing issue for this?

[X] I have searched the existing issues

What happened?
Description:
The dashboard container is not rendering correctly on certain browsers. The container is supposed to have a white background, rounded corners, and a box shadow, but it is not displaying as expected.
Steps to Reproduce:
Open the dashboard page in a browser.
Inspect the dashboard container element.
Notice that the container is not rendering correctly.
Expected Behavior:
The dashboard container should render with a white background, rounded corners, and a box shadow.
Actual Behavior:
The dashboard container is not rendering correctly. The background color is not white, and the corners are not rounded.
Add Screenshots
No response
What browsers are you seeing the problem on?
No response
Record

[X] I agree to follow this project's Code of Conduct
[X] I'm a GSSOC contributor
[X] I want to work on this issue
[X] I'm willing to provide further clarification or assistance if needed.

screenshot?


ImportError: cannot import name 'NewsApiClient' from 'newsapi

pip install newsapi-python
from newsapi import NewsApiClient
its suppost to allow me to all newsapiclient into other scripts but insted it says
line 4, in
from newsapi import NewsApiClient
ImportError: cannot import name 'NewsApiClient' from 'newsapi' (C:\Users"my name"\AppData\Local\Programs\Python\Python311\Lib\site-packages\newsapi_init_.py

https://stackoverflow.com/a/67465185/12494235


Updates OSRS Archipelago Plugin to v1.2

Archipelago Client Plugin is now on Maven! No more direct vendoring of plugin code!
Adds new checkbox for toggling AP messages going to OSRS chatbox
AP Server Password is now an actual password field, using placeholders instead
Replaces hard-coded tasks with data parsed from github, using version information from Archipelago slot data. This should allow for logic changes that don't require a whole new plugin version to use.
Adds in a local DataPackage storage, which will store the Slot's connected player and last received item index. This will make reconnection smoother as well as allowing you to see popups for items received while offline.

It seems like you forgot to include your plugin changes

I did indeed. Updated now.

you haven't added org.java-websocket:Java-WebSocket:1.5.2 to verification-metadata.xml - we've already approved version 1.5.6 for the wikisync plugin, so I'd recommend updating your version to that

Since the automated tests are passing the dependency checks, is there anything else remaining on this before it can get reviewed?

You are writing files outside of .runelite, which is disallowed. You are also using java's built-in object serialization, which we also disallow.

You are writing files outside of .runelite, which is disallowed. You are also using java's built-in object serialization, which we also disallow.

There are no files being written to outside of .runelite. The generated .save file is in .runelite/APData/.
I will remove the Java object serialization and write my own serializer and parser.

Serialization has been removed. If I could get clarification on where I'm writing a file outside of .runelite I could fix that

You should use @Inject to populate the gson instance instead of doing new GsonBuilder()

Donezo. Build successful this time, thanks.

You are writing to the cwd in dev.koifysh.archipelago.Client::loadDataPackage which you call (and still uses java serialization). You also are pinning a bunch of deps that I think you don't use, so don't do that.

That's in an external dependency, would it be sufficient to just override the functions that call those so they aren't used? The pinned dependencies have been found through trial-and-error to be used. The plugin won't build without them.

One of the methods is private, so you can't just override it. I would prefer if they were gone entirely or had to be called manually. Even if you could override it, verifying that you always use the overridden one puts too much burden on reviewers handling your plugin in the future.
Wrt your deps, I deleted half of them and it still built. The error you were seeing earlier is because you didn't have the same excludes in verification-template's build as your plugin's, so it was failing to build because you are pulling more deps there than in your plugin.

So, at this point I'd need to just embed the source of the plugin instead of using the one on maven, so I could rewrite these methods? Or ask the maintainer of that plugin to add in some way to disable them?

Yes

Okay, it's been added to the project, and modified to remove all Gsons, Serializables, and writing to improper directories. While I was in there, since the source is now embedded, I was able to excise the dependency on Apache httpcomponents, which means that now the only dependency change is removing one that the original build of the plugin already had.
Hopefully that'll make the process much simpler for review.


Does uv provide package handling for multiple code locations?

I would like to switch to uv, however Im running dagster that have multiple code locations with some common dependencies but some are only there for one project some only the other, they are now listed in 2 req.txt files:
Location A:
dagster==1.7
dagster-dbt==0.23
pandas==2.1.4

Location B:
dagster==1.7
dagster-dbt==0.23
pyarrow==16.0.0

Is it possible to set this with only one pyproject.toml file?
Something like this:
Common:
dagster==1.7
dagster-dbt==0.23

Location A:
pandas==2.1.4

Location B:
pyarrow==16.0.0

Im open to suggestions if this looks completly wrong.
Also i dont necessarily want to have multiple .python_version files as python is always the same for all locations

By code locations, do you mean indexes, like PyPI?

No, sorry. This is a dagster concept, basically they are just different folders, with seperate requirements as seen above but still in one project

Sounds like https://docs.astral.sh/uv/concepts/workspaces/

We don't let you define the common dependencies in the workspace root yet though

I looked at this, but we have conflicting dependencies, so i see that i should use path dependencies. However its not super clear for me how to use it based on the example in the documentation. Do you have any more examples?


Fix hostname resolution for HA in k8s

Fix https://github.com/Alluxio/alluxio/issues/9448 by delaying hostname resolution so that stale DNS entries can be refreshed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/4553/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/4561/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/4898/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/4909/
Test FAILed.

Tests in error:
alluxio.master.ZkMasterInquireClientTest.testMultipleLeaders(alluxio.master.ZkMasterInquireClientTest)
  Run 1: ZkMasterInquireClientTest.testMultipleLeaders:127 expected:</<IP_ADDRESS>:54321> but was:<<IP_ADDRESS>:54321>
  Run 2: ZkMasterInquireClientTest.testMultipleLeaders:127 » Unavailable Failed to dete...

alluxio.master.ZkMasterInquireClientTest.testSingleLeader(alluxio.master.ZkMasterInquireClientTest)
  Run 1: ZkMasterInquireClientTest.testSingleLeader:107 expected:</<IP_ADDRESS>:12345> but was:<<IP_ADDRESS>:12345>
  Run 2: ZkMasterInquireClientTest.testSingleLeader:107 » Unavailable Failed to determi...

Errors could be due to not resolving master addresses as it looks one of the mismatching addresses is not resolved. (it's missing /)

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/4923/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/4927/
Test FAILed.

Tests run: 3, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 249.017 sec <<< FAILURE! - in alluxio.multi.process.MultiProcessClusterTest
simpleCluster(alluxio.multi.process.MultiProcessClusterTest)  Time elapsed: 120.691 sec  <<< ERROR!
java.util.concurrent.TimeoutException: Timed out waiting for a primary master to be serving options: WaitForOptions{interval=20, timeout=5000}
	at alluxio.multi.process.MultiProcessClusterTest.clusterVerification(MultiProcessClusterTest.java:62)
	at alluxio.multi.process.MultiProcessClusterTest.simpleCluster(MultiProcessClusterTest.java:43)

simpleCluster(alluxio.multi.process.MultiProcessClusterTest)  Time elapsed: 120.612 sec  <<< ERROR!
java.util.concurrent.TimeoutException: Timed out waiting for a primary master to be serving options: WaitForOptions{interval=20, timeout=5000}
	at alluxio.multi.process.MultiProcessClusterTest.clusterVerification(MultiProcessClusterTest.java:62)
	at alluxio.multi.process.MultiProcessClusterTest.simpleCluster(MultiProcessClusterTest.java:43)

jenkins, test this please.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/4931/
Test FAILed.

jenkins, test this please.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/4934/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/4971/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/4972/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/4973/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/4974/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/4975/
Test FAILed.

jenkins, test this please.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/4976/
Test FAILed.

jenkins, test this please.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/4984/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/4986/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/4987/
Test FAILed.

jenkins, test this please.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/4989/
Test FAILed.

jenkins, test this please.

Tests in error:
alluxio.client.cli.fs.JobServiceFaultToleranceShellTest.distributedMv(alluxio.client.cli.fs.JobServiceFaultToleranceShellTest)
  Run 1: JobServiceFaultToleranceShellTest.before:45 » IO java.lang.RuntimeException: a...
  Run 2: JobServiceFaultToleranceShellTest.before:45 » IO java.lang.RuntimeException: a...

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/4990/
Test PASSed.

@madanadit Now there is a hidden requirement on many places for Inet addresses to be unresolved. Fundamentally it's gRPC's fault to not try to resolve/reresolve when using a certain constructor. Why can't we just leave everything as is and pass gRPC unresolved address to avoid the issue?

@ggezer there are 2 levels of resolution. InetSocketAddress resolves the address based on the constructor and then grpc's netty channel resolves the address based on the constructor. the requirement is that all addresses used by the client be unresolved. the test cases need to change as they expected a resolved address before

Why do we need to make all client addresses unresolved? My concern is that it's a requirement which is not easy to document. If using a proper channel builder constructor is enough, I'd stick with that. With the current state of the code, I don't know what will happen if we pass IP addresses instead of hostnames to embedded journal configuration, so we'll need some validation there.

ip addresses will be treated the same as hostnames and resolve to the same ip

Will this cause never tracing IP addresses for clients?

@ggezer unless grpc prints out the client ip, we would have to explicit debug ip at time of channel creation (possibly stale afterwards)

@calvinjia or @ggezer did you have more comments?

@madanadit I would be content with just ignoring resolved addresses at gRPC site.
Have no more comments otherwise.

@ggezer I feel ignoring just before creating the gRPC channel feels like a hack. The bottom line is we should never cache the resolved IP in InetSocketAddress on any client. Can you be more explicit with some of your concerns? Thx

alluxio-bot, merge this please.

alluxio-bot, cherry-pick this to branch-2.0 please.


add logic for adding flavors to skip when flavors are already present

if cloud users have added their own flavors, adding the default flavors may not be needed.  if any flavors are present, no default flavors will be created.

Should we move the nova.flavors definitions into roles/nova-common/defaults/main.yml ?
As a consumer of ursula, if you didn't have defaults file then you would end up w/ a cloud without any flavors.

We make sense to move it to roles/nova-common/defaults/main.yml. I will submit separate PR for it.


Certain JUMP! forms not being able to be fished up.

Describe the bug
I will start out by saying there are many different JUMP! forms for certain pokemon.
32 JUMP! Magikarp
12 JUMP! Shellos
8 JUMP! Clobbopus
9 JUMP! Spheal
8 JUMP! Slugma (and their corresponding magmar palettes)
Currently the entirety of clobbo, spheal, and slugma are not able to be fished up with any kind of rod. (I've spent multiple hours looking pre-reset, especially for slugma which I've probably spent close to 8 hours on) And even if you palette mod the slugma forms, evolving a magby with them in party also won't add the slugma texture to the magmar as its supposed to. (I did make a ticket about this specific issue before I believe)
The shellos forms ARE able to be fished up, however they're incredibly rare. This is fine in my opinion because that just makes fishing them up all the more rewarding, but if you were to make a stickerbook section for them I'd put it into a seperate section besides the Karp one.
Now for the Karps. Out of all 32 forms, both me and Blano have already spent hours fishing for them and have found some issues with 6 of them. 2 of the 6, "White Forehead" and "White Mask" have different names on the wiki than what the palette in game says, which is "Orange Forehead" and "Orange Mask". This itself isn't too big of an issue as long as people are aware they're the same thing.
The other 4 however, "Orange Two-Tone", "Pink Two-Tone", "Blue Raindrops", and "Violet Raindrops", are either not able to be fished up, or are incredibly rarer than they're supposed to be. The wiki for JUMP! forms says that the percentage of getting an Orange Two-Tone from a So-So appraised Old Rod is 11.63%. Blano spent at least 30 minutes fishing with just a So-So rod and got multiple of every other form, but no Orange Two-Tone, even though the Two-Tone in wiki states that its the most common one.
To Reproduce
Steps to reproduce the behavior:

Get an appraised fishing rod
Go to a body of water, or lava for slugma
Fish
Not fishing up the palettes mentioned above

Expected behavior
Able to fish up every JUMP palette as intended.
Username:
Oryso
Server name (all applicable):
Orion (Probably all)
Additional context
Here's the link to the wiki page if you need/want it
https://pixelmonmod.com/wiki/Jump
Also if you need anymore info about it, just let me know. I'm not the best at explaining things.

This is a confirmed pixelmon bug

These have been added through our datapack, closing


Support texture memory in RawKernel

Closes #1811, closes #2278. Although this PR already works (see an example at the bottom), I mark it as WIP for gathering any early feedbacks and comments. Also, I have not yet added tests and docs. Any help for polishing this PR is very welcome and appreciated!
Goal
To provide basic support of texture memory (via "CUDA arrays" or even plain CuPy arrays) in RawKernel.
Implementation decisions and notes

CUDA arrays (cudaArray_t) are supported in the new class cupy.cuda.texture.CUDAArray. 1D, 2D, 3D arrays are all supported. Data can be copied from either CuPy or NumPy arrays to CUDAArray and vice versa, if the datatype and shape match between source and destination.

For the time being I have no thoughts how CUDAArray can fit better with the existing CuPy ndarrays. It'd be awesome if CuPy ndarrays can be backed by texture memory (which in turn is backed by CUDAArray) for readonly purposes, but I don't know how to achieve that. The main technical obstacle is that cudaArray_t is an opaque data type, and it's not possible to do simple pointer arithmetics as we do with linear memories (allocated through cudaMalloc() etc). Another thing is that I don't know CuPy internal well enough 😛

The texture object runtime API is exposed. The end product is the cupy.cuda.texture.TextureObject class, which can be used as an argument of RawKernel.

cudaResourceTypeArray is tested (see below), meaning CUDAArray can back a TextureObject as expected.

mipmap is not supported in this PR.

The user interface mostly follows its CUDA counterpart. Not sure if a better, perhaps more Pythonic public API is possible.

The texture reference API is marked deprecated in the CUDA Toolkit docs, so I didn't add them.

But if we end up supporting it, we should use the driver API instead of the runtime API. The reason is that cuModuleGetTexRef(), a critical function for supporting texture reference, handles CUtexref*, which doesn't seem possible to be converted to runtime's textureReference* (at least it's not documented). <----UPDATE: I confirm that they are indeed incompatible.

UPDATE: The texture reference driver API is exposed. The end product is the cupy.cuda.texture.TextureReference class, which can bind a CUDA array or a CuPy array to a texture reference returned by cupy.RawModule.get_texref().

Example (updated)
The following example illustrates what this PR enables:
import cupy as cp

source=r'''
extern "C"{
__global__ void copyKernel(float* output,
                           cudaTextureObject_t texObj,
                           int width, int height)
{
    unsigned int x = blockIdx.x * blockDim.x + threadIdx.x;
    unsigned int y = blockIdx.y * blockDim.y + threadIdx.y;

    // Read from texture and write to global memory
    float u = x;
    float v = y;
    if (x < width && y < height)
        output[y * width + x] = tex2D<float>(texObj, u, v);
}
}
'''
width = 8
height = 16

# set up a texture object
ch = cp.cuda.texture.ChannelFormatDescriptor(32, 0, 0, 0, cp.cuda.runtime.cudaChannelFormatKindFloat)
arr2 = cp.cuda.texture.CUDAarray(ch, width, height)
res = cp.cuda.texture.ResourceDescriptor(cp.cuda.runtime.cudaResourceTypeArray, cuArr=arr2)
tex = cp.cuda.texture.TextureDescriptor((cp.cuda.runtime.cudaAddressModeClamp, cp.cuda.runtime.cudaAddressModeClamp),
                                        cp.cuda.runtime.cudaFilterModePoint,
                                        cp.cuda.runtime.cudaReadModeElementType)
texobj = cp.cuda.texture.TextureObject(res, tex)

# allocate input/output arrays
tex_data = cp.arange(width*height, dtype=cp.float32).reshape(height, width)
real_output = cp.zeros_like(tex_data)
expected_output = cp.zeros_like(tex_data)
arr2.copy_from(tex_data)
arr2.copy_to(expected_output)

# get the kernel, which copies from texture memory
ker = cp.RawKernel(source, 'copyKernel')

# launch it
block_x = 4
block_y = 4
grid_x = (width + block_x - 1)//block_x
grid_y = (height + block_y - 1)//block_y
ker((grid_x, grid_y), (block_x, block_y), (real_output, texobj, width, height))

# test outcome
if cp.allclose(real_output, expected_output):
    print("OK!")
else:
    print("NOT OK!")

TODO?

[x] Touch cupy_cuda.h for Read the Docs
[x] Support integer types in CUDAArray
[x] Polish the tests I have at hand
[x] Write docs (and perhaps a short tutorial?)
[x] Test cudaResourceDesc.res.linear and cudaResourceDesc.res.pitch2D. (I think they should just work, but I can't guarantee.)
[ ] Remove unused Cython wrappers for CUDA API? To be discussed with maintainers.

It'd be awesome if CuPy ndarrays can be backed by texture memory (which in turn is backed by CUDAArray) for readonly purposes, but I don't know how to achieve that.

As you say, it seems to be pretty difficult ... cudaArrays store data using a space-filling curve indexing to do efficient neighbor access, even if you can get the raw bytes of the cudaArray it might be quite challenging to interface both

cudaArrays store data using a space-filling curve indexing to do efficient neighbor access

@emcastillo are you able to point me to any reference for this? I don't have graphics background and it took me quite a while to just figure out how texture API works. On making better usage of texture, I still have plenty to learn for both my own research work and this PR, and the CUDA Toolkit documentation is not as helpful.

Does CuPy support CUDA's vector types like short3?

are you able to point me to any reference for this?

Sure,
https://devtalk.nvidia.com/default/topic/397677/fast-texture-fetching-an-algorithm-for-fast-texture-fetching/

Not familiar enough with CuPy internals has become a challenge for me. Without much progress, I may have to pause and revisit later. To leave a reference for my future self (or anyone willing to take over this PR), I give a quick summary of my experiments so far.
At the bottom I also post two (more-or-less) equivalent example codes  for reproducing the bug seen in the test suite. One is written in Python based on this PR, and the other is pure CUDA. BUT, only the former shows the bug; the latter works fine.
@emcastillo: what do you think if, as a temporary workaround, we let CUDAArray support transferring to/from NumPy arrays only, and get this PR merged (with more polished tests, docs, etc, for sure)? This'd be a concrete first step in supporting textures.
Notes:

The test TestTexture fails only when the conditions below are all satisfied (see also the above discussion):

When transferring between CUDA arrays and CuPy arrays (NumPy works completely fine whatsoever!).
When there's a previous data movement with another CUDA array of different ndim

It hints a possible memory corruption/pollution due to interference with CuPy internal. In my experiments, the 2D fetch either gave me a result only correct in the 1st row (like https://github.com/cupy/cupy/pull/2432#discussion_r319511851), or a fully zero'd out output.
The way I did data transfer using cudaMemcpy3D works well in any case, so it's not the root of the issue.
cudaTextureObject_t from the runtime API seems to be compatible with cuLaunchKernel from the driver API, even though it's undocumented. This is good news for us.
This doesn't seem to be a memory pool issue (setting allocator to None didn't help).
The deprecated API cudaMemcpyToArray and its relatives only work for 1D CUDA arrays. (This is probably trivial for those familiar with texture...)

Python:
import cupy as cp
cp.cuda.set_allocator(None)
cp.cuda.Device(0).use()

source=r'''
extern "C"{
__global__ void copyKernel(float* output,
                           cudaTextureObject_t texObj,
                           int width, int height)
{
    unsigned int x = blockIdx.x * blockDim.x + threadIdx.x;
    unsigned int y = blockIdx.y * blockDim.y + threadIdx.y;

    // Read from texture and write to global memory
    float u = x;
    float v = y;
    if (x < width && y < height)
    {
        output[y * width + x] = tex2D<float>(texObj, u, v);
    }
}
}
'''
width = 8
height = 16
depth = 4

##### first test: do data movement between 3D CUDA array and a CuPy array #####
##### without this test, the next test works                              #####

arr = cp.arange(width*height*depth, dtype=cp.float32).reshape(depth, height, width)# 3D random array
arr2 = cp.zeros_like(arr)
assert arr.flags['C_CONTIGUOUS']
assert arr2.flags['C_CONTIGUOUS']
# create a CUDA array
ch = cp.cuda.texture.ChannelFormatDescriptor(32, 0, 0, 0, cp.cuda.runtime.cudaChannelFormatKindFloat)
cu_arr = cp.cuda.texture.CUDAArray(ch, width, height, depth)
# copy from input to CUDA array, and back to output
cu_arr.copy_from(arr)
cu_arr.copy_to(arr2)
assert (arr == arr2).all()

##### second test: fetch 2D texture backed by a 2D CUDA array             #####

width = 31
height = 19
depth = 0

ch = cp.cuda.texture.ChannelFormatDescriptor(32, 0, 0, 0, cp.cuda.runtime.cudaChannelFormatKindFloat)
arr3 = cp.cuda.texture.CUDAArray(ch, width, height)
res = cp.cuda.texture.ResourceDescriptor(cp.cuda.runtime.cudaResourceTypeArray, cuArr=arr3)
tex = cp.cuda.texture.TextureDescriptor((cp.cuda.runtime.cudaAddressModeClamp, cp.cuda.runtime.cudaAddressModeClamp),
                                        cp.cuda.runtime.cudaFilterModePoint,
                                        cp.cuda.runtime.cudaReadModeElementType)
texobj = cp.cuda.texture.TextureObject(res, tex)

tex_data = cp.zeros((height, width), dtype=cp.float32)
for i in range(height):
    tex_data[i] = i
real_output = cp.zeros_like(tex_data)
expected_output = cp.zeros_like(tex_data)
arr3.copy_from(tex_data)
arr3.copy_to(expected_output)
assert (tex_data == expected_output).all()

ker = cp.RawKernel(source, 'copyKernel')

block_x = 4
block_y = 4
grid_x = (width + block_x - 1) // block_x
grid_y = (height + block_y - 1) // block_y
ker((grid_x, grid_y), (block_x, block_y), (real_output, texobj, width, height))
print(texobj.ptr)

if cp.allclose(real_output, expected_output):
    print("OK!")
else:
    print("NOT OK:")
    print(real_output)

CUDA:
#include <cuda.h>
#include <stdlib.h>
#include <stdio.h>
#include <assert.h>

__global__ void copyKernel(float* output,
                           cudaTextureObject_t texObj,
                           int width, int height)
{
    unsigned int x = blockIdx.x * blockDim.x + threadIdx.x;
    unsigned int y = blockIdx.y * blockDim.y + threadIdx.y;

    // Read from texture and write to global memory
    float u = x;
    float v = y;
    if (x < width && y < height)
    {
        //printf("th: %i %i\n", x, y);
        output[y * width + x] = tex2D<float>(texObj, u, v);
        //printf("th %i %i: %f\n", x, y, tex2D<float>(texObj, u, v));
    }
}

int main() {
    int width = 8, height = 16, depth = 4;
    float i = 0;
    float * h_data = (float*)malloc(width*height*depth*sizeof(float));
    float * h_result = (float*)malloc(width*height*depth*sizeof(float));

    /* =============================================
       First, only test copy to and from CUDA array:
       1. copy from host mem to device mem
       2. copy from device mem to CUDA array
       3. copy from CUDA array to device mem
       4. copy from device mem to host mem
       ============================================= */

    for(size_t d = 0; d < depth; d++) {
        for(size_t h = 0; h < height; h++) {
            for (size_t w = 0; w< width; w++) {
                h_data[d * width * height + h * width + w] = i;
                h_result[d * width * height + h * width + w] = 0;
                i++;
            }
        }
    }

    // Allocate device memory
    float* d_data;
    cudaMalloc(&d_data, width*height*depth*sizeof(float));
    cudaMemcpy(d_data, h_data, width*height*depth*sizeof(float), cudaMemcpyHostToDevice);

    // Allocate device memory
    float* d_result;
    cudaMalloc(&d_result, width*height*depth*sizeof(float));

    // Allocate CUDA array in device memory
    cudaChannelFormatDesc channelDesc =
               cudaCreateChannelDesc(32, 0, 0, 0,
                                     cudaChannelFormatKindFloat);
    cudaArray_t cuArray;
    cudaMalloc3DArray(&cuArray, &channelDesc, make_cudaExtent(width, height, depth));

    cudaMemcpy3DParms myparms = {0};
    myparms.srcPtr = make_cudaPitchedPtr((void*)d_data, width*sizeof(float), width, height);
    myparms.dstArray = cuArray;
    myparms.extent = make_cudaExtent(width, height, depth);
    myparms.kind = cudaMemcpyDeviceToDevice;
    cudaMemcpy3D(&myparms);

    printf("check CUDA array data...\n");
    memset(&myparms, 0, sizeof(cudaMemcpy3DParms));
    myparms.srcArray = cuArray;
    myparms.extent = make_cudaExtent(width, height, depth);
    myparms.dstPtr = make_cudaPitchedPtr((void*)d_result, width*sizeof(float), width, height);
    myparms.kind = cudaMemcpyDeviceToDevice;
    cudaMemcpy3D(&myparms);

    cudaMemcpy(h_result, d_result, width*height*depth*sizeof(float), cudaMemcpyDeviceToHost);
    for(size_t d = 0; d < depth; d++) {
        for(size_t h = 0; h < height; h++) {
            for (size_t w = 0; w< width; w++) {
                float in = h_data[d * width * height + h * width + w];
                float out = h_result[d * width * height + h * width + w];
                if (in != out){
                    printf("d: %i, h: %i, w: %i: in %f, out %f\n", d, h, w, in, out);
                }
            }
        }
    }

    /* =============================================
       Next, test fetching 2D texture memory:
       1. copy from host mem to device mem
       2. copy from device mem to CUDA array
       3. bind CUDA array to a texture object
       4. call the kernel to fetch from texture memory and update dev mem
       5. copy from device mem to host mem
       ============================================= */

    width = 31;
    height = 19;
    depth = 0;

    cudaArray_t cuArray2;
    cudaMalloc3DArray(&cuArray2, &channelDesc, make_cudaExtent(width, height, depth));
    float * h_data2   = (float*)malloc(width*height*sizeof(float));
    float * h_result2 = (float*)malloc(width*height*sizeof(float));

    i = 0;
    for(size_t h = 0; h < height; h++) {
        for (size_t w = 0; w< width; w++) {
            h_data2[h * width + w] = i;
            h_result2[h * width + w] = 0;
        }
        i++;
    }

    float* d_data2;
    float* d_result2;
    cudaMalloc(&d_data2, width*height*sizeof(float));
    cudaMalloc(&d_result2, width*height*sizeof(float));
    cudaMemcpy(d_data2, h_data2, width*height*sizeof(float), cudaMemcpyHostToDevice);

    memset(&myparms, 0, sizeof(cudaMemcpy3DParms));
    myparms.srcPtr = make_cudaPitchedPtr((void*)d_data2, width*sizeof(float), width, height);
    myparms.dstArray = cuArray2;
    myparms.extent = make_cudaExtent(width, height, 1);
    myparms.kind = cudaMemcpyDeviceToDevice;
    cudaMemcpy3D(&myparms);

    // Specify texture
    struct cudaResourceDesc resDesc;
    memset(&resDesc, 0, sizeof(resDesc));
    resDesc.resType = cudaResourceTypeArray;
    resDesc.res.array.array = cuArray2;

    // Specify texture object parameters
    struct cudaTextureDesc texDesc;
    memset(&texDesc, 0, sizeof(texDesc));
    texDesc.addressMode[0]   = cudaAddressModeClamp;
    texDesc.addressMode[1]   = cudaAddressModeClamp;
    texDesc.filterMode       = cudaFilterModePoint;
    texDesc.readMode         = cudaReadModeElementType;
    //texDesc.normalizedCoords = 1;

    // Create texture object
    cudaTextureObject_t texObj = 0;
    cudaCreateTextureObject(&texObj, &resDesc, &texDesc, NULL);

   // Invoke kernel
    dim3 dimBlock(4,4);
    dim3 dimGrid((width  + dimBlock.x - 1) / dimBlock.x,
                 (height + dimBlock.y - 1) / dimBlock.y);
    copyKernel<<<dimGrid, dimBlock>>>(d_result2, texObj, width, height);

    // /* Alternative: Use driver API to launch kernel */
    // void* args[4] = {&d_result2, &texObj, &width, &height};
    // CUmodule mod;
    // CUfunction func;
    // CUresult status = cuModuleLoad(&mod, "/home/leofang/test_texture_obj_kernel.cubin");
    // status = cuModuleGetFunction(&func, mod, "copyKernel");
    // cuLaunchKernel(func, dimGrid.x, dimGrid.y, 1, dimBlock.x, dimBlock.y, 1, 0, NULL, args, NULL);
    // /* launch finished */

    printf("check gpu data...\n");
    cudaMemcpy(h_result2, d_result2, width*height*sizeof(float), cudaMemcpyDeviceToHost);
    for(size_t h = 0; h < height; h++) {
        for (size_t w = 0; w< width; w++) {
            float in = h_data2[h * width + w];
            float out = h_result2[h * width + w];
            if (in != out){
                printf("h: %i, w: %i: in %f out %f\n", h, w, in, out);
            }
        }
    }

    // Destroy texture object
    cudaDestroyTextureObject(texObj);

    // Free device memory
    cudaFreeArray(cuArray);
    cudaFree(d_result);
    cudaFree(d_data);
    free(h_data);
    free(h_result);

    // Free device memory
    cudaFreeArray(cuArray2);
    cudaFree(d_result2);
    cudaFree(d_data2);
    free(h_data2);
    free(h_result2);

    return 0;

Thanks a lot for the self-contained example

Whole lot of tests are added and passed locally!

Dear @emcastillo, I have three new requests:

With commit 4832ace, the test suite has a pretty good coverage. Do you think you can make a first round of review (whenever you are free, of course!) before I move on to docs and tutorial? This is just to avoid repetitive work if we decide to make changes to the API.

Could you in the meantime kick off Jenkins for me so that I get to see if (and how) it fails on older CUDA? I remember your Jenkins is running 8.0.

Thanks!

The PR looks good to me in general, the only thing I am a bit concerned is that we are allocating memory (cudaArrays) outside cupy allocator.
While there is a clear motivation to do this, I think it would help to move the cudaArray allocating functions to cupy allocator in order to track the memory usage better, and having all the code dealing with memory alloc/deletion in one place.

Jenkins, test this please

Jenkins CI test (for commit 47c55051b5505cbd1cad6ed0cef3e8591bb40672, target branch master) failed with status FAILURE.

I thought about the issue of memory pool allocator too. The problem is two-fold:

cudaArrays are opaque objects, and it is unclear how much extra memory it takes from the global memory on top of the underlying raw data, so it's impossible to determine its precise memory usage;
cudaArrays do not consume the CUDA address space (refs: here and here).

So I don't think I have enough knowledge for tracking its memory usage, given that cudaArray is (purposely?) poorly documented.
We could definitely show a big warning in CUDAarray's docstring to warn users about this. I suspect only a few percent of the user base would every want to use CUDAarray?
Having said that,

having all the code dealing with memory alloc/deletion in one place.

I agree it seems to make sense to move CUDAarray to cupy/cuda/memory.pxd(.pyx). If it's OK to you too I'll make changes.
Thanks!

There're 36 cases [= 1 (xp) * 2 (stream) * 3 (dimensions) * 6 (all int dtypes)] on TestCUDAarray that failed, because Jenkins used a very old NumPy that does not have the dtype argument in numpy.random.randint(). This is now circumvented in
23b0026. It's nice to see the rest of tests on GPU passed!

Jenkins, test this please

I agree it seems to make sense to move CUDAarray to cupy/cuda/memory.pxd(.pyx). If it's OK to you too I'll make changes.

I want to hear the opinion of @kmaehashi or @asi1024 before doing it 🙂

Jenkins CI test (for commit 23b002622093a31b9c809f6b70aaf11f0a41af2b, target branch master) failed with status FAILURE.

Jenkins failure is related to #2444 and irrelevant of this PR! Good to confirm we don't hit any compatibility problem. I'll proceed to docs and tutorials while waiting for comments from @kmaehashi or @asi1024.
One more thing:

Remove unused Cython wrappers for CUDA API? To be discussed with maintainers.

Examples for this include the memcpy2D* family, which were used and tested at some point. Should I clear all of the Cython wrappers for unused CUDA APIs, or it's OK to let them hang and wait when the time comes?

In 2aa37d4, the support of vector types (such as int3, float4, etc) is added to CUDAarray and tested, so that reading texture of vector types in RawKernel is possible.
In 734e2e7, docs and tutorial are added.
Now this PR is ready for review!

I found that the texture reference API can also be supported based on what we already have in this PR, so I just added it. Let me know if you think the leap is too huge and you prefer an incremental change. Thanks.

Two notes:

Both TextureObject and TextureReference now support either CUDAarray or cupy.ndarray. All tests passed locally.
Upon reflection, I no longer think moving CUDAarray to the cupy.cuda.memory module is a good idea. CUDAarray is a very specialized interface and it's independent of most of CuPy functionalities. Putting it under cupy.cuda.texture seems more logical for texture memory users. However, I don't have a strong position on this.

This PR is ready for review (again)!

sorry didn't meant to rush you @emcastillo, I saw there's a new button appearing on the reviewer list and I clicked it without thinking 😛

No prob, I was working on some high priority issues last week so I had to delay this.
sorry for the waiting.

Jenkins, test this please

Jenkins CI test (for commit 27b681c01ffa4417b96f89bb9a2994a7546e06e3, target branch master) failed with status FAILURE.

Jenkins, test this please

Jenkins CI test (for commit 27b681c01ffa4417b96f89bb9a2994a7546e06e3, target branch master) succeeded!

Thank you so much @emcastillo for prompt feedbacks and discussions along the way! I suppose this PR will be tagged v7.0.0b4?

Jenkins CI test (for commit 27b681c01ffa4417b96f89bb9a2994a7546e06e3, target branch master) succeeded!

why has jenkins been triggered?

Don’t know... I thought it’s your internal setup?
Sincerely,
Leo Fang (Fang Yao-Lung*)*

Just wanted to say thanks for this.   I had been wanting to port more code from pycuda to cupy, and texture memory was one of the main sticking points.  Just tried this and it's working great for me.  Much appreciated!

@tlambert03 Thank you for sharing with us your success story! We need more user reports like this for supporting new features😃

Thank you very much for the feature! I managed to port some of my pycuda code to cupy pretty easily, and in a better way (texture references instead of hardcoded textures).

Thanks for letting me know! It's encouraging.

Would it be possible to add support for texture objects in ElementwiseKernel? I can try to do this myself, but if someone could give hints where to start looking that would be great.

Let us move this discussion to #2550.

#4433 extends the support to ElementwiseKernel.


add yaml from /en

add files in zh/ , these files are detected by scripts , see example  https://github.com/kubernetes/website/blob/master/content/en/examples/admin/snowflake-deployment.yaml , but is not exsisted in  zh/ see  https://github.com/kubernetes/website/tree/master/content/zh/examples/admin
/cc @tengqm
/cc @xichengliudui
Please review this pr , Thank  you very much .

/lgtm
/approve


Please update to @actions/core 1.10 to avoid warnings about deprecated set-output

Subject says it all. I would offer a PR but alas I know almost nothing about JavaScript or how to properly perform this update :-(

Thanks for letting me know. I've just released v1.2.12 that will take care of it.

Thank you!


Fix License formatting

Adjust the license so it matches the official format: https://opensource.org/licenses/MIT
This should fix #13

Thanks, that was quick. Hacktoberfest has not started yet, so this won't count to your stats :(
I am not sure if it will count if I go ahead and merge it in October, as I think the PR has to be opened in October. What should we do?

I just checked The Manifest Format
# This is an SPDX 2.1 license expression for this package. Currently
# crates.io will validate the license provided against a whitelist of
# known license and exception identifiers from the SPDX license list
# 2.4. Parentheses are not currently supported.
#
# Multiple licenses can be separated with a `/`, although that usage
# is deprecated. Instead, use a license expression with AND and OR
# operators to get more explicit semantics.
license = "..."

# If a package is using a nonstandard license, then this key may be specified in
# lieu of the above key and must point to a file relative to this manifest
# (similar to the readme key).
license-file = "..."

Do you think it would make sense to replace license-file = "LICENSE" with license = "MIT" for being extra clear?

Silly me, I totally forgot about the october part in Hacktoberfest.
The issue just caught my eye by accident, please feel free to handle this how you seem fit, I dont really care that much about the stats.

Do you think it would make sense to replace license-file = "LICENSE" with license = "MIT" for being extra clear?

Looking at some official crates this does indeed seem like the way to go

Thank you for the change! I appreciate it!
I will go ahead and merge it now, but please feel free to come back anytime (or during October) to get your PRs counting for Octoberfest!


MSID: 92860 Version: 2 DOI: 10.1101/2<IP_ADDRESS>7013

MSID: 92860
Version: 2
Preprint DOI: https://doi.org/10.1101/2<IP_ADDRESS>7013
Step 1. Awaiting reviews
Editorial to post reviews via hypothesis
Useful links:

DocMap: https://data-hub-api.elifesciences.org/enhanced-preprints/docmaps/v2/by-publisher/elife/get-by-manuscript-id?manuscript_id=92860
New model tracking: https://docs.google.com/spreadsheets/d/1_fHaoOy7hjyocptKtVJRijeNpUY4hBS7Ck_aVmx6ZJk/
Reviews on sciety: https://sciety.org/articles/activity/10.1101/2<IP_ADDRESS>7013

For trouble shooting (e.g. no Docmaps available):

DocMap issue addressing: https://miro.com/app/board/uXjVNCwK6EI=/
Explore DataHub DocMaps API: https://lookerstudio.google.com/reporting/4c2f0368-babb-4beb-b5b3-497e7e7b0f08/page/ejphD
Unmatched submissions and preprints: https://lookerstudio.google.com/u/0/reporting/9f86204f-3bf7-477c-9b18-5c5ef141bf69/page/p_gxi57ha93c
Unmatched manuscripts spreadsheet: https://docs.google.com/spreadsheets/d/15QcK8w-ssB7109RQEDtFpJPZ0J5HTGxoHa_2TtpMBbg/edit#gid=1336081641

Step 2. Preview reviewed preprint
Production QC content ahead of publication
Instructions:

QC preview: https://prod--epp.elifesciences.org/previews/92860v2
Update ticket with any problems (add blocked label)
When QC OK, add QC OK label to ticket and add publication date and time to https://docs.google.com/spreadsheets/d/1amAlKvdLcaDp5W8Z8g77NmkwbMF5n_u89ArSqPMO8jg
Move card to next column
(At end of the day post link in #enhanced-preprint and ask for PDF to be generated)

Useful links:

Preprint DOI :  https://doi.org/10.1101/2<IP_ADDRESS>7013
Confirm reviews returned by EPP: https://prod--epp.elifesciences.org/api/reviewed-preprints/92860/v2/reviews
To update the MECA path in the docmap: https://docs.google.com/spreadsheets/d/1mctCQuNFBjSn97Lihy7_vBO6z7-N-oqyLv4clyi6zHg

Step 3: Awaiting search reindex
This step adds the reviewed preprint to the homepage: https://elifesciences.org
The search reindex is triggered once an hour. We need the reviewed preprint to be indexed as the search application serves the journal homepage.
Useful links:

Jenkins pipeline to reindex search can be triggered sooner or monitored here: https://alfred.elifesciences.org/job/process/job/process-reindex-reviewed-preprints/

Step 4: Published! PDF requested
Waiting for PDF to be generated
Useful links:

PDF tracking: https://docs.google.com/spreadsheets/d/106_XeDjmuBae7gexOTNzg60lapeqjl2aRn9DzupGyS8/

Step 5: Introduce PDF to data folder and git repo
Upload PDF to relevent folder in git repo https://github.com/elifesciences/enhanced-preprints-data/
Step 6: Done!

[ ] Kettle is on!

Added bioRender permissions in all figs except fig 8.


D456 unity Slam scene error

Required Info

Camera Model
D456

Firmware Version
5.15.1

Operating System & Version
win10

Platform
PC

SDK Version
2.0

Language
unity

Issue Description
Error message in SLAM scene on start play
ExternalException: rs2_pipeline_start_with_config(pipe:0000019C4225C9A0, config:0000019C4225D4A0)
Rethrow as Exception: Couldn't resolve requests
Intel.RealSense.ErrorMarshaler.MarshalNativeToManaged (System.IntPtr pNativeData) (at :0)
Intel.RealSense.Pipeline.Start (Intel.RealSense.Config cfg) (at :0)
RsDevice.OnEnable () (at Assets/RealSenseSDK2.0/Scripts/RsDevice.cs:71)

Hi @yuhe0925  The Unity SLAM example only works with a now retired RealSense model called the T265 Tracking Camera.

Hi @yuhe0925  Bearing in mind that the SLAM example only works with the RealSense T265 Tracking Camera, do you require further assistance with this case please?  Thanks!

Case closed due to no further comments received.


Raise proper error when deconding 16-bits jpegs

After some investigation, I found that #3613 failed because the image is 16 bit. However our decoding code assumes the image is 8-bit. So our allocated tensor size is too small to hold the decoded data and leads to memory overflow.
However, I can't find an easy way to decode 16-bit images without breaking the current decoding logic(we would have to change too many code places). So I simply issue an error.
Note: In order to support 16-bit png images, we should allocate the tensor with torch::kI16 and change
auto datap = data.accessor<unsigned char, 1>().data();

and
auto ptr = tensor.accessor<uint8_t, 3>().data();

and the logic of read_callback. And although #4051 didn't fix this bug, I still think it's useful..

And there may be 32-bit or 64-bit png images. And I'm not sure the case of jpg.

Thanks @cyyever !

Thanks for the fix @cyyever !


SemanticDB phases

Stripe's Scala team has an intern, and this quarter we're planning work that requires SemanticDB outputs!
I believe SemanticDB support can be easily encapsulated in a new optional phase that:

adds the scalac plugin
declares the new SemanticDB output files

This pattern was already prototyped in higherkindness/rules_scala, so I'm confident that it will work here too.
We'll circle back soon with an update once we can commit to this work.

How I solved this (without direct rules_scala support):

Added semanticdb as a dependency with rules_jvm_external: "org.scalameta:semanticdb-scalac_%s:4.7.1" % scala_version
Set up your scalacopts and compiler plugins accordingly:

semanticdb = "@maven//:org_scalameta_semanticdb_scalac_%s" % scala_version_suffix

semanticdb_scalaopts = [
    "-Xplugin:$(location %s)" % semanticdb,
    "-Yrangepos",
]

semanticdb_deps = [
    semanticdb,
]

compiler_deps = semanticdb_deps

# Ensure your deps provider for the scala compiler includes the compiler deps:
declare_deps_provider(
    name = "my_scala_compile_classpath_provider",
    deps_id = "scala_compile_classpath",
    visibility = ["//visibility:public"],
    deps = [
        "@maven//:org_scala_lang_scala_compiler",
        "@maven//:org_scala_lang_scala_library",
        "@maven//:org_scala_lang_scala_reflect",
    ] + compiler_deps,
)

# In your targets ensure to provide scalaopts and deps:
scala_library(
  # ...
  scalacopts = semanticdb_scalaopts,
  deps = semanticdb_deps,
)
...

As an extra detail, how I managed to use Scalafix after the above was applied:

```shell
# Install coursier (`cs`) with whatever method.

# Install scalafix:
cs install scalafix
scalafix --version

# I wanted to use ZIO scalafix rules, so download and build ZIO scalafix rules:
git clone https://github.com/zio/zio/
cd zio
sbt scalafixRules/publishLocal

# In your repo:
# Build a deploy jar:
bazel build //some/custom/path:library_deploy.jar
# Run scalafix:
scalafix --tool-classpath $HOME/.ivy2/local/dev.zio/scalafixrules_2.13/2.0.5+38-7efadb73-SNAPSHOT/jars/scalafixrules_2.13.jar --rules Zio2Upgrade --files some/custom/path/src/main/scala/ --classpath bazel-bin/some/custom/path/library_deploy.jar

This issue can probably be closed now. SemanticDB support was added in #1508


Unable to specify "x-ms-azs-return-searchid" header in Azure.Search.Documents.SearchClient.SearchAsync()

Linking these two issues so people don't get confused why you are unable to add the headers. https://github.com/Azure/azure-sdk-for-net/issues/17449
The code sample shows you can specify the x-ms-azs-return-searchid as header but this is not possible in the current v11.1.1 or v11.2.0-beta.2 versions

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 3d15016b-314c-5757-2ea0-76bd04cf87c6
Version Independent ID: bc6c02ee-9770-7af5-e496-34e5ef8d0865
Content: Telemetry for search traffic analytics - Azure Cognitive Search
Content Source: articles/search/search-traffic-analytics.md
Service: cognitive-search
GitHub Login: @HeidiSteen
Microsoft Alias: heidist

@rikvandenberg Thank you for the valuable feedback, we are investigating the issue.

@rikvandenberg Thank you for the valuable feedback, we are investigating the issue.

@rikvandenberg Thanks again for feedback! The Doc team is aware of this and it will be updated in the next sprint.
Please refer to below link as current workaround  for setting custom headers for Azure.Core feature.
https://github.com/Azure/azure-sdk-for-net/blob/master/sdk/core/Azure.Core/samples/Diagnostics.md#setting-x-ms-client-request-id-value-sent-with-requests
We will now proceed to close this thread. If there are further questions regarding this matter, please let us know.

@rikvandenberg Thanks again for feedback! The Doc team is aware of this and it will be updated in the next sprint.
Please refer to below link as current workaround  for setting custom headers for Azure.Core feature.
https://github.com/Azure/azure-sdk-for-net/blob/master/sdk/core/Azure.Core/samples/Diagnostics.md#setting-x-ms-client-request-id-value-sent-with-requests
We will now proceed to close this thread. If there are further questions regarding this matter, please let us know.


Unable to use WIF identity to authenticate to Azure environment

Describe the bug
Unable to use WIF identity to authenticate inside the pipeline
Steps to reproduce

I am using the newest tool version and I have configured it manually (not using provided script)
In credentials variable group I have ARM_SUBSCRIPTION_ID and ARM_TENANT_ID set, but ARM_SERVICE_CONNECTION is empty (not sure if that is correct from the documentation)
When I run the pipeline (doesn't matter which one), I got error mentioned in the attached screenshot

Screenshots

ARM_SERVICE_CONNECTION indeed needs to have a value, otherwise we won't use the federated credentials. So please add that to your variables and hopefully that will do the trick.
https://github.com/Azure/AzOps-Accelerator/blob/main/.pipelines/.templates/sharedSteps.yml#L72


Manual chunking + Automatic Chunking

Details
Is it possible to split custom sitemaps and decide how many URLs to keep?
My current problem is that we have 30k URLs products, and the SEO department asked to split multi sitemap at 5k/10k URLs per sitemap.
So I'm wandering if there is a way to do this.

Hey, this sounds quite possible.
Currently, it splits into separate sitemaps per 1k urls using https://nuxtseo.com/sitemap/guides/multi-sitemaps#automatic-chunking.
You would want a config value like automaticSitemapsChunkSize: <value> that would let you split them at 5k or 10k urls?
The alternative is to chunk manually https://nuxtseo.com/sitemap/guides/multi-sitemaps#manual-chunking?

Using the automaticSitemapsChunkSize:  feature would be really helpful. Currently, I'm getting all my URLs from a dynamic API and then manually splitting them into multiple sitemaps. But from what I can see in the documentation, there is no option like SitemapsChunkSize:  to automatically split the custom sitemaps.

I see, thanks for the explanation.
This sounds quite possible but will require a bit of work. Will try to get around to it when I have some availability.

I think this will be too much work for quite a specific use case, so I'm going to close this issue.
Also it does seem possible to solve this on your end with the new v4 release.
You would just need to query the total count of records and dynamically create the sitemap config when your Nuxt app starts.


bug: envd-ssh in tensorchord/python:3.8-ubuntu20.04-cuda11.6-cudnn8 is old

Description
envd-ssh in tensorchord/python:3.8-ubuntu20.04-cuda11.6-cudnn8 is old, which breaks the cuda installation
docker run --rm -it --entrypoint /bin/bash tensorchord/python:3.8-ubuntu20.04-cuda11.6-cudnn8

>  /var/envd/bin/envd-ssh --help

XXXX

VERSION:
   0.0.1

Fixed in https://github.com/tensorchord/envd/pull/681/files to use python3.9-X as base image

Could we close it?


Dev 0.1

add a converter tool

The conflicts can also be resolved, and compatibility with previous original configuration is recommended, which can be verified by unit testing @zhu733756

The conflicts can also be resolved, and compatibility with previous original configuration is recommended, which can be verified by unit testing @zhu733756

Would be fixed. Pls answer my two questions above and  I would readjust it according to your suggestion.


Crash when authenticator is freed before starting replicator

It seems like an authenticator must not be freed until the replicators which have been configured with it are freed.
From the docs, that wasn't clear to me, or maybe the authenticator should be copied when the replicator is created.
F/libc    (24687): Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x10007 in tid 24737 (DartWorker), pid 24687 (issue_142_repro)
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
Build fingerprint: 'Android/sdk_phone_x86/generic_x86:11/RSR1.210210.001.A1/7193139:userdebug/dev-keys'
Revision: '0'
ABI: 'x86'
Timestamp: 2021-09-03 19:00:30+0200
pid: 24687, tid: 24737, name: DartWorker  >>> com.example.cbl_issue_142_repro <<<
uid: 10139
signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x10007
    eax b82a0110  ebx b5177a20  ecx 0000ffff  edx f4716407
    edi ef181464  esi afdfb348
    ebp afdfb328  esp afdfb2f0  eip b4cbb3dc
backtrace:
      #00 pc 0003a3dc  /data/app/~~3MsLwqnhFQUJ0SsOm7zZUg==/com.example.cbl_issue_142_repro-6SjHCojRfwnU4eq_qUnxNg==/lib/x86/libcblite.so (BuildId: 29dc76d7220355d0073d6bc6765ca5e2e50fd6c6)
      #01 pc 00038970  /data/app/~~3MsLwqnhFQUJ0SsOm7zZUg==/com.example.cbl_issue_142_repro-6SjHCojRfwnU4eq_qUnxNg==/lib/x86/libcblite.so (BuildId: 29dc76d7220355d0073d6bc6765ca5e2e50fd6c6)
      #02 pc 00037744  /data/app/~~3MsLwqnhFQUJ0SsOm7zZUg==/com.example.cbl_issue_142_repro-6SjHCojRfwnU4eq_qUnxNg==/lib/x86/libcblite.so (BuildId: 29dc76d7220355d0073d6bc6765ca5e2e50fd6c6)
      #03 pc 000376c8  /data/app/~~3MsLwqnhFQUJ0SsOm7zZUg==/com.example.cbl_issue_142_repro-6SjHCojRfwnU4eq_qUnxNg==/lib/x86/libcblite.so (CBLReplicator_Start+40) (BuildId: 29dc76d7220355d0073d6bc6765ca5e2e50fd6c6)
      #04 pc 00003f5f  <anonymous:c2880000>
Lost connection to device.

Steps to reproduce:

Create an authenticator
Create a replicator with that authenticator
Free authenticator
Add  document replication listener
Start replicator
Crash at line: https://github.com/couchbase/couchbase-lite-C/blob/00ad08293070a02572467b568fb8d727ff9ded49/src/CBLReplicatorConfig.hh#L218

The contract is definitely not for you to need to hang on to this for the entire lifetime of the replicator, but I also don't think it needs to be copied because it should only be used once (the bug is that it is not only used once).
The bug is actually in steps 4 and 5 here.  Adding a document replication listener causes a flag to be set which (in my opinion, mistakenly) re-encodes the entire configuration.
Opened CBL-2337

Fixed in d37c918eade3d144443ff24b8c1f3f62949fbbb2 and 8eeaa2b8c153824aa949178e9708d51944895057.


Keras API in documentation

Since Read the Docs/Sphinx doesn't recognize Google style docstrings, our recent work isn't being added to the documentation.  We should fix Sphinx so that it recognizes these docstrings.

Was the entire project changed to these new style docstrings or just the new keras work?

I was able to get a doc generated for keras.layers.Dense using autodoc
`keras.layers.Dense`
=========

.. autoclass:: tf_encrypted.keras.layers.dense.Dense
  :members:

Will generated

Obviously the spacing here is a little wonky .. we can fix that up with new lines if needed.
Going to look into https://www.sphinx-doc.org/en/master/usage/extensions/napoleon.html first, though.

napoleon worked pretty flawlessly, and even integrates with the older docs that we can eventually roll over
keras.layers.Dense renders as

So we can auto generated the API docs based on all of the files that have docs.  This will render something that looks like

I recommend that we do this and remove the extra things in the docs (installing, getting started, etc)., and only have the API docs. We have the website + github for these things, and since we have to rewrite these things for the docs, they go out of date.

I dug into another possible avenue which I think is interesting.
I was curious about rendering the docs on tf-encrypted.io rather than readthedocs.  I think this is what we would want to do long term and might just be worth setting up right now.  I have hacked together a simple proof of concept that I think shows some promise.  Basically I did the following

Converted tf-encrypted/website to be a gatsby project rather than next.js.  The reason I did this is for the build/data pipeline.  In nextjs you can't build things from the filesystem and we would need to introduce network calls to build blogs/docs.

In gatsby, you can generate pages based on graphql queries, I think this is pretty heavy-handed, as we  can leverage the filesystem pretty easily for now (possible avenue for down the road).  Generating docs will look like this:

We will have to change the build pipeline of docs very slightly.  It is possible to have the sphinx-build command output json, and with naplean above we introduce one more command (explained above)
then, we can create a template to render the doc pages

render() {
    let content = this.props.pageContext.pageContent
    return (

    )
}

This will render a page that looks like this

The doc output json will basically give you the HTML you want, then we can apply CSS as we see fit
E.g. we can easily make it a bit more on brand:
// docs.scss
@include font-face(RobotoMono, fonts/RobotoMono-Regular, null, null, ttf)

code {
    color: #06C6A2;
}

h1 {
    font-family: "RobotoMono";
}

h2 {
    font-family: "RobotoMono";
}

.doc {
    padding: 2rem;
    padding-top: 7rem;
}

will render

Obviously we will need to do a bit more than this.  E.g. making a sidebar, etc

Another bonus is that this same strategy would work for some other things on the site
e.g. right now the tutorials link back to github, but we could host them on the site, but the content is generated via what is in github source.
Same with a blog, if we decided to start one.

Is there another way to include html content inside react without using this property: dangerouslySetInnerHTML ?

Is there another way to include html content inside react without using this property: dangerouslySetInnerHTML ?

No, it is rendered as text otherwise.

So, have dug around sphinx quite a bit and it kind of seems to me like the best way to do docs is more or less how we are doing them now but with the napolean plugin so we can use google style doc strings.
I spent some time digging in to automatically documenting every file/module, as we have a tendency to forget about the docs/ dir, but in my opinion this makes the amount of API docs too daunting.  You can set excludes but then you have to remember to do that, so you have the reverse problem of what we have right now.
In summary, we can have either a single index.rst directory that just includes
.. toctree::
  :glob:
  tf_encrypted*

and you will generate everything
e.g.

Or we can specifically decide what we want to generate
e.g.
.. toctree::

  keras.layers
  keras.engine
  converter

will only make docs for those three modules.
In both cases we can still self host the docs.
cc @mortendahl.  Which would you prefer?

great investigation @bendecoste! 💪

Which would you prefer?

agree that including everything would be too much, so explicitly mentioning modules seems like the best approach. it won't change that much so optimistic that we'll remember to also include new items to the list


all evergreen links, URLs work

@alandefreitas - thanks for reviewing - can we merge it?

thanks for reviewing - can we merge it?

Sure. Can I assume you consider the PR good to go as long as it's not a draft?

@alandefreitas - Normally I think it will be good to go, unless you have found issues I should address


dead code in esp_efuse_mac_get_default (IDFGH-4066)

The following lines in esp_efuse_mac_get_default are dead code:
https://github.com/espressif/esp-idf/blob/c77c4ccf6c43ab09fd89e7c907bf5cf2a3499e3b/components/esp_common/src/mac_addr.c#L99-L102
Regardless of the value of mac_low, the function returns ESP_OK

What do you mean there is a range of 50 values

Yes, there is a range of values.  But, whether the value of mac_low is inside or outside of that range, the function returns ESP_OK. It is clear if you look at what follows the dead code.
https://github.com/espressif/esp-idf/blob/c77c4ccf6c43ab09fd89e7c907bf5cf2a3499e3b/components/esp_common/src/mac_addr.c#L98-L110

I see, the else needs to be removed so that it aborts

Yes, something like that. I'm not entirely sure I understand the intent of the code, so I don't dare suggest a specific change. Thank you for your patience to understand the bug report.


⚠ 366.ru has degraded performance

In bcf6a7c, 366.ru (https://366.ru) experienced degraded performance:

HTTP code: 200
Response time: 6055 ms

Resolved: 366.ru performance has improved in 350618b.


Broken master build (re. mariaDB4j-maven-plugin integration tests)

Recent PRs such as #258 and #260 illustrate that the master build is broken:
[INFO] --- maven-invoker-plugin:3.2.0:run (integration-test) @ mariaDB4j-maven-plugin ---
[INFO] Building: mariadb4j-maven-plugin-test-basic/pom.xml
[INFO] [INFO] Scanning for projects...
[INFO] [INFO] Downloading from local.central: file:///home/travis/.m2/repository/ch/vorburger/mariaDB4j/mariaDB4j-pom/2.4.0-SNAPSHOT/maven-metadata.xml
[INFO] [INFO] Downloading from local.central: file:///home/travis/.m2/repository/ch/vorburger/mariaDB4j/mariaDB4j-pom/2.4.0-SNAPSHOT/mariaDB4j-pom-2.4.0-SNAPSHOT.pom
[INFO] [ERROR] [ERROR] Some problems were encountered while processing the POMs:
[INFO] [FATAL] Non-resolvable parent POM for ch.vorburger.mariaDB4j:mariadb4j-maven-plugin-test-basic:[unknown-version]: Could not find artifact ch.vorburger.mariaDB4j:mariaDB4j-pom:pom:2.4.0-SNAPSHOT in local.central (file:///home/travis/.m2/repository) and 'parent.relativePath' points at wrong local POM @ line 6, column 13
[INFO]  @
[INFO] [ERROR] The build could not read 1 project -> [Help 1]
[INFO] [ERROR]
[INFO] [ERROR]   The project ch.vorburger.mariaDB4j:mariadb4j-maven-plugin-test-basic:[unknown-version] (/home/travis/build/vorburger/MariaDB4j/mariaDB4j-maven-plugin/target/it/mariadb4j-maven-plugin-test-basic/pom.xml) has 1 error
[INFO] [ERROR]     Non-resolvable parent POM for ch.vorburger.mariaDB4j:mariadb4j-maven-plugin-test-basic:[unknown-version]: Could not find artifact ch.vorburger.mariaDB4j:mariaDB4j-pom:pom:2.4.0-SNAPSHOT in local.central (file:///home/travis/.m2/repository) and 'parent.relativePath' points at wrong local POM @ line 6, column 13 -> [Help 2]
[INFO] [ERROR]
[INFO] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[INFO] [ERROR] Re-run Maven using the -X switch to enable full debug logging.
[INFO] [ERROR]
[INFO] [ERROR] For more information about the errors and possible solutions, please read the following articles:
[INFO] [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[INFO] [ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException
[INFO]   The build exited with code 1. See /home/travis/build/vorburger/MariaDB4j/mariaDB4j-maven-plugin/target/it/mariadb4j-maven-plugin-test-basic/build.log for details.
[INFO]           mariadb4j-maven-plugin-test-basic/pom.xml ........ FAILED (2.0 s)
[INFO] Building: start-stop/pom.xml

Would anyone watching this be willing to step up and help maintaing this project, see #213 ?
@srbala  @glittle1972  @mdindoffer  @yiftizur @jtmuhone

I believe this is already fixed by my PR from April https://github.com/vorburger/MariaDB4j/pull/248
As for the patronage I'd have to think about it, but there are probably better candidates.

I believe this is already fixed by my PR from April #248

@mdindoffer indeed it does - but when I originally glanced over your PR I hadn't understood why..
Just FTR, the problem here was that we're at <version>2.4.1-SNAPSHOT everywhere, but somehow mariaDB4j-maven-plugin/src/it/mariadb4j-maven-plugin-test-basic/pom.xml was left behind.
It would be nice if this was caught earlier in the future; I wonder if #263 will help to prevent this.


Revert "updated NPM dependencies in json file"

Reverts mgechev/angular2-seed#177

What was the reason for this revert?


Sure you want to delete?

What version of WebTorrent Desktop?
Version 0.17.2 (0.97.2)
What operating system and version?
Mac OS
What did you do?
Right click -> Remove Data Files
What did you expect to happen?
The "Remove data" button should use the autofocus attribute
What actually happened?
The "are you sure you want to delete this" is a bit annoying. I'm 99.9999% sure  sure about my choice so i just want to hit space bar with my other hand where the thumb is laying perfectly on spacebar most of the time just waiting to be pressed down.
Even better would be a "Don't ask again" option

I think this makes sense, so made a PR #1058.
For the 'Don't ask again' option etc. please follow discussion at #970.

Thanks @mathiasvr!


added feature to notify about recorded user interaction

added a feature to let the user know weather his interaction has been recorded or not

Now has conflicts

This needs to be updated to address conflicts.

ready for review

Just some general feedback. I think this should be optional (it might end up being annoying). I'm also wondering if #63 is a better approach, we could do the notifications in that dialogue instead (showing a list of recorded events would be handy too). I'd have liked to be notified of ZestClientLaunch since that's one of the elements I see missing in the recordings sometimes.

I think you are right we can provide that notification inside floating window itself.


Scrape 2023 schedule

Schedule will be released 5/11

Need to re write scarper ugh

Done, added to database, running deploy now


Mail auth

Hey, Marc3842h!

Do not login to the account if it has mail authentication.
Make it possible with rb / cb to select the checkboxes which accounts to use.

You can also make your own event menu instead of the new cmd that opens.exe

Steam Guard login works but the current release 1.6.0 got a bug that prevents logging in from Windows systems.
 Aren't the available indexes already enough?
Also, I do not quite understand what you mean with "event menu".

Check this private reportbot from russian coder (https://cloud.mail.ru/public/4P8H/kTHhtTVf6) . Here u will see what is "event log". Do the same :) It would be nice if you still changed the skin of the program to about the same as in Linux.


PowerToys Run: Cannot be scrolled using touch

Microsoft PowerToys version
0.82.0
Installation method
PowerToys auto-update
Running as admin
Yes
Area(s) with issue?
PowerToys Run
Steps to reproduce

Open PowerToys Run dialog.
Scroll the list using the up or down swipe gesture on the touch screen.

✔ Expected Behavior
It should be scrolled using touch input.
❌ Actual Behavior
It is not working.
Other Software
No response

I believe this is caused by a bug in the WPF UI library.

Seems to be a duplicate of https://github.com/microsoft/PowerToys/issues/7528. The fix should be pretty simple - just need to set the ScrollViewer.PanningMode on the ScrollViewer.


Bottomsheet changes

BottomSheet configuration changes.

this pr can be canceled, according to the conversation with the team


Some data errors

Hi. Thanks for writing this tool - I couldn't get the python one working, so it's really appreciated.
I'm seeing these data issues. Is anyone else? Any tips on how to overcome them?
Thanks in advance.
2017-09-27 01:44:27,245 INFO  MySqlIngester - Beginning full ingest of epf_paid_application_popularity_per_genre (3169192 records)
2017-09-27 01:46:27,251 INFO  MySqlIngester - ...at record 923600...
2017-09-27 01:48:27,263 INFO  MySqlIngester - ...at record 1844200...
2017-09-27 01:50:27,266 INFO  MySqlIngester - ...at record 2766400...
2017-09-27 01:51:19,345 ERROR Program - An error occured while ingesting data.
java.util.IllegalFormatConversionException: d != java.lang.String

2017-09-27 01:51:19,347 WARN  Program - The following files encountered errors and were not imported:
        itunes/popularity20170913/[album_popularity_per_genre, paid_ipad_application_popularity_per_genre, free_ipad_application_popularity_per_genre, song_popularity_per_genre, free_application_popularity_per_genre, paid_application_popularity_per_genre]

and
ÏÎ·ÏÎµÏÎ¯Î± i-Status ÎµÎ¯Î½Î±Î¹ ÏÏÏÎ¯Ï ÎµÏÎ¹ÏÎ»ÎÎ¿Î½ ÏÏÎÏÏÎ· ÎºÎ±Î¹ Î¼ÏÎ¿ÏÎµÎ¯ Î½Î± ÎµÎ½ÎµÏÎ³Î¿ÏÎ¿Î¹Î·Î¸ÎµÎ¯ Î³Î¹Î± Ï
                                                                                Î¹Ï online ÎµÎ³ÎºÎ±Ï
                                                                                Î±ÏÏ
                                                                                Î¬ÏÎµÎ¹Ï Ï
                                                                                ÏÎ½ ÎµÏÎ±ÏÎ¼Î¿Î³ÏÎ½ Status Î¼Îµ Ï
          ÎºÎ±Î¹ username.', 'New stuff!!', NULL, 'http://www.mainsys.gr', 'http://is3.mzstatic.com/image/thumb/Purple1/v4/a8/f5/26/a8f526ab-d2bc-4a07-cad0-89017fd613b6/source/320x480bb.jpg', NULL, NULL, NULL, '320x480', NULL, NULL, NULL, 'http://is1.mzstatic.com/image/thumb/Purple3/v4/e2/e0/9b/e2e09b60-642a-980f-837f-240c7ec108ae/source/360x480bb.jpg', NULL, NULL, NULL, '360x480', NULL, NULL, NULL)
com.mysql.jdbc.MysqlDataTruncation: Data truncation: Incorrect string value: '\xE2?\xA2 In...' for column 'title' at row 18
 at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4206)
 at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4140)
 at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2597)

Hi @renewooller, this might not be an issue with the tool... it might be an issue with the encoding on the mysql server. Do you have utf8mb4 encoding enabled?

Thanks - I'll try mb4

I have tried setting the charset and collation for the db like so:
ALTER DATABASE epf CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

and
ALTER DATABASE epf CHARACTER SET utf8mb4 COLLATE utf8mb4_bin;

Running the importer just on the 'song' table gets the problem immediately. The song table does not exist yet - so no need to alter the encoding for it.
This is the command I'm using:
docker run --rm -it -v "$PWD:/app" -v "$PWD/../itunes:/app/itunes" -w /app openjdk:7-jre-alpine java -Xmx1024m -jar ./target/epfimporter-0.0.1-SNAPSHOT-jar-with-dependencies.jar -d dev-db-slim.nightlifr.com -u admin -p password -n epf -w '^song$' itunes/itunes20170913

I've had similar problems in other projects before when trying to support emojis - I recall the encoding on the JDBC driver needing to be set. Let me know if you can replicate the problem.
FYI to build the jar I'm using:
git clone https://github.com/billy1380/JavaEPFImporter.git;
cd JavaEPFImporter;
docker run -it --rm --name my-maven-project -v "$PWD":/usr/src/mymaven -w /usr/src/mymaven maven:3.2-jdk-7 mvn package;

I think this may be the problem:
show variables like 'character_set_server';

character_set_server	latin1

This is working now, as it it's not crashing. However, the © symbol in 'copyright' column is showing up as â, and various artist names also have incorrect encodings, eg "Wiener Philharmoniker & Karl BÃ¶hm" or "Augustin Dumay & Maria JoÃ£o Pires"

I think the line in Connection.java
 executeQuery("set names \'utf8\'");

should be:
executeQuery("set names \'utf8mb4\'");

Will test and let you know.

I think that will work, it certainly rings a bell... I think I have made that change somewhere. When you are happy with the results, please do a pull request and I will merge it straight away.
Thanks for going through it.

According to these:
http://info.michael-simons.eu/2013/01/21/java-mysql-and-multi-byte-utf-8-support/
https://mathiasbynens.be/notes/mysql-utf8mb4
having "set names utf8mb4" should have solved it but it doesn't seem to. I've also tried useUnicode=true and characterEncoding=utf-8 and changing all the relevant variables on the server. Nothing seems to be able to get around the problem.
It could be that there are some characters that aren't valid utf8mb4 in the files, and so it's defaulting to a more lenient encoding.
Eg when I try:
mysql --local-infile -h <IP_ADDRESS> -u -p epf \

-e "LOAD DATA LOCAL INFILE './itunes/itunes20170913/artist'
INTO TABLE artist CHARACTER SET utf8mb4
FIELDS TERMINATED BY X'01'
LINES TERMINATED BY '\n'
IGNORE 34 LINES"
ERROR 1300 (HY000) at line 1: Invalid utf8mb4 character string: '张雨生'

Looking into this further it appears that this character is a 6-byte utf8 character: http://graphemica.com/张
Mysql only supports up to 4 byte. So I don't think there's any way of dealing with this one.

Only looks like 3 bytes to me 0xE5 0xBC 0xA0 (e5bca0)

I have this working now:
mysql --local-infile -h <IP_ADDRESS> -u <> -p<> epf \
-e "LOAD DATA LOCAL INFILE './itunes/test/itunes20170913/artist' \
INTO TABLE artist CHARACTER SET utf8mb4 \
FIELDS TERMINATED BY X'01' ESCAPED BY '' \
LINES TERMINATED BY X'020A' \
IGNORE 34 LINES"

The important part being  " escaped by '' " , which overrides the default ''
I wonder if the same issue is behind the fallback to latin-1 in the java importer?

yah, I got the nybble confused with the byte, silly me.

I am trying to think if anything has changed... I tested with data in the smalldata folder (which contains some kangi) do you mind doing me a favour and checking that e.g. https://github.com/billy1380/JavaEPFImporter/blob/master/smalldata/application_detail
imports for you with no trouble.


Authentication fails after update to vdirsyncer 0.4.1

I use vdirsyncer with ownCloud 7.0.4 on a webfaction server. After updating vdirsyncer to version 0.4.1 vdirsyncer does authenticate anymore. With former versions there was no problem and I haven't changed configuration. Until now I used a url = https://oc.[USERNAME].webfactional.com/remote.php/caldav/calendars/[OWNCLOUDUSERNAME]/  together with verify = false and auth=guess.
DEBUG output:
debug: Using 0 maximal workers.
Discovering collections for pair niels_calendar
debug: GET https://oc.[USERNAME].webfactional.com/.well-known/caldav/
debug: {}
debug: None
debug: Sending request...
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:734: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
debug: 302
debug: {'content-length': '242', 'server': 'nginx', 'connection': 'keep-alive', 'location': 'http://oc.[USERNAME].webfactional.com/remote.php/caldav/', 'date': 'Wed, 14 Jan 2015 09:25:36 GMT', 'content-type': 'text/html; charset=iso-8859-1'}
debug: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
debug:
debug: <title>302 Found</title>
debug:
debug:
Found

debug:
The document has moved here.

debug:
debug: PROPFIND http://oc.[USERNAME].webfactional.com/remote.php/caldav/
debug: {'Depth': 0, 'Content-Type': 'application/xml; charset=UTF-8', 'User-Agent': 'vdirsyncer'}
debug:
debug:         <d:propfind xmlns:d="DAV:">
debug:             <d:prop>
debug:                 <d:current-user-principal />
debug:             </d:prop>
debug:         </d:propfind>
debug:
debug: Sending request...
debug: 405
debug: {'date': 'Wed, 14 Jan 2015 09:25:36 GMT', 'content-length': '166', 'content-type': 'text/html', 'connection': 'keep-alive', 'server': 'nginx'}
debug:
debug: <title>405 Not Allowed</title>
debug:
debug:
405 Not Allowed

debug:
nginx
debug:
debug:

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdirsyncer/cli/utils.py", line 155, in _discover_from_config
    discovered = list(cls.discover(**config))
  File "/usr/lib/python2.7/site-packages/vdirsyncer/storage/dav.py", line 317, in discover
    for c in d.discover():
  File "/usr/lib/python2.7/site-packages/vdirsyncer/storage/dav.py", line 173, in discover
    for x in self.find_collections():
  File "/usr/lib/python2.7/site-packages/vdirsyncer/storage/dav.py", line 150, in find_collections
    url = self.find_home()
  File "/usr/lib/python2.7/site-packages/vdirsyncer/storage/dav.py", line 133, in find_home
    url = self.find_principal()
  File "/usr/lib/python2.7/site-packages/vdirsyncer/storage/dav.py", line 124, in find_principal
    data=body, is_subpath=False)
  File "/usr/lib/python2.7/site-packages/vdirsyncer/storage/dav.py", line 242, in request
    return utils.request(method, url, session=self._session, **more)
  File "/usr/lib/python2.7/site-packages/vdirsyncer/utils/__init__.py", line 239, in request
    r.raise_for_status()
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 831, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
HTTPError: 405 Client Error: Not Allowed
critical: Failed to initialize cutuchiqueno_calendar_remote.

I notice two things: 1) There is no .well-known folder not in my webfactional root folder and not in the webapp folder for ownCloud (There was none before when it worked). For some reason – maybe related to the former – vdirsyncer is looking for non https urls. This can not work because there is nothing there, it leads to a webfaction service page and thus it is blocked.
I also tried with verify_fingerprint but there was no differance in the result.
As I have stated before this happened after updating vdirsyncer to the current version. So there seem to be changes in vdirsyncer I can not or don't know how to workaround.

debug: The document has moved here.

You did set up well-known URIs somehow, they redirect to the HTTP version of
your site. This is usually done in your nginx config.
well-known URIs have been added in version 0.4.0 -- I did some changes on
version 0.4.1 which make the behavior stricter but also faster. However, it's
IMO a good thing that this problem came up, since it shows a misconfiguration.

Also I strongly recommend you use verify_fingerprint = ... if your certificate is intentionally invalid.

Informed by your hints I figured out that the problem is the default .htaccess configuration provided by owncloud itself. There are some redirects which should bridge the change of the url ownCloud made in a former version After I commented these out everything works again.
Since the source of the problem is the owncloud installation itself an more people could be affected, it would maybe good add a comment to the tutorial in the documentation which has ownCloud as its use case.
I am using verify_fingerprint now, thx, you are absolutely right.

Glad to hear that it works!

Since the source of the problem is the owncloud installation itself an more people could be affected, it would maybe good add a comment to the tutorial in the documentation which has ownCloud as its use case.

I don't know... these redirects are there for a reason, if they redirect to the wrong port/protocol, it's rather a bug in ownCloud. I will have to modify vdirsyncer to deal with such invalid redirects properly again though.

yes, I agree it is probably a bug in ownCloud. I just thought it could be helpful in a tutorial using ownCloud as an example (probably a very common situation for vdirsyncer).
But if you manage vdirsyncer to deal with those situations it's even more perfect ;-)
Thx!

FWIW, it would be interesting if you managed to actually fix the redirection behavior instead of disabling it. I can't see anything in the .htaccess for 7.0.4 that would cause this bug, which is why I am not sure how it happens on your side.

Actually the line that causes the problem is:
# RewriteRule ^\.well-known/caldav /remote.php/caldav/ [R]

7.0.4 was not the originally installation but the upgrade. I think the original install was 7.0.2 deployed as a webapp automatic  installation provided by webfacitonal.

Ok that line is still there, but for a good reason. I'm not sure if it is either WebFaction or ownCloud who screws up there...