Support Python >= 3.3 in Dataproc (ensure spark-submit shell has access to global env vars)

Hi dataproc team.
tl;dr
Hope this title isn't too bombastic, but it seems dataproc cannot support PySpark workloads in Python version 3.3 and greater. This stems from PySpark checking for a PYTHONHASHSEED env var that, while set, is not detected during execution of spark jobs on a Dataproc cluster. This occurs with other environment variables in different runtimes as well (e.g., remote spark job submittal with a property setting for PYSPARK_PYTHON, see below).
0. details
PySpark's rdd.py module requires a set PYTHONHASHSEED env var for Python 3.3 and higher, ensuring a common seed among workloads distributed across multiple Python instances.
While there is a discussion of whether or not PySpark should set PYTHONHASHSEED on its own, ultimately, PySpark workloads and applications based on Python 3.3 and higher must have PYTHONHASHSEED set. This can be done by setting exports in global profiles or more concisely, in the spark-env.sh config. Some examples of this:

Python 3 on Spark - Return of the PYTHONHASHSEED, and
Set python3 PYTHONHASHSEED for spark + python3 compatibility

1. Setup
To demonstrate, I've implemented a basic shell script that also installs Python 3, like this:
❯❯ wget https://gist.githubusercontent.com/nehalecky/9258c01fb2077f51545a/raw/789f08141dc681cf1ad5da05455c2cd01d1649e8/install-py3-dataproc.sh
❯❯ cat install-py3-dataproc.sh
#!/bin/bash
apt-get -y install python3
echo "export PYSPARK_PYTHON=python3" | tee -a  /etc/profile.d/spark_config.sh  /etc/*bashrc
echo "export PYTHONHASHSEED=123" | tee -a  /etc/profile.d/spark_config.sh  /etc/*bashrc /usr/lib/spark/conf/spark-env.sh
source ~/.bashrc

and target this in as an init action in launching a minimal dataproc cluster
❯❯ gcloud beta dataproc clusters create py3-test
    --initialization-actions \
        gs://bombora-dev-analytics/dataproc-init-actions/install-py3-dataproc.sh #public object

Logging in, one can confirm that PYTHONHASHSEED env var is set across both global profiles along with spark-env.sh config, across both master and workers nodes.
master
❯❯  gcloud compute ssh root@py3-test-m
root@spark-cluster-m:~# echo $PYTHONHASHSEED
123
root@py3-test-m:~# echo $PYSPARK_PYTHON
python3

worker
❯❯  gcloud compute ssh root@py3-test-w-0
root@py3-test-w-0:~# echo $PYTHONHASHSEED
123
root@py3-test-w-0:~#  echo $PYSPARK_PYTHON
python3

2. Testing
To test, I have a simple python script that outputs info on PYTHONHASHSEED, raises an exception if not detected, and attempts a pyspark job to printout the python version detected by executors.
root@spark-cluster-m:~# wget https://raw.githubusercontent.com/nehalecky/dataproc-initialization-actions/feature/conda_init_action/develop/conda/get-sys-exec.py
root@spark-cluster-m:~# cat get-sys-exec.py
import pyspark
import sys
import os

pyhashseed = os.environ['PYTHONHASHSEED']
print(pyhashseed)
print(type(pyhashseed))
print(sys.version)

if sys.version >= '3.3' and 'PYTHONHASHSEED' not in os.environ:
    raise Exception("Randomness of hash of string should be disabled via PYTHONHASHSEED")

else:
   sc = pyspark.SparkContext()
   distData = sc.parallelize(range(100))
   python_distros = distData.map(lambda x: sys.executable).distinct().collect()
   print(python_distros)

Local
I run this locally on the cluster, with a call to spark-submit. While able to read and print out the correct setting for PYTHONHASHSEED, it later hits an exception raised by the exact same logic that it initially passed! Seems that the env var is detected on the master node fine, as  the exception is raised on the worker node. Crazy.
root@spark-cluster-m:~# spark-submit get-sys-exec.py
123
<class 'str'>
3.5.1 |Continuum Analytics, Inc.| (default, Dec  7 2015, 11:16:01)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
16/01/18 01:18:03 INFO akka.event.slf4j.Slf4jLogger: Slf4jLogger started
16/01/18 01:18:03 INFO Remoting: Starting remoting
16/01/18 01:18:03 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@<IP_ADDRESS>:47010]
16/01/18 01:18:04 INFO org.spark-project.jetty.server.Server: jetty-8.y.z-SNAPSHOT
16/01/18 01:18:04 INFO org.spark-project.jetty.server.AbstractConnector: Started SocketConnector@<IP_ADDRESS>:60173
16/01/18 01:18:04 INFO org.spark-project.jetty.server.Server: jetty-8.y.z-SNAPSHOT
16/01/18 01:18:04 INFO org.spark-project.jetty.server.AbstractConnector: Started SelectChannelConnector@<IP_ADDRESS>:4040
16/01/18 01:18:04 WARN org.apache.spark.metrics.MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.
16/01/18 01:18:04 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at spark-cluster-m/<IP_ADDRESS>:8032
16/01/18 01:18:06 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl: Submitted application application_1453072376140_0005
[Stage 0:>                                                          (0 + 2) / 2]16/01/18 01:18:16 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 1.0 in stage 0.0 (TID 1, spark-cluster-w-0.c.bombora-dev.internal): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/hadoop/yarn/nm-local-dir/usercache/root/appcache/application_1453072376140_0005/container_1453072376140_0005_01_000002/pyspark.zip/pyspark/worker.py", line 111, in main
    process()
  File "/hadoop/yarn/nm-local-dir/usercache/root/appcache/application_1453072376140_0005/container_1453072376140_0005_01_000002/pyspark.zip/pyspark/worker.py", line 106, in process
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/hadoop/yarn/nm-local-dir/usercache/root/appcache/application_1453072376140_0005/container_1453072376140_0005_01_000002/pyspark.zip/pyspark/serializers.py", line 133, in dump_stream
    for obj in iterator:
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/rdd.py", line 1704, in add_shuffle_key
  File "/hadoop/yarn/nm-local-dir/usercache/root/appcache/application_1453072376140_0005/container_1453072376140_0005_01_000002/pyspark.zip/pyspark/rdd.py", line 74, in portable_hash
    raise Exception("Randomness of hash of string should be disabled via PYTHONHASHSEED")
Exception: Randomness of hash of string should be disabled via PYTHONHASHSEED

    at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:166)
    at org.apache.spark.api.python.PythonRunner$$anon$1.<init>(PythonRDD.scala:207)
    at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:125)
    at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
    at org.apache.spark.api.python.PairwiseRDD.compute(PythonRDD.scala:342)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Remote
Remote execution completes because, regardless of any setting of PYSPARK_PYTHON env var, it uses the default Python installation (2.7.9). Again, this is likely due to not being able to reference the globally set env vars.
>> gcloud beta dataproc jobs submit pyspark --cluster py3-test get-sys-exec.py
Copying file://get-sys-exec.py [Content-Type=text/x-python]...
Uploading   ...33ab-c06f-4234-b495-92c3bf9ac6e0/get-sys-exec.py: 480 B/480 B
Job [6512a774-fc83-4d2c-b735-f40fa7bac534] submitted.
Waiting for job output...
123
<type 'str'>
2.7.9 (default, Mar  1 2015, 12:57:24)
[GCC 4.9.2]
16/01/18 19:04:30 INFO akka.event.slf4j.Slf4jLogger: Slf4jLogger started
16/01/18 19:04:30 INFO Remoting: Starting remoting
16/01/18 19:04:30 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@<IP_ADDRESS>:43622]
16/01/18 19:04:30 INFO org.spark-project.jetty.server.Server: jetty-8.y.z-SNAPSHOT
16/01/18 19:04:30 INFO org.spark-project.jetty.server.AbstractConnector: Started SocketConnector@<IP_ADDRESS>:52135
16/01/18 19:04:31 INFO org.spark-project.jetty.server.Server: jetty-8.y.z-SNAPSHOT
16/01/18 19:04:31 INFO org.spark-project.jetty.server.AbstractConnector: Started SelectChannelConnector@<IP_ADDRESS>:4040
16/01/18 19:04:31 WARN org.apache.spark.metrics.MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.
16/01/18 19:04:31 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at py3-test-m/<IP_ADDRESS>:8032
16/01/18 19:04:33 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl: Submitted application application_1453141822766_0002
['/usr/bin/python']
16/01/18 19:04:43 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
16/01/18 19:04:43 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
Job [6512a774-fc83-4d2c-b735-f40fa7bac534] finished successfully.
driverControlFilesUri: gs://dataproc-e66c6e3d-80da-4211-b66c-0109bbe4567a-us/google-cloud-dataproc-metainfo/44b233ab-c06f-4234-b495-92c3bf9ac6e0/jobs/6512a774-fc83-4d2c-b735-f40fa7bac534/
driverOutputResourceUri: gs://dataproc-e66c6e3d-80da-4211-b66c-0109bbe4567a-us/google-cloud-dataproc-metainfo/44b233ab-c06f-4234-b495-92c3bf9ac6e0/jobs/6512a774-fc83-4d2c-b735-f40fa7bac534/driveroutput
placement:
  clusterName: py3-test
  clusterUuid: 44b233ab-c06f-4234-b495-92c3bf9ac6e0
pysparkJob:
  loggingConfiguration: {}
  mainPythonFileUri: gs://dataproc-e66c6e3d-80da-4211-b66c-0109bbe4567a-us/google-cloud-dataproc-staging/44b233ab-c06f-4234-b495-92c3bf9ac6e0/get-sys-exec.py
reference:
  jobId: 6512a774-fc83-4d2c-b735-f40fa7bac534
  projectId: bombora-dev
status:
  state: DONE
  stateStartTime: '2016-01-18T19:04:50.779Z'
statusHistory:
- state: PENDING
  stateStartTime: '2016-01-18T19:04:24.164Z'
- state: SETUP_DONE
  stateStartTime: '2016-01-18T19:04:24.320Z'
- details: Agent reported job success
  state: RUNNING
  stateStartTime: '2016-01-18T19:04:29.613Z'

Attempt at passing in a --properties argument has no impact (and note the warning for ignoring PYSPARK_PYTHON).
gcloud beta dataproc jobs submit pyspark --cluster py3-test --properties PYSPARK_PYTHON=/usr/bin/python3 get-sys-exec.py                                                                          ✱ ◼
Copying file://get-sys-exec.py [Content-Type=text/x-python]...
Uploading   ...33ab-c06f-4234-b495-92c3bf9ac6e0/get-sys-exec.py: 480 B/480 B
Job [2ba2a5b3-3e4a-41ce-b463-6970185b03bf] submitted.
Waiting for job output...
Warning: Ignoring non-spark config property: PYSPARK_PYTHON=/usr/bin/python3
123
<type 'str'>
2.7.9 (default, Mar  1 2015, 12:57:24)
[GCC 4.9.2]
16/01/18 19:33:25 INFO akka.event.slf4j.Slf4jLogger: Slf4jLogger started
16/01/18 19:33:25 INFO Remoting: Starting remoting
16/01/18 19:33:25 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@<IP_ADDRESS>:56989]
16/01/18 19:33:26 INFO org.spark-project.jetty.server.Server: jetty-8.y.z-SNAPSHOT
16/01/18 19:33:26 INFO org.spark-project.jetty.server.AbstractConnector: Started SocketConnector@<IP_ADDRESS>:44267
16/01/18 19:33:26 INFO org.spark-project.jetty.server.Server: jetty-8.y.z-SNAPSHOT
16/01/18 19:33:26 INFO org.spark-project.jetty.server.AbstractConnector: Started SelectChannelConnector@<IP_ADDRESS>:4040
16/01/18 19:33:26 WARN org.apache.spark.metrics.MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.
16/01/18 19:33:26 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at py3-test-m/<IP_ADDRESS>:8032
16/01/18 19:33:28 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl: Submitted application application_1453141822766_0005
['/usr/bin/python']
Job [2ba2a5b3-3e4a-41ce-b463-6970185b03bf] finished successfully.
driverControlFilesUri: gs://dataproc-e66c6e3d-80da-4211-b66c-0109bbe4567a-us/google-cloud-dataproc-metainfo/44b233ab-c06f-4234-b495-92c3bf9ac6e0/jobs/2ba2a5b3-3e4a-41ce-b463-6970185b03bf/
driverOutputResourceUri: gs://dataproc-e66c6e3d-80da-4211-b66c-0109bbe4567a-us/google-cloud-dataproc-metainfo/44b233ab-c06f-4234-b495-92c3bf9ac6e0/jobs/2ba2a5b3-3e4a-41ce-b463-6970185b03bf/driveroutput
placement:
  clusterName: py3-test
  clusterUuid: 44b233ab-c06f-4234-b495-92c3bf9ac6e0
pysparkJob:
  loggingConfiguration: {}
  mainPythonFileUri: gs://dataproc-e66c6e3d-80da-4211-b66c-0109bbe4567a-us/google-cloud-dataproc-staging/44b233ab-c06f-4234-b495-92c3bf9ac6e0/get-sys-exec.py
  properties:
    PYSPARK_PYTHON: /usr/bin/python3
reference:
  jobId: 2ba2a5b3-3e4a-41ce-b463-6970185b03bf
  projectId: bombora-dev
status:
  state: DONE
  stateStartTime: '2016-01-18T19:33:45.155Z'
statusHistory:
- state: PENDING
  stateStartTime: '2016-01-18T19:33:20.791Z'
- state: SETUP_DONE
  stateStartTime: '2016-01-18T19:33:20.947Z'
- details: Agent reported job success
  state: RUNNING
  stateStartTime: '2016-01-18T19:33:27.724Z'

3. Hacking
In one last desperate attempt to get things working, I modified the /usr/lib/spark/python/pyspark/rdd.py module to statically define the env var by changing line 74 to this:
os.environ['PYTHONHASHSEED'] = 0
warnings.warn('Environment variable PYTHONHASHSEED not detected, set to 0')
#raise Exception("Randomness of hash of string should be disabled via PYTHONHASHSEED")

This, however, had no effect, which was totally unexpected until I realized that code base wasn't being called. Instead, the traceback references a different directory at: /usr/lib/spark/python/lib/pyspark.zip/pyspark/rdd.py, which indeed I had not modified.
4. Fin
I've exhausted all leads on my side, wanted to hand this off, and hope this helps in identifying and resolving this issue. Appreciate your time and please let me know how more I can help.
Many thanks. :)

@nehalecky first, thanks so much for all the work on the conda init-action! I think I have some good news: after lots (and lots) of banging my head on my desk, I've got a solution to the PYTHONHASHSEED issue.
You need to add this property to spark-defaults.conf:
spark.executorEnv.PYTHONHASHSEED=0
This is, as far as I can tell, the only way to share environment with the executors in YARN deploy modes. Since PYTHONHASHSEED is basically required for Py3 deployments -- as we've learned the hard way -- I'm not sure why it wouldn't be set/available in those environments by default. Feels like an oversight.
Two ways to set this:

When creating a dataproc cluster, add it as a property (for example gcloud beta dataproc clusters create --properties spark:spark.executorEnv.PYTHONHASHSEED=0 ...)

Explicitly add it to the conf file during startup. I put this line in my initialization action:
echo "spark.executorEnv.PYTHONHASHSEED=0" >> spark-defaults.conf

As far as I can tell, you CAN'T add it as a property when submitting a job to dataproc. gcloud will reject it, saying Warning: Ignoring non-spark config property: spark:spark.executorEnv.PYTHONHASHSEED=0. Maybe the strange format (the env var is part of the key, not part of the value, so it's a dynamic property) is why it's rejected. ccing @dennishuo @evilsoapbox in case I'm just making a mistake.
So far, this has worked in my very limited testing. Please let me know if it works for you!

Reviewing some of the other comments on PYSPARK_PYTHON -- I actually haven't run into this issue (weird -- but I'm using 1.6.0 in the preview image and maybe the issue has been fixed there) but there is a complementary property in spark-defaults.conf called  spark.yarn.appMasterEnv. I think it might work to set spark.yarn.appMasterEnv.PYSPARK_PYTHON=/path/to/python. Since it's also a dynamic property, I'm guessing it'll work when creating a cluster but not when submitting a job -- maybe best to write it explicitly.

@jlowin, thank you much for the reply—I can sympathize with struggle (and hope your head doesn't hurt too much).
I finally got around to testing this, what an amazing find with spark.executorEnv.PYTHONHASHSEED! I was able to resolve the issue using the --properties parameter on cluster creation, however, I couldn't get it to work using your PR. I'm going to make a few notes on the PR, so I'll hop over there now.

@jlowin Oh my, this actually worked. The errors are gone.
I need to check if it worked well, but I believe it did :)

Another way around this issue is using: export SPARK_YARN_USER_ENV=PYTHONHASHSEED=0 on the master node. This will propagate to the executors.

This is also now "fixed" in https://issues.apache.org/jira/browse/SPARK-13330 but won't be released until Spark 2.2.0.

Closing this issue as Dataproc 1.2 (new default) has Spark 2.2


How can I use a public OSS bucket without any secret.

I want to use the helm chart https://starship-charts.oss-cn-hangzhou.aliyuncs.com/charts/nacos-2.0.3.tgz as source chart without any secret, because this is public bucket.

Hi, since this is a packaged chart, you can create a helm repo index for it, following the docs here https://helm.sh/docs/helm/helm_repo_index/ . And then you can create a HelmRepository object that points to the index. A HelmChart object can refer to the helm chart in the created HelmRepository and use it.
If it was an unpackaged helm chart, you could create a Bucket object and HelmChart object refer to the bucket object to get the chart.
HelmChart expects the reference to be either a GitRepository, a Bucket or a HelmRepository. Refer https://fluxcd.io/docs/components/helm/helmreleases/#helm-chart-template for use with a HelmRelease.

@darkowlzz Hi, thanks for your patient answer. My question truly is  this  helm chart store in a public bucket, everyone can fetch it without an auth token. But when I create a bucket object pointing to this chart without token will generate an error that I must set one. Can I just want to use this bucket without any secrets?


Update views.md

Just adding the using directive for the AuthorizationService injection.
When creating a new PR, please do the following and delete this template text:

Reference the issue number if there is one:
Fixes #Issue_Number

The "Fixes #nnn" syntax in the PR description causes
GitHub to automatically close the issue when this PR is merged.

Suggest reviewers if you know who should review the PR, using the GitHub Reviewers UI.

This seems like a small (but important) contribution, so no Contribution License Agreement is required at this point. We will now review your pull request.
Thanks,
.NET Foundation Pull Request Bot

@TinkeringAlways thanks.


Hosts file changes

@simplesteph could you explain to me why do I have to change hosts file on machine where I run app that connect to kafka? I guess it's only required for HOST where kafka is running.

Hi @cojack
Kafka has a concept of advertised host, and when you talk to kafka, it will ask you to communicate using the advertised host (all of that is done by the client). So as I set the advertised host to kafka1 for the first broker, when your programs connect to kafka1, they actually need to know that is also means localhost, hence the /etc/hosts file changes.
Makes sense?

@simplesteph nope I does not explain me anything xD But don't worry, I don't have to know everything, just figured out that npm package no-kafka allows me to redirect brokers so I just map kafka{N} to correct host without touching the /etc/hosts files. Thanks a lot for your work with this easy docker kafka setup!


[BUG]

What is the bug?
My scoreboard disappeared when installing the betterteams plugin
Expected Behaviour
Please fix this bug problem
How to Reproduce

I installed the betterteams plugin 2. After I installed it and tried to enter the server suddenly the scoreboard disappeared, when I asked the member on the server he said it suddenly just disappeared.

Error message
No response
Output of /teama version
4.9.4
Additional Information
No response

See this wiki page: https://github.com/booksaw/BetterTeams/wiki/Managing-the-TAB-Menu#conflict


Add menu option for dark/light theme

Merged into https://github.com/OEFW-community/RT-890-custom-firmware/pull/13


problems with groundPrimitives in recent chrome

Just noticed this last week with chrome, it may have started earlier  , pulled the following demo code from issue #3710, things render fine in firefox,
the rectangle geometry appears to be projecting down towards the center of the earth and is rendering above the ground
Windows 7, chrome 49.0.2623.110 m,

`
var viewer = new Cesium.Viewer("cesiumContainer", {
    baseLayerPicker: false,
    terrainProvider: new Cesium.CesiumTerrainProvider({
        url: "//assets.agi.com/stk-terrain/world"
    })
});

var rect = Cesium.Rectangle.fromDegrees(-85.0, 35.99, -84.99, 36.0);
    viewer.camera.setView({destination: rect});

Sandcastle.addToolbarButton('Bug', function () {
    viewer.scene.groundPrimitives.add(new Cesium.GroundPrimitive({
        geometryInstances : new Cesium.GeometryInstance({
            geometry : new Cesium.RectangleGeometry({
                rectangle : rect
            }),
            attributes: {
                color: Cesium.ColorGeometryInstanceAttribute.fromColor(new Cesium.Color(1.0, 1.0, 0.0, 0.5))
            }
        }),
        asynchronous : false
    }));
});

`

Hello @bmckilligan.  Have you tried updating to Cesium 1.20?  We fixed a ground primitive issue in that release that sounds a lot like the one you're seeing. (#3706)

yes, appears to be in 1.20 also, the screen grab was from sandcastle showing Cesium.VERSION 1.20

Okay thanks!  And thanks for providing a code sample and all of the details.  That will make it easier for us to look into the problem

I noticed it late last week when using cesium 1.19 in Chrome.
just looked at http://googlechromereleases.blogspot.ca/  .  I suspect the culprit is there
Stable Channel Update for Chrome OS
Thursday, March 31, 2016
The Stable channel has been updated to 49.0.2623.111 (Platform version: 7834.66.0/7834.67.0) for all Chrome OS devices. This build contains a number of bug fixes, an update to Adobe Flash Player (<IP_ADDRESS>-r1) and security fixes. Systems will be receiving updates over the next several days.

any luck on resolving this ?  has a bug been filed with bugs.chromium.org ?

any word on this, Its still present in Chrome 50.0.2661.94 m (64-bit) ?
has a  Chromium bug been submitted ?

@bmckilligan as far as I know, no bug has been submitted to the Chrome team (not 100% sure if it's a Chrome bug or not yet); I've seen this problem randomly in the past but I can't reproduce it with the code you provided.  Does this happen every time for you?  Can you copy/paste the results of going to http://webglreport.com?  That might help narrow down the issue.

I just tried the code in a 32-bit version of chrome and things worked,
so it looks like it may be a chrome 64-bit issue

Platform:	Win32
Browser User Agent:	Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36
Context Name:	webgl
GL Version:	WebGL 1.0 (OpenGL ES 2.0 Chromium)
Shading Language Version:	WebGL GLSL ES 1.0 (OpenGL ES GLSL ES 1.0 Chromium)
Vendor:	WebKit
Renderer:	WebKit WebGL
Unmasked Vendor:	Google Inc.
Unmasked Renderer:	ANGLE (Intel(R) HD Graphics Family Direct3D9Ex vs_3_0 ps_3_0)
Antialiasing:	Available
ANGLE:	Yes, D3D9
Major Performance Caveat:	No
Vertex Shader
Max Vertex Attributes:	16
Max Vertex Uniform Vectors:	253
Max Vertex Texture Image Units:	4
Max Varying Vectors:	9
Best float precision:	[-2127, 2127] (23)
Transform Feedback
Coming in WebGL 2
Rasterizer
Aliased Line Width Range:	[1, 1]
Aliased Point Size Range:	[1, 256]
Fragment Shader
Max Fragment Uniform Vectors:	221
Max Texture Image Units:	16
float/int precision:	highp/highp
Best float precision:	[-2127, 2127] (23)
Framebuffer
Max Color Buffers:	1
RGBA Bits:	[8, 8, 8, 8]
Depth / Stencil Bits:	[24, 8]
Max Render Buffer Size:	8192
Max Viewport Dimensions:	[8192, 8192]
Textures
Max Texture Size:	8192
Max Cube Map Texture Size:	8192
Max Combined Texture Image Units:	20
Max Anisotropy:	16
Uniform Buffers
Coming in WebGL 2
Supported Extensions:
ANGLE_instanced_arrays
EXT_blend_minmax
EXT_frag_depth
EXT_shader_texture_lod
EXT_texture_filter_anisotropic
WEBKIT_EXT_texture_filter_anisotropic
OES_element_index_uint
OES_standard_derivatives
OES_texture_float
OES_texture_float_linear
OES_texture_half_float
OES_texture_half_float_linear
OES_vertex_array_object
WEBGL_compressed_texture_s3tc
WEBKIT_WEBGL_compressed_texture_s3tc
WEBGL_debug_renderer_info
WEBGL_debug_shaders
WEBGL_depth_texture
WEBKIT_WEBGL_depth_texture
WEBGL_lose_context
WEBKIT_WEBGL_lose_context
To see draft extensions in Chrome, browse to about:flags, enable the "Enable WebGL Draft Extensions" option, and relaunch.

@bmckilligan this may be fixed in Cesium 1.29 as part of the stencil initialization fix.  Can you please retest?

@bmckilligan any update on if this is still an issue for you?  Thanks!

Sorry, I missed the message in January, All looks good on Chrome 64-bit
On 14 June 2017 at 13:32, Patrick Cozzi<EMAIL_ADDRESS>wrote:

@bmckilligan https://github.com/bmckilligan any update on if this is
still an issue for you? Thanks!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/AnalyticalGraphicsInc/cesium/issues/3804#issuecomment-308549615,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AIkllkI-EgYy4r3mwRxyM54fWGKLTb4mks5sEENcgaJpZM4H_a0B
.


New Video Gallery: YouTube preview pane in video lightbox defaults and reverts to small - have to refresh to fix

Describe the bug
Currently, when you first go into Video mode and click on a YouTube video, the preview pane is very small. Doing a Ctrl/F5 makes it revert to the normal size. But if you do something in between (like open a Summary page) it reverts back to the small size.
To Reproduce
Steps to reproduce the behavior:

Go to Search page
Go to video mode
Click on a YouTube video
Note that it's small
Close the preview pane
Refresh (Ctrl/F5)
Reopen the preview pane for the YouTube video - note it's the proper size.
Close preview pane then open the Resource summary page for a video.
Use the back button or click the Search menu
Preview another YouTube video - it's reverted back to being small again.

Expected behavior
The video preview should always show in the larger box
Screenshots
Small

Correct size

Stacktrace
If applicable, a stacktrace to help explain your problem.
Platform:

OpenEquella Version: [e.g. 2021.1]
OS: [e.g. Windows 10, MacOS 10.13]
Browser [e.g. Chrome, Firefox]

Additional context
Add any other context about the problem here.

Tested.

closed by mistake.

Tested.


update util-retry readme.md

doc update only

moved to https://github.com/awslabs/smithy-typescript/pull/952


Remove version: Pidgin.Pidgin version 2.14.6

[Automated] It returns code over 400 in all urls
Microsoft Reviewers: Open in CodeFlow

URL: https://sourceforge.net/projects/pidgin/files/Pidgin/2.14.6/pidgin-2.14.6-offline.exe/download
Status Code: 200
(Automated message - build 896)

Close with reason: Package still available.;


fix: support upload object without content type

Default to use "application/octet-stream" content type
Close https://github.com/treeverse/lakeFS/issues/7127

the test calls the API with an empty content type like in the issue


feat(frontend,meta): support rw_catalog.rw_relation_info

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?

Support rw_catalog.rw_relation_info. Resolve https://github.com/risingwavelabs/risingwave/issues/8264.

Example
dev=> select  schemaname, relationname, relationowner, definition, relationtype, relationid, relationtimezone from rw_catalog.rw_relation_info;
 schemaname | relationname | relationowner |                                             definition                                             |   relationtype    | relationid | relationtimezone
------------+--------------+---------------+----------------------------------------------------------------------------------------------------+-------------------+------------+------------------
 public     | cnt          |             1 | CREATE MATERIALIZED VIEW cnt AS SELECT count(*) FROM sbtest1                                       | MATERIALIZED VIEW |       1004 | UTC
 public     | sbtest1      |             1 | CREATE TABLE sbtest1 (id INT, k INT, c CHARACTER VARYING, pad CHARACTER VARYING, PRIMARY KEY (id)) | TABLE             |       1001 |
 public     | s5           |             1 | CREATE SINK s5 AS SELECT * FROM sbtest1 WITH (connector = 'blackhole')                             | SINK              |       1006 |
 public     | k_1          |             1 | CREATE INDEX k_1 ON sbtest1(k)                                                                     | INDEX             |       1002 |
 public     | ck_1         |             1 | CREATE INDEX ck_1 ON sbtest1(k) INCLUDE(id,c,pad)                                                  | INDEX             |       1003 |
(5 rows)

Checklist For Contributors

[x] I have written necessary rustdoc comments
[x] I have added necessary unit tests and integration tests
[x] I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features #7934).
[x] I have demonstrated that backward compatibility is not broken by breaking changes and created issues to track deprecated features to be removed in the future. (Please refer to the issue)
[x] All checks passed in ./risedev check (or alias, ./risedev c)

Checklist For Reviewers

[ ] I have requested macro/micro-benchmarks as this PR can affect performance substantially, and the results are shown.

Documentation

[ ] My PR DOES NOT contain user-facing changes.

Click here for Documentation
Types of user-facing changes
Please keep the types that apply to your changes, and remove the others.

Installation and deployment
Connector (sources & sinks)
SQL commands, functions, and operators
RisingWave cluster configuration changes
Other (please specify in the release note below)

Release note

@Ad-Bean let's migrate from pg_catalog.pg_matviews to rw_catalog.rw_relation_info.


Add personal portfolio

Using sakura.css and normalize.css as stylesheet.
Add short self introduction and projects.
Add a selfie and screenshot of projects.

The portfolio site is deployed at https://tzhang-sps-summer20.df.r.appspot.com/


release: 1.5.2

:robot: I have created a release beep boop
1.5.2 (2022-12-07)
Performance Improvements

log: perfect caller (0587a0e)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/starudream/go-lib/releases/tag/v1.5.2 :sunflower:


Add docs for API key & fix typo

As the PR title implies, I’ve just added some further information on getting the relevant keys.
Also fixed a typo with the discord type.

Thank you @heychazza lgtm


fix(fossid-webapp): Don't create issue when the URL cannot be mapped

Please have a look at the individual commit messages for the details.

@nnobelis commit-lint fails. You need to use reference-syntax for URLs to work around the line length limitation, like
[1]: https://github.com/package-url/purl-spec/blob/master/PURL-TYPES.rst#generic

and of course refer to [1] from somewhere in the text.


V11 broken

I know that the module is not yet ready for v11 but just to give a feedback: when activated canvas cannot paint background

Yep. Might be a bit longer. v11 broke a lot of stuff for my modules, particularly this one.

v0.5.0 brings v11 compatibility. As I noted in the changelog, still a work in progress. All key functionality should be there, but I had to turn off WebGL shadows.


Model: add mclassdef2node

This was missing, I don't understand why but I need it.

+1. good job

+1


Switch block: Cannot redeclare block-scoped variable

TypeScript Version:  2.0.3, target: es5 or es6:
switch (s) {
		case 0:
			let v = "v.1";
			return v;

		case 1:
			let v = "v.2";
			return v;
	}

Actual behavior:
Cannot redeclare block-scoped variable 'v'
Playground: replay

It seems like a variable scope mess, but most JavaScript engines will ignore second variable declaration let (no strict mode) and will run the output script correctly.
Will it be good if every case statement outputs by default its own block {} in JS? The next code will be compiled with TSC:
switch (s) {
		case 0:
                {
			let v = "v.1";
			return v;
                }
		case 1:
               {
			let v = "v.2";
			return v;
               }
	}

current behavior is per ES spec.  This code yields an error in Node 7, Chrome and Edge:
function f(s) {
	switch (s) {
		case 0:
			let v = "v.1";
			return v;

		case 1:
			let v = "v.2";
			return v;
	}
}

Indeed, ES7 states it clearly, sorry for a wrong issue.
BTW, I used var keyword to test in browsers and most of them know how to run it correctly, while let produces the error due the implementation of ES6+ specs.

Is this intentionally part of the spec? Seems strange...
Note that wrapping the case contents in curly braces correctly defines the scope so this error doesn't occur.

Is this intentionally part of the spec? Seems strange...
Note that wrapping the case contents in curly braces correctly defines the scope so this error doesn't occur.

Thank you for pointing this out. I don't believe the compiler should care if there are braces around the case statement, and even though it's gross I was able to use this as a temporary solution.

This is literally what JavaScript does, why are people upvoting the 2018 comment 144 times? I'm lost.

@RyanCavanaugh It may be what JS does, but is it "correct"? In your example, case 1 ever access a variable declared in case 0? And after the switch, can any variable declared in any case be accessed? No. So then why would it not be okay to use the same-named variable in more than one case statement, aren't they each their own block (with or without curlies)? Furthermore, the fact that case 0/1 return directly means no other case or following code would ever be able to access the variables within the case statement, anyway.

It may be what JS does, but is it "correct"?

Yes.

aren't they each their own block (with or without curlies)

Case labels don't form blocks (more on this in a second). A return statement doesn't make all prior code part of a block, either. Blocks are, well, blocks. Imagine if you wrote this code:
let m;
function fn(s) {
  switch (s) {
    case 0:
      let v = "v.1";
      if (Math.random() > 0.5) return;
    case 1:
      let v = "v.2";
      return;
  }
}

Now it's unambiguously the case that v (sometimes) gets declared twice in the same lexical scope.

means no other case or following code would ever be able to access the variables within the case statement, anyway.

This is wrong. A demonstration is possible:
let m;
function fn(s) {
  switch (s) {
    case 3:
      m = () => v;
    case 0:
      let v = "v.1";
      return;

    case 1:
      // let v = "v.2";
      return;
  }
}
fn(3);
m();

The only way this code works is if v gets hoisted to the enclosing lexical scope, which it does.

  E.g. why is v  ever set to "v.1"? That's the output of the m() call. But case 0 should never have been hit. So JS pulls the declaration of v up (hoists it), when it should not.
If you add curlies around your case bodies, you'll get a different result (at least with Node).
I think the point of this ticket is that Typescript could "correct" what seems to be a flaw in JS.

Changing the runtime semantics of JS to "correct" them has always been outside the scope of the project.


Add content type preflight check for already deployed content

Intent

Resolves #2301
This PR adds a appMode pre-flight check, so a content item selected by mistake with the wrong appMode doesn't get updated.
Type of Change

[x] Bug Fix

[x] New Feature

[ ] Breaking Change

[ ] Documentation

[ ] Refactor

[ ] Tooling

Approach

Additional pre-flight check added. It now shows in the Publisher log and the error displays consistently.

Automated Tests

Unit tests updated.
Directions for Reviewers

To reproduce the issue, (and not have to deal with selectively updating your config manually), you should create a new deployment and then associate it with a different appMode type of content already deployed.
Please also verify that normal re-deployment works without an issue.
Checklist

[ ] I have updated CHANGELOG.md to cover notable changes.

Updated screenshot after PR recommendations:

When this fails, I would expect there to be an option to try to try again with a different URL.
Scenario:
Grab a URL to a piece of content from the server that you want to target
Accidentally chose the wrong URL
Got the proper failure 🎉
Now I get the right URL and want to deploy to that content, but I can't now 😠


round amounts down to 3 decimals

fix #155

lets make it as component with computed value, less prone to errors

Now its as component with 2 props amount and rounding.


Missing dependency

When installing the module as described, I get an error that the module simple_ruuvitag is missing:
Platform error sensor.ruuvi - No module named 'simple_ruuvitag'
Steps to reproduce:

Copy the contents of custom_components into the folder custom_components
Add the documented lines to configuration.yaml
Verify configuration/restart Home Assistant

This is on a HassOS installation.
Supervisor version 2020.11.0
HassOS version 4.17
System version 0.118.2

The same problem for me...

The same problem for me...

I had the same issue initially. For me it worked like this:

Copy files to custom_components (but do not change configuration.yaml yet)
Restart HA (this seems to install the dependencies)
Add sensor to configuration.yaml, configuration should now validate
Restart HA again
Sensor should work

I had the same issue initially. For me it worked like this:

Copy files to custom_components (but do not change configuration.yaml yet)
Restart HA (this seems to install the dependencies)
Add sensor to configuration.yaml, configuration should now validate
Restart HA again
Sensor should work

Many thanks @rkallensee, your procedure solved the problem.

Many thanks @rkallensee, your procedure solved the problem.


Access Azure storage with Managed Identity instead of connection string

This is supported by Dapr. See the Dapr docs here and here.

This only applies to .NET 8 and future versions due to deprecation of .NET 6 and .NET 7 in 2024.

The implementation requires to split components apart per app to allow both web app and image processor to use different (managed) identities. This also requires to apply Dapr scopes and use the route attribute to maintain the API handler's path. The sender component names will need to be adjusted in the source code.

We also need a PowerShell implementation of current-user.sh for Windows users.


stop image loop

hi how i can stop the image from looping per example
we have 3 image :
image 1 image 2 image 3  end

Hi, I'm not sure what you mean, but if you want to stop at specific image, you need to observe page changes and stop it manually by setting slideshowInterval to 0.


Dynamic lib libzmq.so should be copied to the package/ directory

When trying to link against this library, one has to use the static variant, or the link with fail (libzmq.so cannot be found error).
However, the .so libs seem to be generated in the build process, but are not copied across to the package folder.
This fixes the issue:
[~/.conan/data/ZMQ/4.1.1/memsharded/stable]
$ find . -name *.so
./build/42e3e27319bddfce94af67d0ceccbc29de59407f/lib/libzmq.so
./build/2703393d92e3ca970d6bef80175f4a9efdd62bce/lib/libzmq.so
./build/2703393d92e3ca970d6bef80175f4a9efdd62bce/lib/libzmq1.so

$ find . -name *.a
./package/42e3e27319bddfce94af67d0ceccbc29de59407f/lib/libzmq-static.a
./package/2703393d92e3ca970d6bef80175f4a9efdd62bce/lib/libzmq-static.a
./build/42e3e27319bddfce94af67d0ceccbc29de59407f/lib/libzmq-static.a
./build/2703393d92e3ca970d6bef80175f4a9efdd62bce/lib/libzmq-static.a

$ cp ./build/42e3e27319bddfce94af67d0ceccbc29de59407f/lib/libzmq.so ./package/42e3e27319bddfce94af67d0ceccbc29de59407f/lib/

After that the build succeeds.
(whether the binary then actually "works as intended" I've not been able to try yet, though).

Yes, I think this is a problem of the 4.1.1 package recipe. I'd say that the 4.1.5 is much more maintained and already fixed this. It should be fully compatible with 4.1.1. Do you have any special constraint, or could you try to update to see if it works? Thanks!

Perfectly happy to try out 4.1.5 (I am not aware of any reason why not).
However, JFrog acquiring Conan.io has (predictably) completely screwed up the user experience there... how does one go about finding that package???
Searching for zmq first turns up no results in the bintray thingy, then it turns up 17 (seemingly irrelevant) packages... I tried to use
ZMQ/4.1.5@memsharded/stable

but I get the following error trying to install it:
ZMQ/4.1.5@memsharded/stable: WARN: HTTPConnectionPool(host='localhost', port=9300): Max retries exceeded with url: /v1/conans/ZMQ/4.1.5/memsharded/stable/download_urls (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f4c043a03d0>: Failed to establish a new connection: [Errno 111] Connection refused',))

I'm now afraid that if JFrog doesn't get its act together, I may have to reconsider using Conan... as it is, it looks pretty unusable :cry:

Ok - never mind that, figured it out, you moved the package name from ZMQ to libzmq :smile:
(also discovered the "transit" site for Conan - searching stuff there seems more humane)
That's for the good news - the bad news is that 1.4.5 seems to be having the same issue:
$ find libzmq/ -name *.so
libzmq/4.1.5/memsharded/stable/build/42e3e27319bddfce94af67d0ceccbc29de59407f/lib/libzmq.so
[~/.conan/data]
$ find libzmq/ -name *.a
libzmq/4.1.5/memsharded/stable/package/42e3e27319bddfce94af67d0ceccbc29de59407f/lib/libzmq-static.a
libzmq/4.1.5/memsharded/stable/build/42e3e27319bddfce94af67d0ceccbc29de59407f/lib/libzmq-static.a
[~/.conan/data]
$ cp libzmq/4.1.5/memsharded/stable/build/42e3e27319bddfce94af67d0ceccbc29de59407f/lib/libzmq.so libzmq/4.1.5/memsharded/stable/package/42e3e27319bddfce94af67d0ceccbc29de59407f/lib/

fixes the issue - my library builds as a .so dynlib and it all works just fine.

Hi @massenz (Marco)
Yes, we just did the migration to conan-center and conan-transit 9 days ago, and we have still tons of work to do, lets go step by step, Rome was not built in one day, and building this takes some time and sometimes you have to do a small step back (like this lost of usability) to be able to make a great step forward.
Yes, conan-transit is more searchable, and the goal is to have conan-center in the future, not only searchable and populated (right now is empty), but also with high-quality packages that works for most people (as opposed to old conan.io, where packages quality was very variable).
I will have a look to libzmq-4.1.5 to check what can be happening.
PS: Thanks very much, your mom would be proud :) :) :) Thanks to you for all the support and feedback!

I think I have a clue of what could be failing in your case. I am working to regenerate and update all packages and upload them to bintray conan-center.
It could be that you are installing the default package which is the "static" package. You can see the dynamic library there in the build folder, because the libzmq build scripts actually build both static and shared libraries, though only one is package in each different package version. If you want to use the shared library package, install it with:
$ conan install -o libzmq:shared=True

I will update when the new packages are ready in bintray, in case you want to try them.
Please tell me if the above makes sense and works. Thanks!

Bingo!
This reminded me of a similar (but opposite) issue I encountered with gtest - where I had to set it to be False.
So, what I have done was to remove the .so file I had manually copied, make clean, then modified my conanfile to have:
[options]
libzmq:shared=True

re-run conan install ..  and then make - and it all worked just fine!
This is great, I'm closing this issue, please let me know if you'd like a PR to update the README so that other folks will know about this (I already did the same for gtest).
Thanks!

Bingo!
This reminded me of a similar (but opposite) issue I encountered with gtest - where I had to set it to be False.
So, what I have done was to remove the .so file I had manually copied, make clean, then modified my conanfile to have:
[options]
libzmq:shared=True

re-run conan install .. and then make - and it all worked just fine!
This is great, I'm closing this issue, please let me know if you'd like a PR to update the README so that other folks will know about this (I already did the same for gtest).
Thanks!

Hi Marco,
I have submitted an issue to zeromq to let them know about this package: https://github.com/zeromq/libzmq/issues/2600
Depending on the feedback, I will submit my inclusion request to bintray conan-center, so it will be easier to maintain, update and consume them from there.


Remove-Item -Force cannot delete hidden file on Samba

Steps to reproduce
Attempting to delete a hidden file that is stored on a Samba file share (Samba 4.11.6-Ubuntu on Ubuntu 20.04.2 LTS) fails, whereas CMD can delete the file
PowerShell:

PS❯ gci -force  '\\pnjnas\public\FileHistory\LATITUDE7400\Data\C\Users\sto\OneDrive\.NET assembly dependency (2020_09_20 08_01_14 UTC).docx'

    Directory:  S:\MM\catalogs

Mode                LastWriteTime     Length Name
----                -------------     ------ ----
---h-        2016-01-26     09:46    12.67KB .NET assembly dependency (2020_09_20 08_01_14 UTC).docx

PS❯ remove-item '\\pnjnas\public\FileHistory\LATITUDE7400\Data\C\Users\sto\OneDrive\.NET assembly dependency (2020_09_20 08_01_14 UTC).docx'
Remove-Item: You do not have sufficient access rights to perform this operation or the item is hidden, system, or read only.

PS❯ remove-item -force '\\pnjnas\public\FileHistory\LATITUDE7400\Data\C\Users\sto\OneDrive\.NET assembly dependency (2020_09_20 08_01_14 UTC).docx'
Remove-Item: The handle is invalid. : '\\pnjnas\public\FileHistory\LATITUDE7400\Data\C\Users\sto\OneDrive\.NET assembly dependency (2020_09_20 08_01_14 UTC).docx'

CMD:
C:\Users\<REDACTED>>dir /ah "\\pnjnas\public\FileHistory\LATITUDE7400\Data\C\Users\sto\OneDrive\.NET assembly dependency (2020_09_20 08_01_14 UTC).docx
 Volume in drive \\pnjnas\public is Public
 Volume Serial Number is 4664-C5A0

 Directory of \\pnjnas\public\FileHistory\LATITUDE7400\Data\C\Users\sto\OneDrive

2016-01-26  09:46            12,979 .NET assembly dependency (2020_09_20 08_01_14 UTC).docx
               1 File(s)         12,979 bytes
               0 Dir(s)  1,390,021,279,744 bytes free

C:\Users\<REDACTED>>del /ah "\\pnjnas\public\FileHistory\LATITUDE7400\Data\C\Users\sto\OneDrive\.NET assembly dependency (2020_09_20 08_01_14 UTC).docx

C:\Users\<REDACTED>>dir /ah "\\pnjnas\public\FileHistory\LATITUDE7400\Data\C\Users\sto\OneDrive\.NET assembly dependency (2020_09_20 08_01_14 UTC).docx
 Volume in drive \\pnjnas\public is Public
 Volume Serial Number is 4664-C5A0

 Directory of \\pnjnas\public\FileHistory\LATITUDE7400\Data\C\Users\sto\OneDrive

File Not Found

Expected behavior
Remove-Item -Force should delete the file
Actual behavior
Remove-Item -Force  fails with an "invalid handle" error.
Environment data

> $psversiontable

Name                           Value
----                           -----
PSVersion                      7.1.3
PSEdition                      Core
GitCommitId                    7.1.3
OS                             Microsoft Windows 10.0.19042
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           <IP_ADDRESS>
WSManStackVersion              3.0

It seems you are trying to remove OneDrive file which is a Windows reparse point - I guess Samba does not support this.

No I'm not. It's just a coincidence that the Samba-hosted file I was trying to remove happened to be an old backup copy of some file within my OneDrive tree. Sorry for the confusion.
And CMD was able to remove the file.

@sba923 I'd prefer to have an clean repro steps. Can you please create simple test folder tree with PowerShell cmdlets? What results do you see?

Will work on that. Stay tuned.

Here's the test script:
$unixpath = '/data/Public/foo.txt'
$uncpath = '\\pnjnas\Public\foo.txt'

ssh -n sto@pnjnas "rm -f $unixpath"

if (Test-Path -LiteralPath $uncpath)
{
    Write-Host -ForegroundColor Red ("Failed to remove test file")
    Exit(1)
}

"foo" | out-file $uncpath

attrib -a $uncpath
attrib +h $uncpath

ssh -n sto@pnjnas "chmod 544 $unixpath"

ssh -n sto@pnjnas "ls -ld $unixpath"

Get-ChildItem -Force -LiteralPath $uncpath

Remove-Item -LiteralPath $uncpath

Remove-Item -Force -LiteralPath $uncpath

And the outcome:

PS❯ S:\powershell\test\Remove-SambaHidden544.ps1
-r-xr--r-- 1 sto sto 5 Mar 15 08:48 /data/Public/foo.txt

    Directory:  C:\Users\<REDACTED>

Mode                LastWriteTime     Length Name
----                -------------     ------ ----
---h-        2021-03-15     08:48        5   foo.txt
Remove-Item: S:\powershell\test\Remove-SambaHidden544.ps1:23
Line |
  23 |  Remove-Item -LiteralPath $uncpath
     |  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | You do not have sufficient access rights to perform this operation or the item is hidden, system, or
     | read only.

Remove-Item: S:\powershell\test\Remove-SambaHidden544.ps1:25
Line |
  25 |  Remove-Item -Force -LiteralPath $uncpath
     |  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | The handle is invalid. : '\\pnjnas\Public\foo.txt'

You should use -Force switch to remove hidden file objects.

That's exactly what I'm doing, and that triggers the "invalid handle" error.

Thanks, I see.
Please try .Net methods to remove the file:
$a=[io.fileinfo]::new('\\pnjnas\Public\foo.txt')
$a.Delete()

This works:
$unixpath = '/data/Public/foo.txt'
$uncpath = '\\pnjnas\Public\foo.txt'

ssh -n sto@pnjnas "rm -f $unixpath"

if (Test-Path -LiteralPath $uncpath)
{
    Write-Host -ForegroundColor Red ("Failed to remove test file")
    Exit(1)
}

"foo" | out-file $uncpath

attrib -a $uncpath
attrib +h $uncpath

ssh -n sto@pnjnas "chmod 544 $unixpath"

ssh -n sto@pnjnas "ls -ld $unixpath"

Get-ChildItem -Force -LiteralPath $uncpath

Remove-Item -LiteralPath $uncpath

Remove-Item -Force -LiteralPath $uncpath

$a=[io.fileinfo]::new($uncpath)
$a.Delete()

if (Test-Path -LiteralPath $uncpath)
{
    Write-Host -ForegroundColor Red ("Failed to remove test file")
    Exit(1)
}
else {
    Write-Host -ForegroundColor Green ("Test file removal succeeded")
}

PS❯ S:\powershell\test\Remove-SambaHidden544.ps1
-r-xr--r-- 1 sto sto 5 Mar 15 13:40 /data/Public/foo.txt

    Directory:  Microsoft.PowerShell.Core\FileSystem::\\pnjnas\Musique\Twonky\Playlists

Mode                LastWriteTime     Length Name
----                -------------     ------ ----
---h-        2021-03-15     13:40        5   foo.txt
Remove-Item: S:\powershell\test\Remove-SambaHidden544.ps1:23
Line |
  23 |  Remove-Item -LiteralPath $uncpath
     |  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | You do not have sufficient access rights to perform this operation or the item is hidden, system, or
     | read only.

Remove-Item: S:\powershell\test\Remove-SambaHidden544.ps1:25
Line |
  25 |  Remove-Item -Force -LiteralPath $uncpath
     |  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | The handle is invalid. : '\\pnjnas\Public\foo.txt'

Test file removal succeeded

Thanks! Since .Net API works I believe it is PowerShell bug.

You're very welcome.

Note: this is not linked to PowerShell running on Linux!

Yes, I checked the code works well for Windows shares. So the label is only about Samba context.

Fine.
Don't hesitate to involve me in testing (candidate) fixes!

The issue here is the chmod 544 is setting permissions for the Samba user to just read and execute without any write access. For a delete operation this is fine as you just need to make sure you open the file with that request right. When you do
$a=[io.fileinfo]::new($uncpath)
$a.Delete()

# Or just
[IO.File]::Delete($uncpath)

It will open the file with the Delete access mask which should still be fine.
When you do Remove-Item ... it is actually opening the file with read and write access to the attributes and setting the attributes to Normal. I assume this is because -Force is meant to force the deletion that fails when the ReadOnly flag is set.

Because the samba user does not have write access it is failing to open the handle to that file and thus fails to delete the attributes. The fact that it's a compound request is probably the reason why the error back is invalid handle rather than access denied because the last request failed with that.
For fixing this problem I see 2 ways this could be done

Ignore this particular error if it occurs and just try the delete anyway, make removing the ReadOnly flag a best effort and use the actual deletion request what is reported on
Try to delete the file first and then unset the attribute if that failed with access denied

Personally I think the first option is the way to go, it doesn't add/remove any extra steps than what is currently done and if the user is truly unable to delete the file then the error back from the deletion request will still reflect that. The 2nd option might be quicker in normal cases but it could add an extra step if it does need to remove the attribute (and it's not needed).

The issue here is the chmod 544 is setting permissions for the Samba user to just read and execute without any write access.

Could you please share a screenshot with how Windows maps this to Windows permissions? I'd want to see Security tab for the Samba file.

So I have 2 files in a share
[root@083f251e5344 app]# ls -al /srv/samba/share/
total 0
drwxr-xr-x. 1 smbuser smbgroup 34 Mar 16 04:43 .
drwxr-xr-x. 1 root    root     54 Mar 16 04:41 ..
-r-xr--r--. 1 smbuser smbuser   0 Mar 16 04:42 foo.txt
-rwxr--r--. 1 smbuser smbuser   0 Mar 16 04:43 normal.txt

For foo.txt (chmod 544) here are the permissions in the Windows GUI

For normal.txt (chmod 744) here are the permissions in the Windows GUI

So this confirms that the Unix permissions affect the permissions that Samba uses over SMB. It makes sense as well because if the Samba user is unable to write to the file then it is unable to write data (like attributes) to it. You can even replicate this outside of PowerShell and get the same Invalid handle error message back by trying to change the attributes after removing write access on the remote side.
If you are wanting to emulate the fix just make sure that the user does not have the Write Attributes access mask because that is the access this particular operation is requesting and failing to request. You can see in the Wireshark screenshot that it's simply requesting WA/RA/Synchronize and WA is what is actually used in the SetInfo request with that handle.

there may be file open operations due to the versatility of this code and it may not be easy to fix.

I don't think the proposed fix changes anything about that. All I'm proposing is to not make that 1 operation a failure but rather a at best operation. If it truly cannot delete the file because it does not have access (or it's in use) then the subsequent delete call will still fail and report that error back to the user.

For example, this should works for Remove-Item -Force -Path \server\share\folder*\file*.txt

In this case it does not, the problem lies with how the FileSystem provider tries to delete a file when -Force is present (it must be present here because the file is hidden). All this logic is contained in RemoveFileSystemItem with the offending code in particular being https://github.com/PowerShell/PowerShell/blob/4c40ab11401ff2ca19ee1206b7654ed299f373e6/src/System.Management.Automation/namespaces/FileSystemProvider.cs#L3272-L3276.
We can see that if -Force is set it always removes the ReadOnly | Hidden | System access mask before it moves onto the actual delete. By ignoring the failure when setting the attributes here we would actually have a successful deletion. Even if removing the attributes failed because the user didn't have permissions that permission error would be raised in the subsequent Delete call.

So I have 2 files in a share

Can you confirm the files is hidden? Can you share info for non-hidden files?

I can confirm, I set it myself. The fact it is hidden has no bearing over the permissions. It’s only to expose this bug that happens when using -Force and the user does not have write attributes access to the file.

AFAIK there are no complications or extra steps required to delete a file that is Hidden or System so I'm not sure why those are being cleared out here.

It is exclusive PowerShell smart feature :-) By design. We should keep this.
So the proposed fix is to put the "if (force)" to "try-catch" - I am inclined to accept this.

I confirm that if the file is not hidden, Remove-Item -Force still fails with the "invalid handle" error:
$unixpath = '/data/Public/foo.txt'
$uncpath = '\\pnjnas\Public\foo.txt'

ssh -n sto@pnjnas "rm -f $unixpath"

if (Test-Path -LiteralPath $uncpath)
{
    Write-Host -ForegroundColor Red ("Failed to remove test file")
    Exit(1)
}

"foo" | out-file $uncpath

# attrib -a $uncpath
# attrib +h $uncpath

ssh -n sto@pnjnas "chmod 544 $unixpath"

ssh -n sto@pnjnas "ls -ld $unixpath"

Get-ChildItem -Force -LiteralPath $uncpath

# Remove-Item -LiteralPath $uncpath

Remove-Item -Force -LiteralPath $uncpath

# $a=[io.fileinfo]::new($uncpath)
# $a.Delete()

if (Test-Path -LiteralPath $uncpath)
{
    Write-Host -ForegroundColor Red ("Failed to remove test file")
    Exit(1)
}
else {
    Write-Host -ForegroundColor Green ("Test file removal succeeded")
}

PS❯ S:\powershell\test\Remove-SambaHidden544.ps1
-r-xr--r-- 1 sto sto 5 Mar 16 09:41 /data/Public/foo.txt

    Directory:  Microsoft.PowerShell.Core\FileSystem::\\pnjnas\Musique\Twonky\Playlists

Mode                LastWriteTime     Length Name
----                -------------     ------ ----
-a---        2021-03-16     09:41        5   foo.txt
Remove-Item: S:\powershell\test\Remove-SambaHidden544.ps1:25
Line |
  25 |  Remove-Item -Force -LiteralPath $uncpath
     |  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | The handle is invalid. : '\\pnjnas\Public\foo.txt'

Failed to remove test file

For fixing this problem I see 2 ways this could be done

Ignore this particular error if it occurs and just try the delete anyway, make removing the ReadOnly flag a best effort and use the actual deletion request what is reported on
Try to delete the file first and then unset the attribute if that failed with access denied

Personally I think the first option is the way to go, it doesn't add/remove any extra steps than what is currently done and if the user is truly unable to delete the file then the error back from the deletion request will still reflect that. The 2nd option might be quicker in normal cases but it could add an extra step if it does need to remove the attribute (and it's not needed).

We need to make sure that in no circumstances the operation could fail but have the side effect of modifying the file's attributes.

Totally understand it’s a feature of PowerShell :) My main point is that the code is unilaterally removing all 3 attributes (ReadOnly, System, Hidden) in this case but really it’s only ReadOnly that needs to be removed. It could even be smarter and only remove the attribute if it’s not currently set to reduce the number of operations that need to occur.
I still think a try/catch is also good for this situation but both options will fix this problem.

We need to make sure that in no circumstances the operation could fail but have the side effect of modifying the file's attributes.

That’s already handled in the current code. If the attributes were changed but the delete failed then the attributes are reset back.

Totally understand it’s a feature of PowerShell :) My main point is that the code is unilaterally removing all 3 attributes (ReadOnly, System, Hidden) in this case but really it’s only ReadOnly that needs to be removed. It could even be smarter and only remove the attribute if it’s not currently set to reduce the number of operations that need to occur.
I still think a try/catch is also good for this situation but both options will fix this problem.

I see your point. Only concern is about how this attributes are mapped to Unix attributes by Samba (I'd want to get a confirmation that .Net API can remove files with the attributes). If there are no issues due to the mapping I think we can accept the fix with the two changes since the code has good test coverage.

Only concern is about how this attributes are mapped to Unix attributes by Samba

From my understanding of Samba it can store the attributes Hidden, ReadOnly, System, and Archive in 2 different ways.

When store dos attributes is yes (default since 4.9

The attributes are stored in an extended attribute which is like special metadata on a file

When store dos attributes = no and the relevant map <attribute> is set to yes

Uses the mode of the file to determine if the 4 attributes are set
Hidden - world executable (001)
System - group executable (010)
Archive - owner executable (100)
ReadOnly - owner write (200) (there are slight variants here but doesn't matter as much)

In my case I'm testing against a Samba setup that has store dos attributes = yes, potentially @sba923 might be using the owner write bit hence the chmod example.
I can confirm that when using store dos attributes = yes then .NET is able to delete the file when Hidden, System, Archive is set. It only fails when ReadOnly is set which aligns with how I understand Windows works. Even the documentation for DeleteFile states

To delete a read-only file, first you must remove the read-only attribute.

There are no other mentions for the other attributes and considering SMB is a Microsoft protocol it would make sense that Samba replicates the same behaviour.
I also tested store dos attributes = no with the relevant map <attribute> = yes options enabled and it has the same behaviour.

@jborean93 Thanks for your research! I think we can offer the community fix.

FWIW I'm running Samba version 4.11.6-Ubuntu and the config file contains nothing but a commented-out store dos attributes = yes

Note that I also get Remove-Item: The handle is invalid upon a Remove-Item -LiteralPath foo -Verbose -Force, for a non-hidden file that is just not writable for the user used for the connection to the Samba share, i.e.:
-rwxrw-r-- 1 otheruser otheruser 21215 Jul 25  2010 foo

What's next towards a solution to this issue?


IS-CAM formula error?

🚀 Feature
https://frgfm.github.io/torch-cam/methods.html#activation-based-methods
Mk /ak formula of ISCAM in  the link above is not the same as the original paper.
And I wonder if there is an original implement of ISCAM? Because  I find that alpha calculated in paper is sumed by N time. But in your code  it is sumed by k time.
Motivation & pitch
.
Alternatives
No response
Additional context
No response

Hi @noobgrow 👋
Thanks for reporting this! So actually, the author reached out to me before the paper even came out to experiment with it. He validated the implementation back then, so I didn't question whether there would be some difference with the upcoming paper 😅
So let me check the difference you're talking about and get back to you 🙂
Cheers ✌

OK! thanks for your fantastic work! Waiting for you @frgfm ^.^

So, this is going to be annoying without LaTex for math rendering, but let's try 😅
In the paper:
alpha_k^c = 1 / N * sum\limits_{i=0}^N (C(M_i))
# Let's expand M_i
alpha_k^c = 1 / N * sum\limits_{i=0}^N (C(sum\limits_{j=0}^{i-1} j / N * X_0 * A_l^k))
# Now we can simplify the sum
alpha_k^c = 1 / N * sum\limits_{i=0}^N (C(\frac{i * (i - 1)}{2N} * X_0 * A_l^k))

In the code, X_0 * A_l^k is stored as scored_inputs (cf. https://github.com/frgfm/torch-cam/blob/master/torchcam/methods/activation.py#L411-L414)
and \frac{i * (i - 1)}{2N} is stored as _coeff (cf. https://github.com/frgfm/torch-cam/blob/master/torchcam/methods/activation.py#L430)
So this line https://github.com/frgfm/torch-cam/blob/master/torchcam/methods/activation.py#L437 computes C(M_i) from the paper, they are summed in the following lines, and finally I divide by N here https://github.com/frgfm/torch-cam/blob/master/torchcam/methods/activation.py#L445
However, I realize that the docstring has some errors, I'll correct it to fix the documentation 👍
Let me know if some things are still unclear!

Closing this as the docstring has been fixed and the rest of the explanations have been detailed! Feel free to reopen @noobgrow if that's not the case :)


freeze & crash when directed to open an invalid gopher:// URL

I am running Lagrange natively on Arch Linux:
When a gopher address that has been entered into the address bar doesn't really exist, and it tries to load it for some time, the entire client (browser) often completely freezes, rendering the UI unresponsive:

I have not observed the same behavior with non-existant Gemini addresses, which properly result in a not-found error.
Possible solution...?
When the UI becomes unresponsive, and the app is force quit, it should clear its address bar and not try to load the same non-existant address as it does now, because that results in an infinite loop of freeze.

I wonder if this is related to #302... I'll need to try to reproduce this on the systems available to me. I can run Arch Linux in a VM, but it's not a given that the problem can be reproduced inside a virtual machine.

I can run Arch Linux in a VM, but it's not a given that the problem can be reproduced inside a virtual machine.

I would be surprised if this was Arch Linux specific.
Mostly I think a non-existent gopher URL (such as the one in the screenshot) is the problem, in that Lagrange tries to reach it for an infinite time maybe?, and then freezes if that fails, instead of giving up, and stopping its attempt to load the address altogether, if it say, takes longer than 5-6 seconds?

Please test with the https://github.com/skyjake/lagrange/releases/tag/v1.6.3 release.

Please test with the https://github.com/skyjake/lagrange/releases/tag/v1.6.3 release.

With me testing this today, it seems to be fixed (i.e freezing, while loading an incorrect address no longer happens, instead the correct Network/TLS Failure displayed, and the app's been fully responsive all the while. Awesome. I'll continue testing, and I will close this issue now as resolved.
Thanks!


Add support for abs to ParameterExpression

What is the expected enhancement?
The ParameterExpression supports common functions like cos and sin, but not abs. A minimal example:
import numpy as np
from qiskit.circuit import Parameter
p=Parameter('x')
q=Parameter('.1')
[np.cos(p), np.cos(q)]

[np.abs(p), np.abs(q)]  # fails

Enhencemant: add abs to the ParameterExpression class.
Notes: Methods like cos have been added in this PR: https://github.com/Qiskit/qiskit-terra/pull/4854
There is an open PR to add relational operations for the ParameterExpression https://github.com/Qiskit/qiskit-terra/pull/5278, but that does not add abs.

Hey @peendebak
Shall I have a go at this issue?


Regenerate protobuf files with newer protoc versions

protoc-gen-go v1.26.0
protoc        v3.17.2
Let's discuss the protobuf here. I installed protoc, generated files and here are changes that popped up.
BTW how to run make proto? I had to add option go_package = "/pb"; to the message to generate it and then remove it (might be my lower knowledge of protobuf and goland maybe?)
protoc-gen-go: unable to determine Go import path for "pb/flow.proto"

Please specify either:
	• a "go_package" option in the .proto source file, or
	• a "M" argument on the command line.

See https://developers.google.com/protocol-buffers/docs/reference/go-generated#package for more information.

--go_out: protoc-gen-go: Plugin failed with status code 1.
make: *** [proto] Error 1

Hi @simPod, thank you for regenerating the files.
In a different PR I updated the Makefile and the protobuf flags to avoid the issue you encountered. Also updated the documentation.
Weirdly, it did not generate the same FileDescriptor. I will have to merge the other one.

Cool, now it works and produces no diff 👍🏾


[BUG] import .less from node_modules not translate to .css

Bug Report Quick Checklist

[x] I am on the latest version of Snowpack & all plugins<EMAIL_ADDRESS>[x] I use package manager<EMAIL_ADDRESS>(Fill in: npm, yarn, pnpm, etc).
[x] I run Snowpack on OS Mac (Fill in: Windows, Mac, Linux, etc).
[x] I run Snowpack on Node.js v12+<EMAIL_ADDRESS>
Describe the bug
I want to import .less from node_modules but it not translate to .css.
To Reproduce

Create a new snowpack project

npm init test
...

add ./package.json

{
  "name": "test",
  "version": "1.0.0",
  "description": "test",
  "main": "./build/index.js",
  "scripts": {
    "start": "snowpack dev"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "@snowpack/plugin-sass": "^1.4.0",
    "antd": "^4.15.5",
    "element-plus": "^1.0.2-beta.44",
    "less": "^4.1.1",
    "sass": "^1.32.13",
    "snowpack": "^3.4.0",
    "snowpack-plugin-less": "^1.0.7"
  }
}

add ./snowpack.config.js

// Snowpack Configuration File
// See all supported options: https://www.snowpack.dev/reference/configuration

/** @type {import("snowpack").SnowpackUserConfig } */
module.exports = {
  workspaceRoot: './src',
  extends: '',
  mount: {
    src: { url: '/' }
  },
  plugins: [
    [
      'snowpack-plugin-less',
      {
        javascriptEnabled: true,
      }
    ],
    [
      '@snowpack/plugin-sass',
    ]
  ],
};

add ./src/index.html

<!DOCTYPE html>

    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <meta name="description" content="Starter Snowpack App" />
    <title>Starter Snowpack App</title>

Welcome to Snowpack!

add ./src/index.ts

import 'antd/dist/antd.less';
// import 'antd/dist/antd.css';
import 'element-plus/packages/theme-chalk/src/index.scss';
// import 'element-plus/lib/theme-chalk/index.css'

console.log(123)

install

npm install

build

npx snowpack build --verbose --clean

Go to ./build ,you can see antd.less and  index.scss, not .css

Expected behavior
[03:53:24] [snowpack] run command: build
[03:53:24] [snowpack] loaded plugin: /Users/user/test/node_modules/snowpack-plugin-less/dist/index.js
[03:53:24] [snowpack] loaded plugin<EMAIL_ADDRESS>[03:53:24] [snowpack] config loaded: {
  root: '/Users/user/test',
  plugins: [
    {
      name: 'snowpack-plugin-less',
      resolve: [Object],
      onChange: [Function: onChange],
      load: [Function: load],
      markChanged: [Function (anonymous)]
    },
    {
      name: '@snowpack/plugin-sass',
      resolve: [Object],
      _markImportersAsChanged: [Function: _markImportersAsChanged],
      onChange: [Function: onChange],
      load: [AsyncFunction: load],
      markChanged: [Function (anonymous)]
    },
    {
      name: '@snowpack/plugin-esbuild',
      resolve: [Object],
      load: [AsyncFunction: load],
      cleanup: [Function: cleanup]
    }
  ],
  alias: {},
  env: {},
  exclude: [
    '**/_*.{sass,scss}',
    '**.d.ts',
    '/Users/user/test/build/**',
    '**/node_modules/**'
  ],
  routes: [],
  dependencies: {},
  devOptions: {
    secure: false,
    hostname: 'localhost',
    port: 8080,
    hmrDelay: 0,
    hmrPort: undefined,
    hmrErrorOverlay: true
  },
  buildOptions: {
    out: '/Users/user/test/build',
    baseUrl: '/',
    metaUrlPath: '/_snowpack',
    clean: true,
    sourcemap: false,
    watch: false,
    htmlFragments: false,
    ssr: false,
    resolveProxyImports: true
  },
  testOptions: { files: [ '__tests__/**/*', '**/*.@(spec|test).*' ] },
  packageOptions: {
    source: 'local',
    external: [],
    packageLookupFields: [],
    knownEntrypoints: [],
    rollup: { plugins: [] }
  },
  workspaceRoot: '/Users/user/test/src',
  extends: '',
  mount: {
    '/Users/user/test/src': { url: '/', static: false, resolve: true }
  },
  experiments: {},
  mode: 'production',
  optimize: undefined,
  _extensionMap: {
    '.less': [ '.css' ],
    '.scss': [ '.css' ],
    '.sass': [ '.css' ],
    '.mjs': [ '.js' ],
    '.jsx': [ '.js' ],
    '.ts': [ '.js' ],
    '.tsx': [ '.js' ]
  }
}
[03:53:24] [snowpack] lockfile not loaded
[03:53:24] [snowpack] Mounting directory: '/Users/user/test/src' as URL '/'
[03:53:24] [snowpack] Using in-memory cache: [object Object]
[03:53:24] [snowpack] Mounting directory: '/Users/user/test/src' as URL '/'
[03:53:24] [snowpack] ! building files...
[03:53:24] [snowpack] QUEUE: /index.html,/index.js
[03:53:24] [snowpack] BUILD: /index.html
[03:53:24] [snowpack] Scanning src/index.html for imports as HTML
[03:53:24] [snowpack] BUILD: /index.js
[03:53:24] [@snowpack/plugin-esbuild] load() starting… [src/index.ts]
[03:53:24] [@snowpack/plugin-esbuild] ✔ load() success [src/index.ts]
[03:53:24] [snowpack] Scanning src/index.ts for imports as JS
[03:53:24] [snowpack] ADD: antd/dist/antd.less
[03:53:24] [snowpack] ADD: element-plus/packages/theme-chalk/src/index.scss
[03:53:24] [snowpack] ✔ files built. [0.12s]
[03:53:24] [snowpack] ! building dependencies...
[03:53:24] [snowpack] Successfully ran esinstall.
[03:53:24] [snowpack] ✔ dependencies built. [0.01s]
[03:53:24] [snowpack] ! writing to disk...
[03:53:24] [snowpack] QUEUE: /index.html,/index.js
[03:53:24] [snowpack] BUILD: /index.html
[03:53:24] [snowpack] Scanning src/index.html for imports as HTML
[03:53:24] [snowpack] BUILD: /index.js
[03:53:24] [snowpack] Scanning src/index.ts for imports as JS
[03:53:24] [snowpack] ADD: /_snowpack/pkg/antd/dist/antd.less.proxy.js
[03:53:24] [snowpack] ADD: /_snowpack/pkg/element-plus/packages/theme-chalk/src/index.scss.proxy.js
[03:53:24] [snowpack] BUILD: /_snowpack/pkg/antd/dist/antd.less.proxy.js
[03:53:24] [snowpack] BUILD: /_snowpack/pkg/element-plus/packages/theme-chalk/src/index.scss.proxy.js
[03:53:24] [snowpack] ✔ write complete. [0.02s]
[03:53:24] [snowpack] ▶ Build Complete!

Anything else?
If I import .css in node_modules, it work well.

you can add rollup less plugin to packageOptions，then include antd less path in node_module.
It will work

you can add rollup less plugin to packageOptions，then include antd less path in node_module.
It will work

Could you help to give some code snippets on it? more detail. I'm running into this issue too. Can't solve it now.

you can add rollup less plugin to packageOptions，then include antd less path in node_module.
It will work

@KnightSpirit Could you help to give some code snippets on it? more detail. I'm running into this issue too. Can't solve it now.

packageOptions: {
/* ... /
rollup: {
plugins: [
require("rollup-plugin-less")({
include: ["**/.less"],
exclude: ["1"],
option: options,
}),
],
},
},

you can add rollup less plugin to packageOptions，then include antd less path in node_module.
It will work

@KnightSpirit Could you help to give some code snippets on it? more detail. I'm running into this issue too. Can't solve it now.

 packageOptions: {     /* ... */     rollup: {       plugins: [         require("rollup-plugin-less")({           include: ["**/*.less"],           exclude: ["1"],           option: options,         }),       ],     },   },


Group interpreters in interpreter quick picker using separators

https://github.com/microsoft/vscode/issues/137745

Likely related #17944

Thanks, as your open was opened first, I'll close this issue is favor of https://github.com/microsoft/vscode-python/issues/17944 👍


execute rvm:check fails capistrano

$ cap production deploy --trace
** Invoke production (first_time)
** Execute production
** Invoke load:defaults (first_time)
** Execute load:defaults
** Invoke rvm:hook (first_time)
** Execute rvm:hook
** Invoke rvm:check (first_time)
** Execute rvm:check
cap aborted!
Net::SSH::AuthenticationFailed: Authentication failed for user<EMAIL_ADDRESS>/home/hashmi/.rvm/gems/ruby-2.4.0/bundler/gems/net-ssh-ecc40c7cdea0/lib/net/ssh.rb:250:in start' /home/hashmi/.rvm/gems/ruby-2.4.0/gems/sshkit-1.12.0/lib/sshkit/backends/connection_pool.rb:59:in call'
/home/hashmi/.rvm/gems/ruby-2.4.0/gems/sshkit-1.12.0/lib/sshkit/backends/connection_pool.rb:59:in with' /home/hashmi/.rvm/gems/ruby-2.4.0/gems/sshkit-1.12.0/lib/sshkit/backends/netssh.rb:155:in with_ssh'
/home/hashmi/.rvm/gems/ruby-2.4.0/gems/sshkit-1.12.0/lib/sshkit/backends/netssh.rb:108:in execute_command' /home/hashmi/.rvm/gems/ruby-2.4.0/gems/sshkit-1.12.0/lib/sshkit/backends/abstract.rb:141:in block in create_command_and_execute'
/home/hashmi/.rvm/gems/ruby-2.4.0/gems/sshkit-1.12.0/lib/sshkit/backends/abstract.rb:141:in tap' /home/hashmi/.rvm/gems/ruby-2.4.0/gems/sshkit-1.12.0/lib/sshkit/backends/abstract.rb:141:in create_command_and_execute'
/home/hashmi/.rvm/gems/ruby-2.4.0/gems/sshkit-1.12.0/lib/sshkit/backends/abstract.rb:60:in capture' /home/hashmi/.rvm/gems/ruby-2.4.0/gems/capistrano-rvm-0.1.2/lib/capistrano/tasks/rvm.rake:9:in block (3 levels) in <top (required)>'
/home/hashmi/.rvm/gems/ruby-2.4.0/gems/sshkit-1.12.0/lib/sshkit/backends/abstract.rb:29:in instance_exec' /home/hashmi/.rvm/gems/ruby-2.4.0/gems/sshkit-1.12.0/lib/sshkit/backends/abstract.rb:29:in run'
/home/hashmi/.rvm/gems/ruby-2.4.0/gems/sshkit-1.12.0/lib/sshkit/runners/parallel.rb:12:in `block (2 levels) in execute'
Tasks: TOP => rvm:check

Are you sure this is Capistrano's fault? It looks like the connection failed to authenticate:
Authentication failed for user<EMAIL_ADDRESS>
Try ssh<EMAIL_ADDRESS>to debug.

Hi, I recommend checking the Capistrano documentation here:
http://capistranorb.com/documentation/getting-started/authentication-and-authorisation/
Also, @mikegee makes a good point: check that normal ssh works.
If you are still stuck, a good place to get help is Stack Overflow. Include output of cap production doctor in your question to give more context. Good luck!

GitHub issues are for feature requests or bug reports. Since this is more of a how-to question, I am closing this issue. The Capistrano team recommends you use Stack Overflow for general questions. For more details, please see our contribution policy.


Remove dependence on $HOME environment variable

Because of obfGetPw and other functions of that familiy, we are required to set $HOME and create a .irods directory in which to save the .irodsA file required by the iRODS authentication plugins.
We should move this to /etc/irods_client_rest_cpp along with all of the other configuration so that it is securely stored and more hands-free.

The workaround until this is fixed is to edit the init.d script to export $HOME manually...
https://github.com/irods/irods_client_rest_cpp/blob/987a1d077e6c88ab24a86d6710ba8196307ff35d/packaging/irods_client_rest_cpp#L54-L60
As this is not generic (b/c it's a workaround), you may need to tweak the actual location of HOME= for your deployment environment:
start_irods_client_rest_cpp() {
+   export HOME="/root"
    for API in `ls /usr/sbin/$PREFIX*`
    do
        ${API}&
    done
    mkdir -p "${HOME}"/.irods
}


ERROR: UndefVarError: load_gguf_model not defined

Trying to follow the instructions in the README and immediately get the above error.

The instructions in the readme were missing a using Llama2. I updated the readme accordingly.


Fix nimble warnings

Warning: Package 'futhark' has an incorrect structure. The top level of the package source directory should contain at most one module, named 'futhark.nim', but a file named 'libclang.nim' was found. This will be an error in the future.
Hint: If this is the primary source file in the package, rename it to 'futhark.nim'. If it's a source file required by the main module, or if it is one of several modules exposed by 'futhark', then move it into a 'futhark' subdirectory. If it's a test file or otherwise not required to build the package 'futhark.nim', prevent its installation by adding skipFiles = @["libclang.nim"] to the .nimble file. See https://github.com/nim-lang/nimble#libraries for more info.

Nimble has had this issue since the dawn of time, it is very likely to never be turned into an error.
That being said I don't really see an issue with actually fixing this. Could you fix the conflicts with src/opir.nim?


No wands

For some reasons there are none of the wands shown and I don´t know why. When I tried it before I saw all the wand and could choose one but now that doesn´t work. I´d like some advise if possible. See the screenshot down below.

Please list all of your addons and content.
Example:

[Content]
TF2
L4D2

[Addons]
Wiremod
Drones:Rewrite
CW2.0
SProps


Ensure captive core LCL is not ahead of horizon last ingested ledger during restarts

If Horizon is running captive core with BucketsDB enabled, Horizon should be able to restart quickly as long as we run captive core so that it resumes from the LCL (last closed ledger) recorded in the core DB. We have code in Horizon's ingestion library to detect this case:
https://github.com/stellar/go/blob/horizon-v2.27.0-rc2/ingest/ledgerbackend/stellar_core_runner.go#L376-L379
But, there is a scenario where the LCL can be ahead of the last ingested ledger in Horizon's DB:
When captive core writes a ledger to the file pipe, the captive core process (which is single threaded) blocks until the ledger is read out of the file pipe. However, Horizon mitigates the possibility that core is blocked on writing ledgers over the pipe by continuously reading ledgers out of the file pipe until an in-memory buffer is filled up:
https://github.com/stellar/go/blob/horizon-v2.27.0-rc2/ingest/ledgerbackend/buffered_meta_pipe_reader.go
If the speed of Horizon ingestion is faster than the rate at which captive core emits ledgers, the in-memory ledger buffer will not have a chance to accrue many ledgers. But there is always the possibility that, when Horizon receives the shutdown signal, the in-memory ledger buffer will have accumulated some ledgers. In that scenario, the ledgers stored in the buffer are discarded which means that the LCL in the captive core DB is ahead of Horizon's last ingested ledger.
We should mitigate this scenario where the LCL is ahead of horizon's DB so that Horizon can be restarted quickly.

We could modify Horizon ingestion's termination logic so that we ingest whatever ledgers are left in the in-memory buffer before shutting down. But, I think that will be tricky to implement because of concurrency issues and coupling between the Horizon ingestion state machine and the captive core process manager.
Another approach which I think could be promising is to save the ledgers from the in-memory buffer into a file before shutting down. Then when Horizon is restarted it can read the ledgers from the file and insert them into the in-memory buffer before starting up captive core.

Horizon mitigates the possibility that core is blocked on writing ledgers over the pipe by continuously reading ledgers out of the file pipe until an in-memory buffer is filled up

I wonder if we capture metrics on how often it happens that this in-memory buffer is used. Presumably/theoretically, if Horizon is ingesting faster than captive-core is generating ledgers, it shouldn't happen often (or at all?), but obviously we added this for a reason.
I asked this because maybe we should explore if it has any utility to us at all, and if so, if we should even have it. I ask this because there was a recent slack conversation, where @graydon actually pointed out that core has/had attempted to design captive core so as to remove the possibility of this happening (in the case of core itself crashing). And it seems like we re-introduced that possibility by the introduction of this in-memory queue. Context:

...architecturally we also really want to have a moment where we're sure we've handed off a given txmeta (and flushed the stream, and know the reader got it) before we issue a durable commit to disk. because if we don't, and we commit first and then crash while flushing to horizon, if we restart we'll be a ledger ahead and horizon won't be able to recover that ledger from us.

 Horizon "mitigating" this feature by reading ahead into a non-durable (in-memory) buffer is actually defeating that design.
(Thinking about this a bit more, I can see that we might even need a subsequent durability-ack signal from horizon in the form of, say, an additional synchronous dummy frame we write-and-flush and horizon reads in order to ack the previously-flushed non-dummy txmeta; but the idea is absolutely to do-what-is-necessary to prevent his scenario!)

(if it's a problem to have a captive block while horizon commits, that's a different kettle of fish; we assumed this is not a problem, since captives are guaranteed not to be validators we assumed that it's ok for them to block a bit / fall behind the network a bit, nobody's waiting for their vote)

Point posed by @tamirms at grooming just now. It seems like core commits as soon as Horizon reads from the pipe, however, at that point, Horizon hasn't committed to it's own database. So there's still a possibility that Horizon could crash or be shut down while processing that ledger, and before it commits to it's own database.
The durability-ack idea you propose could potentially solve this. Of course, Horizon would need to remove the in-memory queue on it's side for that to be workable. Either way, it would definitely be valuable for us to understand the history of why that queue was introduced.

I found the issue which motivated adding the queue: https://github.com/stellar/go/issues/3132
Another way to solve this issue is if core stored the n most recently applied ledgers in its db and allowed catching up from any ledger in the range of [LCL - n, LCL]

if considering an extended custom serial transport protocol over core's meta o/s pipe with ack messaging, it is worth considering oss network libraries that run a framed transport underneath to accomplish the same, like 0mq REQ/REP logical socket pair, which can be applied in a reliable messaging pattern, in context of captive core usage on same-host machine, REQ/REP  pair could be bound to ipc or socket transports, the former translates to o/s pipes and the latter to tcp sockets on the host machine, then horizon would be the REQ and the captive core process would be the REP, both using 0mq sdk to do send/recv's in a non-blocking way.

Another way to solve this issue is if core stored the n most recently applied ledgers in its db

Core has the MAX_SLOTS_TO_REMEMBER config, which by default is configured as 12, and indicates the number of ledgers that it remembers.

https://github.com/stellar/go/pull/4845 added a metric which can be used to calculate how many ledgers are stored in the in-memory queue. Note that these metrics are only available on staging because they have not yet been deployed in production. In the past 30 days we have never accumulated more than 1 ledger in the in-memory queue:
https://grafana.stellar-ops.com/explore?left={"datasource":"Prometheus","queries":[{"expr":"horizon_ingest_captive_stellar_core_latest_ledger{horizon_network%3D\"pubnet\", environment%3D\"stg\"}\n-\n\nhorizon_ingest_local_latest_ledger{environment%3D~\"stg\", horizon_network%3D~\"pubnet\"}","format":"time_series","instant":false,"interval":"","intervalFactor":4,"refId":"A","datasource":{"type":"prometheus","uid":"000000001"}}],"range":{"from":"now-30d","to":"now"}}&orgId=1

We discussed this issue during the onsite last week with the stellar core team. Core cannot restart from a ledger earlier than the LCL because we cannot rewind the state of bucket db to a previous ledger. So it looks like we need to implement a solution on the Horizon side (perhaps one of the two options mentioned in https://github.com/stellar/go/issues/5123#issuecomment-1820899429 )

Note that, even if we remove the explicit queue, we are still using a pipe to read the data (with an associated buffer).

As @sreuland says, replacing the pipe with a two-way communication protocol would solve the problem but that's probably a lot of work to implement and propagate downstream.

As @sreuland says, replacing the pipe with a two-way communication protocol would solve the problem but that's probably a lot of work to implement and propagate downstream.

two-way protocol could possibly allow us to decouple core and horizon or soroban-rpc in the future. This would be a big win as currently we have to bundle core in docker images and there is no way to mange versions independently.

Uhm, you still need to manage the captive core process (restart it on error etc ...)

Uhm, you still need to manage the captive core process (restart it on error etc ...)

Yes absolutely - this would have to be folded into the protocol. Core might also need to grow some additional functionality to allow that. I appreciate that this is hard of course but IMO this is the best long term strategy.
Right now, because of this tight coupling, we have to produce large-ish docker images and they go against best practices of having 1 pid per container. It also complicates our release process because core has to be baked into multiple images. We may be OK leaving it as is but I'm worried that one day we'll have to address this problem at inconvenient time. Doing this proactively would be nice.


Events that have a response event, should have retry attempts made if a response is not received.

JOIN should retry if JRES isnt received.
RDAT should retry if GDAT isnt received.


support for global enums

It would be good to have support for enum defined at components/schemas level and used in data models by $ref
Global Enum model
Files.FileLocation:
 type: string
 enum:
   - deleted
   - location1
   - location2

Files.FileUploadReply:
 type: object
 properties:
   location:
     $ref: '#/components/schemas/Files.FileLocation'
   filename:
     type: string

Generated Python
class FilesFileLocation(OpenApiObject):
    DELETED = "deleted" # type: str
    LOCATION1 = "location1" # type: str
    LOCATION2 = "location2" # type: str

class FilesFileUploadReply(OpenApiObject):
    _TYPES = {
        "location": {
            "type": str,
            "enum": [
                "deleted"
                "location1",
                "location2"
            ],
        }
    }

    @property
    def location(self):
        # type: -> Union[Listeral["deleted"],Literal["location1"],Literal["location2"]]
        return self._get_property("location")

    @location.setter
    def location(self, value):
        # type: (Union[Listeral["deleted"],Literal["location1"],Literal["location2"]])
        self._set_property("location", value)

Sample usage
upload_reply = FilesFileUploadReply()
upload_reply.location = FilesFileLocation.LOCATION1
assert upload_reply.location == FilesFileLocation.LOCATION1

This is already supported. OpenAPIArt uses x-include to specifically inline code as opposed to $ref which expects global objects.
Declare the global enum as follows:
Files.FileLocation:
 type: string
 enum:
   - deleted
   - location1
   - location2

Inline the enum where needed in the model using the x-include extension.
Object1:
  type: object
  properties:
    global_enum_1:
      x-include:
      - '#/components/schemas/Files.FileLocation'

Object2:
  type: object
  properties:
    global_enum_2:
      x-include:
      - '#/components/schemas/Files.FileLocation'

SDK code will be generated on a per attribute basis.


Non-breaking space added when focussing (empty) textbox

When I enter an empty textbox for the first time it starts with a non-breaking space character ( ). I have not found any way to avoid this.

Can you please update the issue with a Plunker re-creating it.
I will also start looking to see if I can reproduce.

Trying - haven't been able to reproduce (in Plunker) yet though

Closing this issue due to inactivity. Please feel free to open a new issue once you have a plunker demonstrating it.


BSimElasticPlugin: The lsh.zip archive does not contain the jar file

Describe the bug
After installing the BSimElasticPlugin extension, the lsh.zip archive does not contain the jar file lsh.jar that Elasticsearch needs. Installing this plugin in Elasticsearch will show up as successful, but rebooting Elasticsearch after the install leads to Elasticsearch crashing with the error:
Could not find plugin class [org.elasticsearch.plugin.analysis.lsh.AnalysisLSHPlugin]

Looking at the lsh.zip contents makes the error clear. The usual jar file is not present. It appears that the issue stems from the removal of libsDirName='ziplayout' from BSimElasticPlugin's build.gradle file in version 11.2. Adding this line back to the build file, building Ghidra, and installing the extension results in a normal looking lsh.zip file containing both files.
To Reproduce
Steps to reproduce the behavior:

Build Ghidra version 11.2 using gradle. I used the following commands:

gradle --init-script gradle/support/fetchDependencies.gradle init --overwrite
gradle buildNatives_linux64
gradle sleighCompile
gradle buildGhidra

Install the Extension either through the GUI or by unzipping the ghidra_11.2_DEV_20241014_BSimElasticPlugin.zip file to Ghidra/Extensions (the latter is the method I used).
Locate the lsh.zip file which should be in Ghidra/Extensions/BSimElasticPlugin/data/ and inspect the contents.
The only file in the archive is the plugin-descriptor.properties

Expected behavior
The archive should contain the lsh.jar file as well as the plugin-descriptor.properties file.
Screenshots
N/A
Attachments
N/A
Environment (please complete the following information):

OS: Ubuntu 22.04
Java Version: 21
Ghidra Version: 11.2
Ghidra Origin: locally built

Additional context
N/A

I checked this on my Windows installation of Ghidra and it is also missing the jar file. On Windows 10, I am running Ghidra version 11.2 with JDK 21.0.4. I installed Ghidra from the official GitHub release.


Moved log file location to $XDG_CACHE/ytermusic/log.txt

Moved the location that the log file is stored from cwd to $HOME/.cache/ytermusic/log.txt or OS alternative plus minor refactors

I'm sorry but I'll have to close this PR this as already been implemented on beta the exact same way...
Thanks a lot for contributing.


Timeline infinite looping does not work

Version

Phaser Version: 3.19

Operating system: Win10

Browser: Chrome

Description
If you set Timeline.loop = -1, it will not work. You can check it here: https://labs.phaser.io/view.html?src=src\tweens\timelines\loop%20timeline.js and set loop: -1.

Example Test Code

`timeline = this.tweens.timeline({
    targets: image,
    loop: -1,

    tweens: [
    {
        x: 700,
        ease: 'Sine.easeInOut',
        duration: 2000,
    }]});`

Additional Information

Works fine for me. If I change the loop counter to -1 then the crate just circles around the screen forever. Works in 3.20, 3.19 and 3.18 (but not 3.17).

That's strange because it does not work for me. Please check this video: https://drive.google.com/file/d/14Wt_w8AuahNTklCWAsMttuz_GQKVBsjD/view?usp=sharing

As I said, it doesn't work in 3.17.


OCPBUGS-38037: unable to build graph image in enclave environment

Description
When running oc-mirror v2 techPreview in mirror to disk mode in an air gapped environment, oc-mirror cannot reach api.openshift.com to download graph.tar.gz. This caused the mirroring to fail.
With this fix, and only for the context of air gapped environments (ie UPDATE_URL_OVERRIDE environment variable is defined) oc-mirror attempts to find the graph image in the oc-mirror cache instead, and skips the graph image (without failing) if the graph image isn't found.
Fixes # OCPBUGS-38037
Type of change
Please delete options that are not relevant.

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?
Unit test.
Automated E2E test
Expected Outcome
Unit tests & E2E tests pass.

/jira refresh

/jira refresh

/cherry-pick release-4.17

/label qe-approved

/lgtm


Azure TTS streaming support

Hi,
it seems that all supported TTS services utilize chunked streaming output except for Azure TTS.
Any particular reason for that?

The only reason was time. We've finally got around to doing this. This PR is in review: https://github.com/pipecat-ai/pipecat/pull/702.

#702 is merged. This will be in the next release.


Formating function in shared tooltips returns  "options.seriesIndex = 0"

tooltip: {
            shared: true,
             y: {
                formatter: function(value,options){
                  return options.seriesIndex;
              }
}

If tooltip is set to "shared:true" then "options.seriesIndex" is always 0,
but if tooltip is set to "shared:false" then "options.seriesIndex" is set accordingly
so if tooltip is shared you cannot find that value property in "options.series" with "seriesIndex" and "dataPointIndex"
see it in action
https://codepen.io/petr-suchomel/pen/dLGBya

i can confirm this issue, but for me seriesIndex is always 0, no matter what shared is set

@petrsuchomel25 I can see the problem, but seriesIndex is really not captured in shared tooltip because fetching the value just based on dataPointIndex works fine for tooltips. I will see how this can be changed.
@andreas83 Can you post a codepen? If the tooltip is not shared, then seriesIndex should be different based on mouse position in a multi-series chart.

@junedchhipa Thanks, very much so... in case You are wondering - Usecase is displaying percent values in tooltip...
https://codepen.io/petr-suchomel/pen/axmQxG

Hi I have the same issue and this CodePen shows what the issue is that I am facing.
I have two different sets of series. One is the main data that I want to show in the tooltip and the other is a normalized version of that data (and therefore the data in the arrays are different).
N.B. For simplicity ive just set the normalized arrays to have random data.
So essentially what I want to do is have the lines in the graph use the data from the normlizedArray and the tooltip to use the corresponding data from the realData array.
As you can see in the CodePen if you hover over the individual point it shows the value for that point for all 3 of the lines. If the seriesIndex returned the correct index I the code I have implemented would work fine. Switching off shared would work but I need to show all of the data within the tooltip.
I hope I have explained this well enough, if not let me know.
CodePen Example