status: add certificates

Summary
Add certificate status.
Related issues
Fixes https://github.com/pomerium/internal/issues/875
Checklist

[x] reference any related issues
[ ] updated docs
[ ] updated unit tests
[ ] updated UPGRADING.md
[x] add appropriate tag (improvement / bug / etc)
[x] ready for review

Seems to work:
Status:
  Certificates:
    Pomerium - Wildcard - Tls:  Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:<PHONE_NUMBER>8576882198563812213559728104 (0x1672d002603afdfd287d2832bfd917e8)
    Signature Algorithm: SHA256-RSA
        Issuer: O=mkcert development<EMAIL_ADDRESS>(Caleb Doxsey),CN=mkcert<EMAIL_ADDRESS>(Caleb Doxsey)
        Validity
            Not Before: Jan 3 22:57:51 2023 UTC
            Not After : Apr 3 21:57:51 2025 UTC
        Subject: O=mkcert development certificate,OU=caleb@lemur (Caleb Doxsey)
        Subject Public Key Info:
            Public Key Algorithm: RSA
                Public-Key: (2048 bit)
                Modulus:
                    af:60:1d:5b:d3:d8:d5:10:e5:4d:97:49:ca:89:b2:
                    8d:d6:4d:d9:ff:55:77:2e:7c:f4:f9:c6:57:39:32:
                    eb:51:10:6c:47:17:8d:8f:77:b2:99:6b:5b:17:64:
                    2d:bd:2b:2b:c8:69:55:d6:33:50:99:7c:f5:f2:73:
                    e5:08:52:4c:fe:54:91:ce:09:bf:a7:08:57:e4:f0:
                    92:08:c4:a8:e7:49:71:55:85:fd:6a:6e:b1:c3:49:
                    25:4d:c5:4e:7e:25:fe:93:20:5d:8c:de:8a:f4:7f:
                    59:d9:6a:93:65:60:d1:bd:82:f4:e1:73:87:e2:b7:
                    14:6f:d5:38:92:d1:4c:48:aa:94:fb:8e:ea:69:ef:
                    64:bd:7f:16:10:6a:26:dc:9f:01:df:ab:6b:f2:4b:
                    50:3f:ad:cc:5b:0b:68:a2:d7:1a:22:25:58:a9:f4:
                    f9:18:c6:81:52:2c:f6:de:ae:51:0b:c2:65:8c:5d:
                    e2:0a:14:d8:d4:47:f5:de:dc:c0:8c:3d:b3:77:7d:
                    48:60:5c:6f:93:75:c4:b3:b8:d0:b3:ba:96:88:55:
                    44:b5:aa:7c:26:75:92:28:08:ed:4d:b7:cc:f9:08:
                    35:54:ee:34:fc:32:dc:ec:ac:60:1d:df:84:b3:a1:
                    47:06:7a:c1:01:cb:cc:cf:d8:e3:eb:66:bb:06:b1:
                    8b
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                Server Authentication
            X509v3 Authority Key Identifier:
                keyid:72:44:53:B5:17:BC:DE:33:20:97:D0:EB:E7:1F:7C:2C:83:30:9C:BA
            X509v3 Subject Alternative Name:
                DNS:*.localhost.pomerium.io
    Signature Algorithm: SHA256-RSA
         cd:76:70:7a:d7:6c:de:05:a6:58:3b:6b:03:f0:7c:f7:bb:00:
         e1:78:27:ec:61:1e:57:5e:38:a4:50:2a:70:d4:c7:2c:48:07:
         c1:3c:57:3f:33:d7:82:0f:c0:e2:d5:e5:cd:81:3c:64:1d:99:
         8c:33:87:ba:6a:4f:2f:f7:6f:19:b9:39:9e:74:1d:0e:fb:f2:
         dc:0f:d4:e1:9b:66:fe:6f:c5:fc:4b:de:2b:f7:98:43:86:2d:
         70:ba:52:0e:61:e6:af:a2:d4:26:e0:1c:c0:63:46:e8:95:2e:
         1e:a8:41:64:53:c3:e8:03:91:7f:fd:b7:63:41:48:13:5b:3e:
         36:2c:5e:f6:83:fc:c5:6b:8b:c9:59:84:d8:c5:1b:76:da:8c:
         05:aa:63:33:b6:6e:17:6a:13:a1:73:3f:78:43:00:80:d8:c6:
         37:54:5d:c9:53:95:68:27:da:67:04:99:94:09:be:a5:e9:c8:
         e2:1a:b6:8e:45:42:7d:b8:5d:a0:ea:9b:9f:6c:42:8d:e1:eb:
         78:19:56:e4:79:e0:71:7c:61:b3:72:01:b8:f2:a5:da:22:3d:
         74:4e:3e:af:75:90:c6:bb:91:4a:4c:9c:d9:94:66:b3:58:dd:
         73:70:29:4c:85:f8:f4:1f:6a:dd:30:12:64:f8:32:0d:ba:38:
         43:f2:96:66:55:4e:8c:60:e2:63:82:a5:05:59:e6:aa:d7:ef:
         51:19:ee:06:15:dc:99:bb:4e:73:c1:ca:70:4b:6b:46:90:30:
         2b:da:1e:77:3c:a3:38:e3:ca:cb:e4:63:db:3b:fc:9e:8a:0d:
         bc:55:01:ad:4e:7a:98:a2:1d:85:b8:65:b9:35:bf:05:20:35:
         8b:35:d8:08:6d:2f:f5:3f:35:84:2d:65:c6:d7:75:69:78:3c:
         1d:fd:0c:05:e4:79:e8:f2:00:f7:b5:ed:1e:c1:81:f4:5c:b1:
         a8:e5:26:ef:c6:86:5c:ce:63:e1:23:b9:57:c9:92:3d:90:b6:
         a8:6d:34:12:16:4c

  Ingress:
    pomerium/verify:
      Observed At:          2023-01-03T23:13:38Z
      Observed Generation:  1
      Reconciled:           true
  Settings Status:
    Observed At:          2023-01-03T23:13:38Z
    Observed Generation:  2
    Reconciled:           true
    Warnings:
      storage: please specify a persistent storage backend, please see https://www.pomerium.com/docs/topics/data-storage#persistence

This was implemented improperly:

The status information should not be set by pomerium/config.go. It should be populated by the reporter. Since the reporter doesn't have the relevant information to construct this status, it will need to fetch this data.
The status text is too verbose. We should include a minimal amount of data. Since we rely on a 3rd party library to produce this text, we will need to build our own.


ReferenceError: self is not defined in Next.js pages

Describe the bug
When i use version 12 using page router self is not defined
Import trace for requested module: ./node_modules/.pnpm/@ffmpeg+ffmpeg<EMAIL_ADDRESS>./node_modules/.pnpm/@ffmpeg+ffmpeg<EMAIL_ADDRESS>./node_modules/.pnpm/@ffmpeg+ffmpeg<EMAIL_ADDRESS>⨯ ReferenceError: self is not defined     at Object.<anonymous> (/Users/redacted/Documents/test-app/node_modules/.pnpm/@ffmpeg+ffmpeg@0.12.6/node_modules/@ffmpeg/ffmpeg/dist/umd/ffmpeg.js:1:202)     at Module._compile (node:internal/modules/cjs/loader:1241:14)     at Module._extensions..js (node:internal/modules/cjs/loader:1295:10)     at Module.load (node:internal/modules/cjs/loader:1091:32)     at Module._load (node:internal/modules/cjs/loader:938:12)     at Module.require (node:internal/modules/cjs/loader:1115:19)     at mod.require (/Users/redacted/Documents/test-app/node_modules/.pnpm/next@13.5.4_react-dom@18.2.0_react@18.2.0/node_modules/next/dist/server/require-hook.js:64:28)     at require (node:internal/modules/helpers:130:18)     at @ffmpeg/ffmpeg (/Users/redacted/Documents/test-app/.next/server/pages/index.js:76:18)     at __webpack_require__ (/Users/redacted/Documents/test-app/.next/server/webpack-runtime.js:33:42)     at eval (webpack-internal:///./pages/index.tsx:7:72)     at ./pages/index.tsx (/Users/redacted/Documents/test-app/.next/server/pages/index.js:55:1)     at __webpack_require__ (/Users/redacted/Documents/test-app/.next/server/webpack-runtime.js:33:42)     at eval (webpack-internal:///./node_modules/.pnpm/next@13.5.4_react-dom@18.2.0_react@18.2.0/node_modules/next/dist/build/webpack/loaders/next-route-loader/index.js?kind=PAGES&page=%2F&preferredRegion=&absolutePagePath=.%2Fpages%2Findex.tsx&absoluteAppPath=private-next-pages%2F_app&absoluteDocumentPath=private-next-pages%2F_document&middlewareConfigBase64=e30%3D!:22:74)     at ./node_modules/.pnpm/next@13.5.4_react-dom@18.2.0_react@18.2.0/node_modules/next/dist/build/webpack/loaders/next-route-loader/index.js?kind=PAGES&page=%2F&preferredRegion=&absolutePagePath=.%2Fpages%2Findex.tsx&absoluteAppPath=private-next-pages%2F_app&absoluteDocumentPath=private-next-pages%2F_document&middlewareConfigBase64=e30%3D! (/Users/redacted/Documents/test-app/.next/server/pages/index.js:22:1) {   page: '/' }
Note that using app router - it runs well, i'm using ffmpeg.wasm in an existing project using page router therefore we are not ready to upgrade.
To Reproduce

npx create-next-app@latest

✔ What is your project named? … test-app
✔ Would you like to use TypeScript? …  Yes
✔ Would you like to use ESLint? … Yes
✔ Would you like to use Tailwind CSS? … Yes
✔ Would you like to use src/ directory? … No
✔ Would you like to use App Router? (recommended) … No
✔ Would you like to customize the default import alias (@/*)? … No

pnpm i @ffmpeg/ffmpeg @ffmpeg/util
copy example code from ffmpeg.wasm next.js example
run the app

This issue is observed to be solved by changing
const ffmpegRef = useRef(new FFmpeg()) to const ffmpegRef = useRef<FFmpeg>();
const ffmpeg = ffmpegRef.current to ffmpegRef.current = new FFmpeg();


Need to support on HPU servers

The feature
Support for HPU Habana hardware accelerator in runhouse
Motivation
With the increasing demand for high-performance computing and the need for faster processing of large-scale machine learning and deep learning workloads, HPUs have emerged as powerful hardware accelerators. These accelerators offer significant performance advantages over traditional CPUs and GPUs when it comes to tasks involving LLM, neural networks, large-scale data processing, and scientific simulations.
What the ideal solution looks like
By integrating support for HPUs in runhouse, you would provide developers with a platform that enables them to leverage these advanced hardware accelerators seamlessly. This would open up new possibilities for building and running computationally intensive applications and workflows directly on runhouse infrastructure.
For example:  the client would be able to remote launch applications on a HPU aws server by:
rh.cluster(name='rh-gaudi', instance_type='dl1.24xlarge', provider='aws').save()
https://aws.amazon.com/ec2/instance-types/dl1/
https://developer.habana.ai/
Additional context
HPU self hosted server as well.

Thanks for raising this! We'll speak with the SkyPilot folks, this might already be on their radar to support on-demand. In terms of self hosted support, curious if you had any issues using the BYO abstraction? Meaning something along the lines of:
c = rh.cluster(name="hpu-cluster", ips=['ip_address'], ssh_creds={'ssh_user': '...', 'ssh_private_key': '<path_to_key>'})

I am currently working on it. and runing into some irrelevent issues from time to time:
such as during the model execution. I believe that may be ssh configuration related issue.
File "/home/coco/.local/lib/python3.10/site-packages/sshtunnel.py", line 1174, in _raise
raise exception(reason)
sshtunnel.BaseSSHTunnelForwarderError: Could not establish session to SSH gateway

update, it is currently working now, so close this ticket.
Thanks @jlewitt1

Hey Henry, glad to hear it's working, and we're currently replacing the SSH tunnels to make them more stable (cloud agnostic networking is tricky...). Have you gotten any large LLMs running on HPU?

Thanks @dongreenberg, there are several LLMs currently working on HPU  now, thanks.


Inserting a char at the first position with already existing chars after that position.

The definition of the position of the inserted character cannot be determined by the insert inbetween algoirtm.
Ideas to fix:

Find a solution during the insertion on the client side.
Extend the insertion algorithm to include this use case
Update the complete list of positions when inserting a new first character
Possibly best solution: when inserting characters before the first, set the counter to negative numbers from -1 going more into the negative space with every new character

Filling an # with Position [0, 0] into the List without showing it to the user, so there is no case inserting a new character before another without a position Before the new character


Anonymizing

I think tou can (and should) anonymizing the data (not only the user view, really anonymizing) using other data source like bluetooth or wifi closeness between users.
If you need whatever, count with me :)

Actually I also had data annoymizing in mind too but couldn't wrap my head around how to do it. Please send me a mail here<EMAIL_ADDRESS>and lets discuss it.

Sended :)


RDNA3/7900XTX support

HI,
I'm trying to test ML stuff with my 7900XTX and am having issues with the latest MIOpen. I built tensorflow-upstream with RDNA3 support (https://1drv.ms/u/s!AiqVlRyhZQZtiJF5cBtF39kAWfAsqw?e=CxtIBI) and it works OK using ROCm5.6. However, if I install it in the latest MIOpen docker image, it crashes with the following error.

2023-08-29 14:45:10.883603: W tensorflow/compiler/mlir/tools/kernel_gen/tf_gpu_runtime_wrappers.cc:30] 'hipModuleLoadData(&module, data)' failed with 'hipErrorInvalidKernelFile'
2023-08-29 14:45:10.883628: W tensorflow/compiler/mlir/tools/kernel_gen/tf_gpu_runtime_wrappers.cc:30] 'hipModuleGetFunction(&function, module, kernel_name)' failed with 'hipErrorInvalidHandle'
2023-08-29 14:45:10.883640: W tensorflow/core/framework/op_kernel.cc:1827] INTERNAL: 'hipModuleLaunchKernel( function, gridX, gridY, gridZ, blockX, blockY, blockZ, 0, reinterpret_cast<hipStream_t>(stream), params, nullptr)' failed with 'hipErrorInvalidHandle'
Traceback (most recent call last):
File "/pwd/Downloads/bench.py", line 15, in
mod = tf.keras.Sequential([tf.keras.layers.InputLayer(inp_units), tf.keras.layers.Dense(2500, activation='relu'), tf.keras.layers.Dense(2500, activation='relu'), tf.keras.layers.Dense(250, activation='relu'), tf.keras.layers.Dense(nclasses, activation='softmax')])
File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/trackable/base.py", line 204, in _method_wrapper
result = method(self, *args, **kwargs)
File "/usr/local/lib/python3.9/dist-packages/keras/src/utils/traceback_utils.py", line 70, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/usr/local/lib/python3.9/dist-packages/keras/src/backend.py", line 2100, in random_uniform
return tf.random.stateless_uniform(
tensorflow.python.framework.errors_impl.InternalError: {{function_node _wrapped__Sub_device/job:localhost/replica:0/task:0/device:GPU:0}} 'hipModuleLaunchKernel( function, gridX, gridY, gridZ, blockX, blockY, blockZ, 0, reinterpret_cast<hipStream_t>(stream), params, nullptr)' failed with 'hipErrorInvalidHandle' [Op:Sub] name:

Does AMD plan to release RDNA3 support in 5.7? or Do we need to wait for 6.0. I see references to both versions being worked on and was hoping that it would be released sooner rather than later.
By the way, 7900XTX tensorflow performance is very bad. I wanted to confirm myself the benchmark numbers from https://cprimozic.net/notes/posts/machine-learning-benchmarks-on-the-7900-xtx/ and that's how I found this problem. I was hoping that ROCm5.7 woudl resolve the performance issue but I can't even get it to run. :(

I'm using a docker image from https://hub.docker.com/r/rocm/miopen/tags, ci_a15150 which was built a few days ago.

I got some more benchmark numbers.
https://github.com/RadeonOpenCompute/ROCm/issues/2098#issuecomment-1705482476
Direct link to git gist at https://gist.github.com/briansp2020/e885f0eb6cbec45fcaf0c2eac8c3ee11

BTW, I noticed that many of ROCmSoftwarePlatform docker images from https://hub.docker.com/u/rocm uses different base images. For containers that are not working on sub-components (tensorflow, pytorch, triton?), they should use same base image. It seems like all of them use different and use up too much space. Also, I had to figure out the environment every time I tried out a different container. They all had different Python verions (I saw 3.9, 3.10, 3.11 from both Ubuntu repo and conda) It will make people's lives much easier if AMD can standardize on the base image and python version. Not sure how easy that is to be honest since AI/ML is developing very fast and every body seems to have their own way of setting up their environment. But it would be helpful if AMD could standardize as much as possible.

AFAIK, images in https://hub.docker.com/u/rocm are not for end-users. End-users should work with the official releases of their framework of choice - TF, PyTorch, Triton, etc. Every framework decides what is the most appropriate for their community, such as the best version of Python and its type. Standardization would have to override those decisions to the extend, which requires a separate customer support track for each framework, along with thier dependencies.

@dmikushin
Though I agree that not all containers there are for end-users, it just seemed too fragmented. So much so that if people are trying to understand components of ROCm so that they can eventually contribute to AMD's open-source effort, they will have a tough time getting up to speed.
Also, I think it's a good idea for AMD to release ROCm+TF, ROCm+PT, ROCm+Triton reference docker images. Sure they can push that responsibility upstream. But since it takes time for the upstream to incorporate ROCm specific changes, it would be nice if AMD maintained a up-to-date container and/or instructions to set up end-user-facing ML/AI environments & examples using AMD hardware. As smaller player in ML/AI space, ROCm will not get as much attention from the upstream folks and AMD should promote ROCm on their own.
evshiron has a nice page with containers for various AI examples. It's a bit stale and it would be nice for AMD to officially have such a project to show people how to set up popular projects using AMD hardware.
Just my 2 cents.

@briansp2020 Please take a deep breath and accept the fragmentation as it is now. Otherwise, any generous attempt to reduce fragmentation would result into just one more piece of fragmentation. There is often a fragmentation even between two hands: Jupyter's own docker image is conda, however jupyterhub is mainline Python. Life is pain.

https://xkcd.com/927/

BTW. It would be nice to hear from people who know whether the 7900XTX performance issue is mainly from MIOpen or if there are other issues that need to be resolved. I have linked all 7900XTX issue tickets that mention TensorFlow to this one since I was assuming that once MIOpen is properly optimized for 7900XTX, the main issue will be resolved.
I have not linked 7900XTX issues that mention PyTorch only to this because I don't know whether MIOpen is the main bottleneck there as well.
Also, am I correct to assume that, eventually, GFX1100 will have its own kernels written in assembly? I don't see any files with gfx1100 in the name in src/kernels directory. So, I assumed 7900XTX has not seen any optimization at all other than compiler improvements. Is that a correct assumption? Sorry to ask so many questions but I'm very excited to see that we are very close to seeing the full potential of 7900XTX in ML/AI. It would be nice to know how much performance we can expect to get out of 7900XTX versus how much we are currently getting.

gfx1100 already has its own optimized kernels - you have to search for gfx11, not gfx1100. And ROCm by and large works for me, on a 7900XTX. It's a bit buggy, might leak memory and can get the entire kernel driver to enter a bad state and crash after a while, I certainly hope that'll be fixed. From my experience, when it works, Torch is mostly fine, Tensorflow is wonky and slow, and Triton is hot garbage on RDNA3 at this point. Don't know if ROCm 5.7 will improve things (I sure hope so!), but I believe some RDNA3 specific tweaks and optimizations are scheduled for ROCm 6.0.

@wsippel
What version of PyTorch are you using? I wasted 2 days trying to get Pytorch to run on my 7900XTX. The officially released version (PyTorch 2.0.1 for ROCm5.4.2) crashes when I try to create a variable on the GPU, which is understandable since it was released a long time ago.
However, the latest nightly build for ROCm 5.6 did not work either. It was driving me crazy because I was able to run some PyTorch code at some point in the past when trying out different configurations. It turned out that the only nightly build for ROCm5.5 is somewhat working. It seems their automated nightly build system is broken. I understand that things will break during development (ex. ROCm 5.7 test build has issues with TensorFlow at the moment) But ROCm5.6 is a released version and a nightly build for it should not fail so catastrophically...
BTW, I see files with gfx11 in the name in src/kernels directory. But I do not see any packages with gfx11 in the name. All I see are
miopen-hip-gfx1030-36kdb5.6.1
miopen-hip-gfx900-56kdb5.6.1
miopen-hip-gfx900-64kdb5.6.1
miopen-hip-gfx906-60kdb5.6.1
miopen-hip-gfx906-64kdb5.6.1
miopen-hip-gfx908-120kdb5.6.1
miopen-hip-gfx90a-104kdb5.6.1
miopen-hip-gfx90a-110kdb5.6.1

Should there be a miopen package for gfx11?
Also, do you know if there is an instruction on how to build PyTorch for ROCm? I can build TensorFlow & MIOpen. But I haven't seen the instructions on how to build PyTorch for ROCm.
Any pointer will be appreciated!

The missing kdb files are not an issue as far as I understand, missing kernels are compiled at runtime. And you have to use Pytorch nightlies. Pytorch comes bundled with its own copy of ROCm, stable still ships with ROCm 5.4, and you need at least 5.5 for RDNA3 support. And I've also noticed that some nightlies bundled with 5.6 don't work, so I use the ones with 5.5 on those days. Or you can build Pytorch 2.0 with gfx11 support if you want, it's what I did during ROCm 5.5 beta. I wrote those instructions back then for Stable Diffusion, might be a little outdated but should hopefully still work (for use in a Python venv):
export PYTORCH_ROCM_ARCH="gfx1100"
export CMAKE_PREFIX_PATH=[PATH TO YOUR VENV GOES HERE]
wget https://github.com/pytorch/pytorch/releases/download/v2.0.0/pytorch-v2.0.0.tar.gz
tar -xzvf pytorch-v2.0.0.tar.gz && cd pytorch-v2.0.0
pip install cmake ninja
pip install -r requirements.txt
pip install mkl mkl-include
python3 tools/amd_build/build_amd.py
python3 setup.py install

@wsippel
Thank you for the instruction. I followed your instructions and built it using ROCm5.6.1 and am getting the same error as using the nightly build.
(venv) root@rocm-ubuntu:~# python
Python 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'2.0.0a0'
>>> ls
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'ls' is not defined
>>> x=torch.rand(20000000, 128, device = 'cuda')
>>> x
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/venv/lib/python3.10/site-packages/torch/_tensor.py", line 426, in __repr__
    return torch._tensor_str._str(self, tensor_contents=tensor_contents)
  File "/root/venv/lib/python3.10/site-packages/torch/_tensor_str.py", line 636, in _str
    return _str_intern(self, tensor_contents=tensor_contents)
  File "/root/venv/lib/python3.10/site-packages/torch/_tensor_str.py", line 567, in _str_intern
    tensor_str = _tensor_str(self, indent)
  File "/root/venv/lib/python3.10/site-packages/torch/_tensor_str.py", line 327, in _tensor_str
    formatter = _Formatter(get_summarized_data(self) if summarize else self)
  File "/root/venv/lib/python3.10/site-packages/torch/_tensor_str.py", line 115, in __init__
    nonzero_finite_vals = torch.masked_select(
RuntimeError: HIP error: the operation cannot be performed in the present state
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing HIP_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

>>>

I'll try building with ROCm5.5 and see if that helps.

Weird...
Python 3.11.5 (main, Aug 28 2023, 20:02:58) [GCC 13.2.1 20230801] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'2.1.0.dev20230827+rocm5.6'
>>> ls
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'ls' is not defined
>>> x=torch.rand(20000000, 128, device = 'cuda')
>>> x
tensor([[0.5519, 0.3130, 0.8105,  ..., 0.1176, 0.5674, 0.1628],
        [0.4238, 0.6082, 0.1566,  ..., 0.8261, 0.5781, 0.2428],
        [0.9779, 0.4911, 0.8231,  ..., 0.9800, 0.3798, 0.9466],
        ...,
        [0.5105, 0.2872, 0.3043,  ..., 0.1184, 0.1820, 0.3528],
        [0.9375, 0.1680, 0.7187,  ..., 0.2480, 0.0707, 0.4545],
        [0.2336, 0.9598, 0.3362,  ..., 0.3665, 0.1882, 0.4274]],
       device='cuda:0')
>>>

@briansp2020 Please test with latest ROCm 6.1.2. If issue is resolved, please close ticket. Thanks!


extract() doesn't honor order of libs

The extract() function doesn't honor the order of the libs array (I'm not sure how it is ordering them).
I have some libraries that depend on others to be loaded prior. For example, many of my libs depend on jQuery so I want that first. I am also using Bootstrap with a few other related plugins, like Bootstrap Datepicker and Bootstrap Select, both of which need to be loaded after Bootstrap.
My webpack.mix.js file contains the following:
mix
    .js('resources/assets/js/app.js', 'public/js')
    .extract([
        'jquery',               // 1
        'bootstrap-sass',       // 2
        'bootstrap-datepicker', // 3
        'bootstrap-select',     // 4
        'moment',               // 5
        'vue',                  // 6
    ])
;

However, vendor.js combined them in the following order:
'moment',               // 5
'jquery',               // 1
'bootstrap-datepicker', // 3
'bootstrap-sass',       // 2
'bootstrap-select',     // 4
'vue',                  // 6

Laravel Mix Version: 0.9.2
Node Version: 7.8.0
NPM Version: 4.4.4
OS: macOS Sierra 10.12.4

It's expected behavior, webpack just throws specified chunks into bundle, it doesn't do much else unless you tell it so. For example, it is oblivious to the fact that you attach jquery to window object in the browser before loading bootstrap.
Thence you should autoload some dependencies like jQuery if you want to use extract. This will instruct webpack to inject them when requested rather than when it feels like it. Than everything will work like you expect.

Thank you, I reviewed the autoloading documentation and was able to utilize it like you suggested.


change locales to en_US.utf8

To resolve such problems (encountered with meson):
WARNING: You are using 'ANSI_X3.4-1968' which is not a a Unicode-compatible locale.
WARNING: You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents.
WARNING: Please switch to a UTF-8 locale for your platform.

Yep - sorry for forgetting this!!


Please invite me to the community

Name
theashishgavade
Additional Context
join

Invitation sent to join the GitHub Organization. Welcome to Infraform 🎉Please check your mail to accept the invitation sent by GitHub! 🤖About usInfraform is an open source solution for serverless platforms, it is designed for the development solutions and enhance creators to create their infrastructure 💻 Great to have you in our community ✨ Looking forward to your valuable contributions! 🤓 You can also check our community ✨ Github ✨ LinkedIn 💜 You can also share our community with other folks who want to learn development, DevOps, tech, and contribute together and with us 👉💜👈


allow customization of readiness probe settings

fixes: https://github.com/kiali/kiali/issues/7832

The PR https://github.com/kiali/kiali-operator/pull/847 supercedes this PR. Closing.


Preliminary changes allowing sharing relevant code between browser frontends

The 'add tab' icon is missing with this applied

There's propType warnings in https://travis-ci.org/mozilla/tofino/jobs/156845260

@bgrins Fixed the add tab problem. The propType warnings were there before too.

For some reason the builds are stalling at the end on CI. Will look into it, but everything else should be reviewable.

The move looks fine to me, just make sure to figure out the CI failures before landing

Coverage increased (+0.7%) to 42.95% when pulling 53c9b351e1d6cec222ceea9c4911b74cf99fede5 on victorporof:frontends-sharing into 590ed9d2561dd3406a75e846ce9b71b9b2cf9504 on mozilla:master.

Coverage increased (+0.7%) to 42.95% when pulling d560681f80974e257103c4967dc92bbcfe1babc1 on victorporof:frontends-sharing into 590ed9d2561dd3406a75e846ce9b71b9b2cf9504 on mozilla:master.

Coverage increased (+0.7%) to 42.95% when pulling e99970f197c6c04483a3d2a834983c13fad51392 on victorporof:frontends-sharing into 590ed9d2561dd3406a75e846ce9b71b9b2cf9504 on mozilla:master.

Coverage increased (+0.7%) to 42.95% when pulling e99970f197c6c04483a3d2a834983c13fad51392 on victorporof:frontends-sharing into 590ed9d2561dd3406a75e846ce9b71b9b2cf9504 on mozilla:master.


No such file or directory: 'params.npz'

Is params.npz a dataset? Anyone can tell me where can i get it please?

params.npz is a weight file that I created to compare the two implementations (the official one with tf and mine with spyx), so that I can have the same weights for each neural network.
I just pushed the file if you would like to test, but I strongly suggest fixing the code to use random initialization (it should be just uncommenting/commenting lines).

params.npz is a weight file that I created to compare the two implementations (the official one with tf and mine with spyx), so that I can have the same weights for each neural network.
I just pushed the file if you would like to test, but I strongly suggest fixing the code to use random initialization (it should be just uncommenting/commenting lines).

Yes, I saw that initialization process, and the code has run successfully. I can finally learn step by step. Thank you.


Reflect application list filters in the URL

If I applied a filter on the application list, like a text search and click on an application,
I want to easily navigate back from the application page to the filter application list.
Currently the filter settings are lost while after the view changed.
The filter settings could be exposed to the URL as query parameters.

Totally. And we should also support browser history. So glad we have react router ;)


Improve credentials management - [Vault provided]

Closes/fixes/resolves issue(s)?

closes #64

What was added/changed/fixed?
Related issue(s)? [Optional]
Others [Optional]
Checklist (after created PR)

[x] Added reviewers
[X] Added assignee(s)
[X] Added label(s)
[X] Added to project
[X] Added to milestone

Should the README be updated? :spiral_notepad:


Add qualifier to DriversSeat in TCK

This qualifier is required as part of the test configuration, putting it here means it is part of the codebase and not from how the implementation configures the TCK runner.

I think the TCK instructions are pretty clear in that this is supposed to be a configuration thing, and hence not part of the codebase. Similarly, you could also add @Named("spare") to the SpareTire class.
Now, I have no idea why it's been designed like that, but I have to assume that it was a conscious choice. At least, I'd expect the instructions in the README (and in the Tck class) to be updated in this PR as well.

What's the motivation for this change?
Like Ladislav said, it is not the sole occurrence and I'd also like to believe that the choice was conscious, Besides, it might be breaking for whoever is running the TCKs,

This change allows HK2 to pass, and it does not prevent Weld from passing (which currently does)

I'd also like to highlight that forking AtInject into Jakarta doesn't mean we can change it willy nilly. At least some of the well-known previous implementors of AtInject are well aware that this exists and are considering to support it:

https://github.com/google/guice/issues/1383
https://github.com/google/dagger/issues/2058

So, I do think we should keep these projects in mind.

and it does not prevent Weld from passing (which currently does)

Did you try? I didn't, so I can't guarantee it ;-)

So, I do think we should keep these projects in mind.

+1, I'd say we don't change it. What's blocking HK2 from passing it in this form of other implementations can pass it?


'IncTrainer' object has no attribute '_load_state_dict_in_model'

System Info
| name         | version                  |
|--------------|--------------------------|
| Linux Kernel | 5.4.0-122-generic x86_64 |
| CUDA Version | 11.7                     |
| CUDA Driver  | 515.65.01                |
| Python       | 3.10.5                   |
| torch        | 1.12.1+cu116             |
| transformers | 4.22.1                   |
| optimum      | 1.4.0                    |

Who can help?
@echarlaix
Information

[ ] The official example scripts
[X] My own modified scripts

Tasks

[X] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[ ] My own task or dataset (give details below)

Reproduction
The use of load_best_model_at_end causes an exception when training with the IncTrainer
from pathlib import Path

DATA_FOLDER = Path("a-folder")
DATA_FOLDER.mkdir(exist_ok=True, parents=True)

from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
model = AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased")

from typing import Any, Dict
from datasets import load_dataset

def tokenize(row: Dict[str, Any]) -> Dict[str, Any]:
    return tokenizer(row["sentence"], return_attention_mask=False, truncation=True)

sst_dataset = load_dataset("sst2")
sst_dataset = sst_dataset.map(tokenize)
sst_dataset = sst_dataset.remove_columns(["idx", "sentence"])
train_dataset = sst_dataset["train"]
eval_dataset = sst_dataset["validation"]

from typing import Any, Dict
from transformers import EvalPrediction
import evaluate

clf_metrics = evaluate.combine(["accuracy", "f1", "precision", "recall"])

def compute_metrics(results: EvalPrediction) -> Dict[str, Any]:
    return clf_metrics.compute(
        predictions=results.predictions.argmax(axis=1),
        references=results.label_ids,
    )

from transformers import TrainingArguments, DataCollatorWithPadding
from optimum.intel.neural_compressor import IncTrainer
from optimum.intel.neural_compressor.trainer import logger
import logging

logger.info = lambda *args: None
# logger.setLevel(logging.ERROR)

data_collator = DataCollatorWithPadding(tokenizer=tokenizer)

# Initialize our IncTrainer
training_arguments = TrainingArguments(
    output_dir=DATA_FOLDER / "runs",
    report_to=[],
    num_train_epochs=3.0,
    evaluation_strategy="steps",
    load_best_model_at_end=True,
    metric_for_best_model="accuracy",
    greater_is_better=True,
)
trainer = IncTrainer(
    model=model,
    args=training_arguments,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    compute_metrics=compute_metrics,
    tokenizer=tokenizer,
    data_collator=data_collator,
)

trainer.train()

This code produces the error:
File ~/.local/share/virtualenvs/blog-1tuLwbZm/lib/python3.10/site-packages/optimum/intel/neural_compressor/trainer.py:455, in IncTrainer.train(self, agent, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)
    453     state_dict = torch.load(best_model_path, map_location="cpu")
    454     # If the model is on the GPU, it still works!
--> 455     self._load_state_dict_in_model(state_dict)
    456 else:
    457     logger.warn(
    458         f"Could not locate the best model at {best_model_path}, if you are running a distributed training "
    459         "on multiple nodes, you should activate `--save_on_each_node`."
    460     )

AttributeError: 'IncTrainer' object has no attribute '_load_state_dict_in_model'

Expected behavior
The documentation suggests that IncTrainer is a drop in replacement for the transformers Trainer. As such I would expect this code to work.

Hi @matthewfranglen,
Thanks for reporting this issue. Could you upgrade you transformers version to 4.20.0 or higher ?


[WIP] Upgrade spark-3.3 to 3.3.1

What problem does this PR solve?
What is changed and how it works?
Check List
Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Code changes

Has exported function/method change
Has exported variable/fields change
Has interface methods change
Has persistent data change

Side effects

Possible performance regression
Increased code complexity
Breaking backward compatibility

Related changes

Need to cherry-pick to the release branch
Need to update the documentation
Need to update the tidb-ansible repository
Need to be included in the release note

/run-all-tests

3.3.1 is not released by Spark official. If you want to use TiSpark with it, you need to build it yourself. We will upgrade after it releases

I know. I'm the Spark 3.3.1 release manager. This PR is for testing compatibility.


Winless minifying everything (even when its set not to)

Hi,
As the description says - Winless is minifying everything for me, even when I checked the settings and I chose not to, I even checked the settings XML file and verified the auto minify tag is set to false but this is still happening.
Can anyone help me to solve this issue?
Many thanks

I tried replicating this issue but couldn't seem to get the same result you described. I attempted it with both version 1.8 and 1.9 by compiling with the minification option set to both true and false. I also tried enabling minification by default in the settings and still did not get the behavior. I am not a contributor in this project, therefore can give you no solid answer. One thing I would check is to ensure is that you are viewing the correct output file.


pushd /gopath/src/github.com/etcd-io/etcdlabs is error

should update to pushd /gopath/src/github.com/coreos/etcdlabs

Try updating the Dockerfile on line 1 to FROM ubuntu:18.04, works for me.


9p shared folders can't be written from guest

On Ubuntu Trusty, libvirtd starts qemu procs as user libvirtd-qemu, group kvm.   vagrant-libvirt writes the libvirt XML with "accessmode=passthrough" (hardcoded), and that means that the uid/gid are passed through from the guest to the host when creating files; this leads to a permission error from qemu when attempting to write a shared file of type 9p, since vagrant/vagrant doesn't exist on the vmhost.
Workaround: use "accessmode=mapped" instead, which translates vagrant/vagrant to libvirtd-qemu/kvm.  I did this by hacking filesystem.xml.erb, but that's because I don't speak enough Ruby to easily figure out how to make it settable in the Vagrantfile through cap/synced_folder.rb's folder_opts.merge!() call.

same here trying 9p for the first time.
vagrant@leap-wheezy:~$ mount | grep vagrant
/vagrant on /vagrant type 9p (rw,relatime,sync,dirsync,trans=virtio,version=9p2000.L)

vagrant@leap-wheezy:~$ touch /vagrant/test
touch: cannot touch `/vagrant/test': Permission denied

from the domain xml:
…
    <filesystem type='mount' accessmode='passthrough'>
      <driver type='path' wrpolicy='immediate'/>
      <source dir='/home/varac/vagrant/vms/wheezy-tmp'/>
      <target dir='/vagrant'/>
      <alias name='fs0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </filesystem>
…

btw, this is not too important for me, i'll just stick to nfs for now.

With latest version of vagrant-libvirt (0.0.24) it is possible to specify the access mode:
config.vm.synced_folder ".", "/vagrant", type: "9p", accessmode: "mapped"

it works, but not out of the box.
it i create a directory owned by myself, and fire up a vagrant vm in it, everthing works.
if i add a file on the host, i see it on th guest.
BUT when i try to modify this file (test-host), i get a permission denied:
vagrant@leap-wheezy:/vagrant$ echo test >> test-host
-bash: test-host: Permission denied

if i try to create a file on the guest, also:
vagrant@leap-wheezy:/vagrant$ touch test-guest
touch: cannot touch `test-guest': Permission denied

if make the directory on the host world-writable (chmod 777 .), i can create a file from the guest (test-guest), but it has libvirt-qemu as owner:
--- vms/wheezy-tmp » ls -al
total 20
drwxrwxrwx 3 varac        varac 4096 Mar 20 16:57 .
drwxrwxr-x 7 varac        varac 4096 Feb  2 11:07 ..
-rw------- 1 libvirt-qemu kvm      0 Mar 20 16:57 test-guest
-rw-rw-r-- 1 varac        varac    0 Mar 20 16:56 test-host
drwxrwxr-x 3 varac        varac 4096 Feb  2 09:43 .vagrant
-rw-rw-r-- 1 varac        varac 3134 Mar 20 16:55 Vagrantfile

is there a way to have the owner mapped to the user who runs vagrant ? this would be much more convinient

https://github.com/pradels/vagrant-libvirt/commit/4af0163c5eef4dd0a897d65601517136f768e3c9
for drop priv. in guest - use accessmode squash
and on mount step you can specify owner
should like
config.vm.synced_folder './', '/vagrant', type: '9p', disabled: false, accessmode: "squash", owner: 'anyuser'


Unhandled exception on removing related app

Bug Description
The traefik k8s operator throws an unhandled exception when a related app is removed.
To Reproduce

Build and deploy traefik from current git main
Set and external hostname
Add a relation with an IngressPerAppRequirer
Remove the relation after it is established.

Environment
Built and deployed from github on microk8s
Relevant log output
unit-traefik-ingress-0: 10:20:19 ERROR unit.traefik-ingress/0.juju-log ingress:1: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "./src/charm.py", line 603, in <module>
    main(TraefikIngressCharm)
  File "/var/lib/juju/agents/unit-traefik-ingress-0/charm/venv/ops/main.py", line 426, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-traefik-ingress-0/charm/venv/ops/main.py", line 142, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-traefik-ingress-0/charm/venv/ops/framework.py", line 276, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-traefik-ingress-0/charm/venv/ops/framework.py", line 736, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-traefik-ingress-0/charm/venv/ops/framework.py", line 783, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-traefik-ingress-0/charm/lib/charms/traefik_k8s/v0/ingress.py", line 250, in _handle_relation_broken
    self.on.data_removed.emit(event.relation)
  File "/var/lib/juju/agents/unit-traefik-ingress-0/charm/venv/ops/framework.py", line 276, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-traefik-ingress-0/charm/venv/ops/framework.py", line 736, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-traefik-ingress-0/charm/venv/ops/framework.py", line 783, in _reemit
    custom_handler(event)
  File "./src/charm.py", line 273, in _handle_ingress_data_removed
    self._wipe_ingress_for_relation(event.relation)
  File "./src/charm.py", line 498, in _wipe_ingress_for_relation
    if request := self.ingress_per_app.get_request(relation):
AttributeError: 'IngressPerAppProvider' object has no attribute 'get_request'
unit-traefik-ingress-0: 10:20:19 ERROR juju.worker.uniter.operation hook "ingress-relation-broken" (via hook dispatching script: dispatch) failed: exit status 1

Additional context
No response

Huh. thought I fixed that two days ago. Thanks!


Ingress settings with Nebari helm extension are not working

OS system and architecture in which you are running QHub
Linux
Expected behavior
I am trying to setup Airflow through Nebari helm extension. I am able to set it up, I could verify it running with port forward to localhost. I am trying to make it available on our domain, so I setup following ingress route and middleware. When I try to access the URL, I could see the request in the Airflow webserver log, but returning 404 error. Expected result is to return Http 200 OK
Actual behavior
Getting 404 error
How to Reproduce the problem?
Please see the attached qhub-config.yml, ingress.yml, middleware.yml to reproduce the issue on Linux env.
airflow-ingress.zip
Command output
No response
Versions and dependencies used.
conda = 4.12.0
Qhub = 0.4.3
kubectl
Client Version: v1.24.4
Kustomize Version: v4.5.4
Compute environment
AWS
Integrations
No response
Anything else?
No response

Hi @konkapv could you post the error msg output (GET) from the api server?

@viniciusdc

Connected to mss-onyx-dev.metrostar.cloud (<IP_ADDRESS>) port 443 (#0)
ALPN, offering h2
ALPN, offering http/1.1
TLSv1.3 (OUT), TLS handshake, Client hello (1):
TLSv1.3 (IN), TLS handshake, Server hello (2):
TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
TLSv1.3 (IN), TLS handshake, Certificate (11):
TLSv1.3 (IN), TLS handshake, CERT verify (15):
TLSv1.3 (IN), TLS handshake, Finished (20):
TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
TLSv1.3 (OUT), TLS handshake, Finished (20):
SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
ALPN, server accepted to use h2
Server certificate:
subject: CN=TRAEFIK DEFAULT CERT
start date: Oct  3 20:15:25 2022 GMT
expire date: Oct  3 20:15:25 2023 GMT
issuer: CN=TRAEFIK DEFAULT CERT
SSL certificate verify result: self signed certificate (18), continuing anyway.
Using HTTP2, server supports multiplexing
Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
h2h3 [:method: HEAD]
h2h3 [:path: /airflow/]
h2h3 [:scheme: https]
h2h3 [:authority: mss-onyx-dev.metrostar.cloud]
h2h3 [user-agent: curl/7.82.0]
h2h3 [accept: /]
Using Stream ID: 1 (easy handle 0x7fffd4d0be90)

HEAD /airflow/ HTTP/2
Host: mss-onyx-dev.metrostar.cloud
user-agent: curl/7.82.0
accept: /

TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 404
HTTP/2 404
< content-type: text/plain; charset=utf-8
content-type: text/plain; charset=utf-8
< x-content-type-options: nosniff
x-content-type-options: nosniff
< content-length: 19
content-length: 19
< date: Mon, 03 Oct 2022 20:17:12 GMT
date: Mon, 03 Oct 2022 20:17:12 GMT

<

Connection #0 to host mss-onyx-dev.metrostar.cloud left intact

Thanks, @konkapv. I will try to replicate this here and will follow back after some debugging.

We will need to try and recreate this issue and then we can follow up.


refactor(card): convert card to shadow

There quite a lot of CSS that will stop working, which might be fine, but in that case it should be removed


Locators/markers timecode in cmx3600 and otio formats

Hi all,
If I'm not mistaken, OTIO stores the EDL information without absolute time codes for the edited timeline, but using durations instead. That is, it keeps an ordered list of gaps and clips and accumulates their duration, so we don't have the explicit start time for each segment (although we can compute it easily if we have all the data, as it is implicit).
This is becoming problematic in the case of the locators in the cmx format. From the screening_example.edl, the position of the first marker is specified as:
004  ZZ100_50 V     C        01:00:10:01 01:00:14:20 00:59:58:00 01:00:02:19

FROM CLIP NAME:  ZZ100_504C (LAY1)
LOC: 01:00:01:14 RED     ANIM FIX NEEDED

That is, using a time code in the absolute range (edited timeline). If we use the cmx adapter to read it as 'otio', that same time code is used, but it does not mean anything in the OTIO world, because we lost all the rest of absolute time codes. After conversion, it looks like this:
"start_time": {
"OTIO_SCHEMA": "RationalTime.1",
"rate": 24.0,
"value": 86438.0  # corresponds to 01:00:01:14
}
Then, after this point, every time we want to use this otio edl, this marker won't be read correctly. For example, trying to visualize it with the otioview.py tool, it won't be displayed, as it tries to place it in the clip range and that value is out of it. Also, if we try to save it back in the cmx format, it will be written as:
004  AX       V     C        01:00:10:01 01:00:14:20 00:00:04:13 00:00:09:08

FROM CLIP NAME:  ZZ100_504C (LAY1)
LOC: 01:00:01:14 RED     ANIM FIX NEEDED

which is incorrect, as it is using the original absolute time code, and in the new cmx edl the start-end range of the corresponding clip changed, so the marker falls out of it again.
I'm not sure if this behavior is the expected, but in my opinion the absolute time code of the markers should also be stored as a relative time code within the corresponding clip, so we can always convert it between ranges and formats. I have a fix that does this, and works for my use cases, but I wanted to check with you guys to see if I'm missing something (I started with this a few days ago, so I might be confused) or it is an actual problem. If this is the case, I could push my changes and send a pull request.
Thanks in advance.

Thanks for the detailed bug report!
OTIO supports markers at the clip level, at the sequence level, or on any other subclass of Item. In this case it sounds like the EDL adapter is not doing the right thing with the markers. It should either put them at the sequence level with a time value relative to the whole thing, or it should put them inside the clip with a time relative to that clip. Currently it does a mix of the two which is incorrect.
In your use case, do the application(s) making or consuming EDLs support markers at the clip level vs the sequence level? I'm not sure if there is a way to distinguish the two in an EDL, so we might have to just pick one.
Either way, we should provide some sample code for moving a marker from a clip to the sequence and vice versa to illustrate the difference.
If you have a fix already, you can make a pull request so we can look at it in detail.

@jminor thanks for your reply!
So far, we evaluate markers in cmx3600 EDLs at a sequence level, but we still don't have a real life definitive use case, so I think we can adapt to any of both alternatives. In our provisional fix, while reading, I convert the time code of the marker and use a time relative to the clip where it is placed. Then, if needed, I convert it back to absolute, e.g. for writing in cmx format.
I'll try to make a pull request in the following days, so we can discuss further after you have a look.

@jafyvilla Did you get this worked out?  Its been quiet for some time, just wanted to clean out old issues in the issue tracker.
Thanks!

Hi @ssteinbach, sorry for the late reply, I overlooked your comment.
Unfortunately I had to stop working on this for some time but during the next few weeks I'll probably need to get back to it. I will post an update if I face the issue again, and if my patch does the trick I'll send a PR.
Thank you!


powerd process in macOS Ventura Beta 4/5

If you encounter powerd process or the inability to sleep (automatically wake up) in macOS Ventura Beta 4/5.Please run sudo pmset schedule cancelall in terminal.And enable Kernel-Patch-Disable RTC wake scheduling

macOS Ventura Beta 6 has fixed it up.


Please Update vue-template-compiler to the latest version

Laravel Mix Version:1.2.2
Node Version: 6.11.1
Yarn Version: 0.27.5
OS: Ubuntu 16.04

Description:
as the title says, i have an issue when using vue version 2.4.1 its requires<EMAIL_ADDRESS>

@bunnypro
For the time being you can install your own on top of Laravel mix
npm install<EMAIL_ADDRESS>--save-dev
# YARN
yarn add<EMAIL_ADDRESS>
Now webpack will use your version instead of installed by laravel mix. You may want to update vue loader as well.

If it ever gets out of sync, either manually install vue-template-compiler like @ankurk91 suggested, or nuke your node_modules directory and reinstall.

a simpler solution
yarn remove laravel-mix
# then
yarn add laravel-mix

@ctf0 your solution worked for me.
Deleting node_modules and reinstalling did not.

If you delete your node_modes to update you may have to delete your lock file too.


:books: Refine error message to show how to install mmengine.

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.
Motivation
Refine error message to tell the user how to install mmengine.
Modification
Modifies the error message in mim with:
Please install mmengine to use the download command: mim install mmengine.'

BC-breaking (Optional)
Does the modification introduce changes that break the back-compatibility of the downstream repos?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.
Use cases (Optional)
If this PR introduces a new feature, it is better to list some use cases here, and update the documentation.
Checklist

Pre-commit or other linting tools are used to fix the potential lint issues.
The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
If the modification has potential influence on downstream projects, this PR should be tested with downstream projects, like MMDet or MMCls.
The documentation has been modified accordingly, like docstring or example tutorials.

Reference issue: https://github.com/open-mmlab/mmsegmentation/issues/2072

Hi @saratrajput , thanks for the contribution!
The gridsearch command have the same error message should be modified: https://github.com/open-mmlab/mim/blob/ffaff8a4ece8a833b20dc7c6c1a7f8647d9eec15/mim/commands/gridsearch.py#L275
And make sure you have pre-commit your code before committing. ^_^
pip install -U pre-commit
pre-commit install
git commit -m "xxx"  # this will trigger pre-commit

@ice-tong Sorry for the late update. I've split the error message to avoid the linting errors, and updated the error message in gridsearch.py as well.
The error message is displayed as shown in the below images.

Hi @saratrajput, thanks for the kind contribution!
Since the main branch has been changed, you should rebase the main branch and modify that again. ^_^

@ice-tong Done.

Hi @saratrajput , thanks for your contribution!
please @zhouzaida  have a look at this PR~


bug: fixes format validation #199

The fix for CircularReferenceCounter is not sufficient.
The value should be larger: openapi3.CircularReferenceCounter = 20
It also needs to be added to LoadDocumentFromURI


[Bug Fix] : Fetch actions by id during refactor action name to solve IncorrectResultSizeDataAccessException for pages where duplicate action names exist

Fixes #2684

Server side LGTM, I'm guessing the test case from the previous PR applies to this one as well? @trishaanand

Server side LGTM, I'm guessing the test case from the previous PR applies to this one as well? @trishaanand

After the PR for disallowing duplicate named actions, we wouldn't be able to recreate this scenario in test. Let me try to write a test case for this without using the createAction service function to reproduce the bug and assert that the fix works.


Error pages do not associate events with elements

Describe the bug
In error pages, in production mode and rendered on the server, events are not associated with elements. This can be seen when we click on the search lupe, and it goes to a page not found:
http://localhost:3000/undefined/search?SearchableText=
In vanilla Plone this is not a problem, as we don't have many clickable elements on the page. But on sites that have more links this is a problem, since no Universal Link works.
To Reproduce
Steps to reproduce the behavior:

Start Volto in production mode:

pnpm build && pnpm --filter @plone/volto start:prod

In the Browser's address bar, type a non-existent URL. For example: http://localhost:3000/xxxxx
Press Enter.
Click on the search lupe.

Expected behavior
The Browser should be redirected to the search page:
http://localhost:3000/search?SearchableText=
Software (please complete the following information):

OS: Ubuntu
Browser: Chrome
Volto Version: 18
Plone Version: 6.0.11

Additional context
In the situation described above, javascripts are not inserted into the page, causing the problem to occur.
This is because the extractScripts parameter is passed as undefined here:
https://github.com/plone/volto/blob/082e919d372a96af0ab40938375a65c6bdeb0c34/packages/volto/src/server.jsx#L289-L292
This causes this code not to be executed:
https://github.com/plone/volto/blob/0edcd8083afee783a6463a14b18d0f712cf7c25b/packages/volto/src/helpers/Html/Html.jsx#L189-L197
This has been implemented by @tiberiuichim in:
https://github.com/plone/volto/pull/1948
I think this code should be executed, regardless of whether it is in development or production mode. Is there any reason not to load scripts on error pages?

@wesleybl From the PR you linked, it looks like this was done to avoid hydrating the markup returned from the server (which causes additional, unnecessary requests).
But we've also had to change this for some projects, and it seems quite bad to have it work differently in production mode only. I would support fixing this.

From the PR you linked, it looks like this was done to avoid hydrating the markup returned from the server (which causes additional, unnecessary requests).

@davisagli maybe that was a problem at the time. But today I don't see that problem. The only extra request, when using extractScripts true, is the request to assemble the navigation. But this is common and occurs on all pages.
I am in favor of removing the extractScripts prop:
https://github.com/plone/volto/blob/0edcd8083afee783a6463a14b18d0f712cf7c25b/packages/volto/src/helpers/Html/Html.jsx#L190
Do you agree?

I'm +1 on getting rid of extractScripts


move to an electron-like api

fix #10, fix #26, Fix #35

Sketch v49 just landed, any clue when this will be merged? I am starting a project and would love to use this.

This is not really related to Sketch 49. There are still quite a few bugs in this PR and I, unfortunately, don't have much time to work on it. If you want to test it a bit more, I'd happy to merge it once it's in a good shape!

@jenil It's also possible to install and use it from the branch if you're willing to take the risk of future breaking changes.
npm install https://github.com/skpm/sketch-module-web-view.git#electron-api

okay got it, I'm taking the risk. Thanks @mathieudutour @cameronmcefee

🎉 Nice. For my own solution I just hid the title bar entirely which removed the problem


How to clear cache of existing data

Hello, When I log into another account with my app, data from the old account are shown for the new one. I have tried using fetchPolicy: FetchPolicy.noCache, but to no avail. I also tried using a reset on the NormalisedInMemoryCache on user login. But I dont think I am doing it right. I need help please.

Quoting https://github.com/zino-app/graphql-flutter/issues/354#issuecomment-585860138:
client.cache.reset(); // empty the hash map
await client.cache.save(); // persist empty map to storage

if you're doing so and the cache still isn't clearing, or if fetchPolicy: FetchPolicy.noCache is incorrectly reading from the cache, please open new issue with code demonstrating the issue


 Each attempt was met with the same nebulous error:

In this case, the "specified part" turned out to be a missing LICENSE.md (something that wasn't a blocker until recently). Had the error stated which file(s) I neglected to include in the .vsix, it would've saved days of frustration.
Expected behaviour
An error message that elucidates which "specified part" is being referred to.
Desktop

OS: macOS 12.6.7 (Monterey), build 21G651
Browser: Observed with Firefox 114.0.2 (default browser); Edge exhibits the same behaviour.

Additional context
I generate this .vsix manually in lieu of using vsce.

The reference to LICENSE.md exists in the package manifest for about a year (since April, 5th, 2022), but copying the file itself was added to packaging target in Makefile only couple of days ago (I'd guess as an attempt to address the publishing failure). So, this is the web site complaining (in rather unhelpful way) about a file that is declared in manifest, but missing in the package. Assuming that "2 years ago" from the screenshot is actual previous publish time, that's not a new requirement that you ran into, but a result of a change of the manifest that happened after the package was published last (a year ago).
Regardless, the message needs to be clearer.


Add structural awareness of moved nodes in get_tree_diff

Is your feature request related to a problem? Please describe.
I'd like to be able to see not only what the newly added and the removed nodes are in the tree, but I'd like the builtin diff method (get_tree_diff) to also let me know if a certain node has moved to a different location in the tree, rather than showing it as a removed node and an added node.
Describe the solution you'd like
In addition to showing the annotated (+) and (-) also add (moved from) and (moved to). Maybe with an option to list out the full or relative path moved from/to, i.e. (moved from: a/b/c) and (moved to: a/b/d).
Describe alternatives you've considered
nutree has a nice diff output shown here.

They even have a really nice way of visualizing the diff here

Additional context
I love this repo by the way, easy to use and very clear! You've done a great job here.
Final Check

[X] I will watch this issue to stay updated with any comments or requests for additional information. I understand that timely responses will help in the evaluation and potential implementation of the feature.

[X] I have reviewed the documentation and existing feature requests to ensure that my request is unique and hasn't been previously proposed.

Thanks for your support! I have not heard of nutree before, will check that repo out. Just to check - in terms of indicating  and  does it reconcile this 'movement' based on the nodes having the same name?

This is now implemented in bigtree v0.22.0, do perform pip install --upgrade bigtree.
The usage is as such:
from bigtree import get_tree_diff, list_to_tree
root = list_to_tree([
    "Downloads/Pictures/photo1.jpg",
    "Downloads/Pictures/photo2.jpg",
    "Downloads/file1.doc",
    "Downloads/photo3.jpg",
])
root.show()
# Downloads
# ├── Pictures
# │   ├── photo1.jpg
# │   └── photo2.jpg
# ├── file1.doc
# └── photo3.jpg

root_other = list_to_tree([
    "Downloads/Pictures/photo1.jpg",
    "Downloads/Pictures/photo3.jpg",
    "Downloads/file1.doc",
    "Downloads/file2.doc",
])
root_other.show()
# Downloads
# ├── Pictures
# │   ├── photo1.jpg
# │   └── photo3.jpg
# ├── file1.doc
# └── file2.doc

tree_diff = get_tree_diff(root, root_other, detail=True)
tree_diff.show()

This will result in a tree
Downloads
├── Pictures
│   ├── photo2.jpg (removed)
│   └── photo3.jpg (moved to)
├── file2.doc (added)
└── photo3.jpg (moved from)

This is also reflected in the updated docs. In terms of visualisation, there is nothing changed, you can already indicate different node/edge appearance for different nodes. This way there is more flexibility for your graph appearance. For example,
from bigtree import Node, tree_to_dot

def get_node_attribute(node: Node):
    if node.node_name.endswith(("(moved from)", "(removed)")):
        return {"style": "filled", "fillcolor": "red"}
    elif node.node_name.endswith(("(moved to)", "(added)")):
        return {"style": "filled", "fillcolor": "green"}
    return {}

graph = tree_to_dot(tree_diff, node_attr=get_node_attribute)
graph.write_png("tree_diff_detail.png")

@kayjan Wow! That was quick! Thanks for the adding that feature in!
I just tested it out on the original example and I get the following output:
Tree Diff
Root
├── A
│   ├── a1
│   │   └── a11 (removed)
│   └── a2
│       └── a21 (added)
├── B
│   └── b1 (moved from)
│       └── b11 (moved from)
└── C (added)
    └── b1 (moved to)
        └── b11 (moved to)

I don't think node b11 should be shown as being moved, since even tho the full path has changed for it, it's path relative to the parent (or superparent) which has moved, has not changed. This causes some redunduncies and clutters the diff output, as with moved to/moved from we only want to show parents being changed (i.e. the structural change). b11 remains intact.

Sweet! I do have another nit tho... Sorry...
When I run the above example I now get the below output:
Old Tree
Root
├── A
│   ├── a1
│   │   ├── a11
│   │   └── a12
│   └── a2
└── B
    └── b1
        └── b11

New Tree
Root
├── A
│   ├── a1
│   │   └── a12
│   └── a2
│       └── a21
├── B
└── C
    └── b1
        └── b11

Tree Diff
Root
├── A
│   ├── a1
│   │   ├── a11 (removed)
│   │   └── a12
│   └── a2
│       └── a21 (added)
├── B
│   └── b1 (moved from)
│       └── b11
└── C (added)
    └── b1 (moved to)
        └── b11

Which now correctly shows that b11 has not moved with respective to its parent, (i.e. the node's parent has not been changed), but b11 still shows up twice. Having an option to only show the b11 node once would be great for visual clutter.

Hello! No worries about it, it is a relative new enhancement and there are a lot of ways to perform 'tree diff' so I was also taking these few days to think through the algorithm.
I also had another dilemma after implementing the "detail" - in the case where there is a clash in the names. It is captured in this test case (link) that if it is purely detail=True it will capture both new g as (moved to). Whereas in the same scenario (link) but with detail=True, aggregate=True, one of the g is considered moved but the other g is considered as added. Of course this is probably an edge case and given the implementation it makes sense what the code classify as "move" and what the code classify as "added/removed". Not sure if you have thoughts about this - happy to hear it!
And for your reply - will look into it further!

Thanks for adding more support to your tree diff @kayjan ! I'll try to live with the subtrees, (I need to keep only_diff to be True so I can see the nodes that haven't been added/removed/modified/moved.
You'll hate me even more for this, but would it also be possible to show nodes that have been moved and modified? I.e. let's say we add a a3 node in our example. a3 has some attribute that changes and it also has been moved. I.e. we would want to see (~) and (moved to) applied to it.
Example:
import pandas
import bigtree

# Create dataframe
# TODO need to show modified with moved
old_tree_df = pandas.DataFrame(
    data={
        "child": ["A", "B", "a1", "a2", "a3", "b1", "b11", "a11", "a12"],
        "parent": ["Root", "Root", "A", "A", "A", "B", "b1", "a1", "a1"],
        "age": [1, 2, 3, 4, 5, 6, 7, 8, 9],
    }
)

new_tree_df = pandas.DataFrame(
    data={
        "child": ["A", "B", "C", "a1", "a2", "a3", "b1", "a12", "a21", "b11"],
        "parent": ["Root", "Root", "Root", "A", "A", "C", "C", "a1", "a2", "b1"],
        "age": [1, 2, 10, 37, 4, 57, 6, 9, 11, 7],
    }
)

# Create tree from data
old_root = bigtree.dataframe_to_tree_by_relation(old_tree_df)
print("Old Tree")
bigtree.print_tree(old_root, all_attrs=True)
print("\n")

new_root = bigtree.dataframe_to_tree_by_relation(new_tree_df)
print("New Tree")
bigtree.print_tree(new_root, all_attrs=True)
print("\n")

print("Tree Diff")
tree_diff = bigtree.get_tree_diff(
    old_root, new_root, only_diff=False, aggregate=True, detail=True, attr_list=["age"]
)
tree_diff.show(attr_list=["age"])
print("\n")

def get_node_attribute(node: bigtree.Node):
    if node.node_name.endswith("(removed)"):
        return {"style": "solid", "color": "red"}
    elif node.node_name.endswith("(added)"):
        return {"style": "solid", "color": "green"}
    elif node.node_name.endswith("(~)"):
        node.name = str(node.node_name.rstrip(" (~)"))
        return {"style": "solid", "color": "orange"}
    return {}

def get_edge_attribute(node: bigtree.Node):
    if node.node_name.endswith(("(removed)", "(moved from)")):
        node.name = str(node.node_name.rstrip(" (moved from)").rstrip(" (removed)"))
        return {"style": "filled", "color": "red"}
    elif node.node_name.endswith(("(added)", "(moved to)")):
        node.name = node.node_name.rstrip(" (moved to)").rstrip(" (added)")
        return {"style": "filled", "color": "green"}
    return {}

graph = bigtree.tree_to_dot(
    tree_diff, node_attr=get_node_attribute, edge_attr=get_edge_attribute
)
graph.write_png("result.png")

Output:
Old Tree
Root
├── A [age=1]
│   ├── a1 [age=3]
│   │   ├── a11 [age=8]
│   │   └── a12 [age=9]
│   ├── a2 [age=4]
│   └── a3 [age=5]
└── B [age=2]
    └── b1 [age=6]
        └── b11 [age=7]

New Tree
Root
├── A [age=1]
│   ├── a1 [age=37]
│   │   └── a12 [age=9]
│   └── a2 [age=4]
│       └── a21 [age=11]
├── B [age=2]
└── C [age=10]
    ├── a3 [age=57]
    └── b1 [age=6]
        └── b11 [age=7]

Tree Diff
Root
├── A
│   ├── a1 (~) [age=(3.0, 37.0)]
│   │   ├── a11 (removed)
│   │   └── a12
│   ├── a2
│   │   └── a21 (added)
│   └── a3 (moved from)
├── B
│   └── b1 (moved from)
│       └── b11
└── C (added)
    ├── a3 (moved to)
    └── b1 (moved to)
        └── b11

I would like to be able to highlight the a3 node orange as I've done with the a1 node.

By the way, nutree might be a good reference here. Take a look at their diff example

We can see:

added nodes
removed nodes
moved nodes
modified nodes
and nodes that have been moved and modified

all in a nice visual without duplicate nodes being displayed
I feel like at this point I'm essentially asking you to redo your tree diff, so please free feel to ignore me. I don't want you to feel obligated at all.

Hahaha okay let me check out the tree diff in nutree - for now moved and modified are separate concepts e.g those nodes that are not moved are checked for modification.

@acxz hello, after some consideration, I have added a new method get_tree_diff_dataframe that retrieves the tree differences in a dataframe - this allows for more customised processing and handling. You can post-process and create the tree diff "tree" from the tree diff "dataframe" 😄


os: fork+exec incompatible with 'write error returned by close'

[Split off from discussion in #22220, which is about a related ETXTBSY problem.]
The conventional wisdom is that you must always be careful to check the result of close(2), because write(2) may accept data and then fail to write it, and the OS will only tell you in close(2). Usually there is mention of NFS at this point, but honestly I'm not sure how to demonstrate this behavior reliably in a real program. (If anyone knows, that would help.)
That is, let's assume this can happen:

T1: fd = open
T1: write(fd), success
T1: close(fd), error

In a multithreaded program that does fork+exec you have to worry about fds leaking into the programs started by exec. So we mark all our fds O_CLOEXEC. They still make it into the child process during fork, but they are closed automatically when the child does the +exec.
So this can happen:

T1: fd = open
T1: write(fd), success
3: T2: start fork+exec, forking P2 with its own copy of fd, called fd2 here
4: P2 does exec, which does close(fd2)
5: T1: close(fd)

Or this can happen:

T1: fd = open
T1: write(fd), success
3: T2: start fork+exec, forking P2 with its own copy of fd, called fd2 here
4: T1: close(fd)
5: P2 does exec, which does close(fd2)

In both cases, we end up with the same fd open in two processes, the parent that originally opened it and is going to fastidiously check the result of close, and the child that inherited it and is not going to check the result of close, because the close is implicit in the exec.
My question is this: if you have two fds pointing at the same underlying open file, and that open file has a pending write error, and then both fds get closed, which close operation reports the write error?
The first? The second? Neither of those is a good answer, because the parent, who we want to get the error, can be either the first or the second depending on the race with the child.
Both? That would be helpful in this case, although I think it would be surprising behavior.
The one that isn't a close-on-exec? That's not even possible, because if the parent close happens first then the kernel has no idea whether the child is going to close fd2 explicitly or during exec.
The first close that isn't a close-on-exec? That would work too, although I think it would also be surprising.
Looking at the Linux kernel I see no opportunity for differentiating the different closes. It looks like they will both do:
if (filp->f_op->flush)
	retval = filp->f_op->flush(filp, id);

I grepped all of Linux v4.12-rc6 (what I already had checked out) for \.flush *= and found that only fs/cifs, fs/ecryptfs, and handful of device drivers seem to have implementations. So it looks like, no, close really doesn't fail in practice, so this is moot.
If you ever did want to guarantee to see the error returned by close, though, I don't think Go can guarantee that - unless the kernel returns the error from all closes of an underlying open file.
/cc @ianlancetaylor @crawshaw @aclements

I guess I will close this.

Twitter points out to me that apparently POSIX defines that close can return EINTR, meaning the fd didn't actually get closed and it's up to the caller to repeat the call! But on Linux at least that's not what EINTR means, if it can happen at all. https://lwn.net/Articles/576478/
The comments on that article also suggest that AFS (in particular OpenAFS) is the file system that most often returned errors on close, which makes sense I suppose.

The Linux kernel NFS implementation does set .flush, and the implementation does return errors, so the standard reference to NFS seems accurate.
But I agree that it seems possible to lose those errors if there is a fork.
We could handle this in the syscall package, but is it worth it?

NFS is the best theoretical example, but if you want another you can construct a test around, there are a variety of ways to do this with a TCP socket.
The easiest way I can think of involves a fake network device interfering with packets. It may be possible to do something more portable with SO_LINGER.

We could handle this in the syscall package, but is it worth it?

If the flush implementation returns errors from only the first close, not also from subsequent closes, then even this single-threaded C program will miss the error:
fd = open(name, O_WRONLY|O_CLOEXEC)
if(write(fd) < 0)
	error()
if(fork() == 0) {
	exec()
	exit(1)
}
wait()
if(close(fd) < 0)
	error()

The implicit close during the child's exec will consume the error. Given that a single-threaded C program has the problem, I don't see what we can do in package syscall.

The alternative of course is that all the closes get an error, that fsync returns an error every time you call it, not just the first time. I got lost following the NFS code trying to figure out if that's true.

As far as I can see the NFS code does not save the close/flush value, but I am not certain.
We could fix this in the syscall package as follows: syscall.Close acquires a read lock on a rwmutex across the actual system call.  forkAndExecInChild acquires a write lock on the same rwmutex.  Thus Close and forkAndExecInChild can not occur simultaneously (and, unfortunately, only one fork can happen at a time).  In forkAndExecInChild the parent opens a pipe before the fork (on GNU/Linux this already happens if some SysProcAttr fields are set).  After the fork, the parent waits for a response on the pipe.  After the fork, the child walks through all the open file descriptors (we've cleverly stored the maximum file descriptor value from syscall.Open, syscall.Pipe, syscall.Socket, etc.).  For each file descriptor marked close-on-exec, the child closes it.  Any error from close is reported back to the parent on the pipe.  When all file descriptors are closed, the child notifies the parent on the pipe that this is complete.  The parent saves the errors, indexed by file descriptor; this is safe because the descriptors are, of course, open in the parent.  When the parent sees the notification on the pipe, it releases the lock, permitting syscall.Close calls to proceed.  When a syscall.Close closes a descriptor, it checks (and clears) the saved error.  If the close call succeeded but there is a saved error, syscall.Close returns the saved error.

syscall.Close acquires a read lock on a rwmutex across the actual system call

Things like syscall.Dup2 can also close FDs so they'd need blocking too.


AWS Construct Library Concepts section needs discussion of DEFINED vs IMPORTED resources

The discussion must also include the "IFoo" interfaces we use to represent them.

Moving guide issues to doc repo.


[Bug] Kyuubi will not connect to Spark Driver in new ApplicationMaster when the old Spark Driver is killed.

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Search before asking

[X] I have searched in the issues and found no similar issues.

Describe the bug
In the Spark on YARN cluster mode, when submitting a job with beeline, a Spark Driver will be created in ApplicationMaster hosted in NodeManager, and the Kyuubi will store the IP address in a ZNode in Zookeeper (or etcd).
According to the design of YARN, when an ApplicationMaster (Spark Driver) fails, the ApplicationMaster will be recreated for resiliency.
The issue is that Kyuubi is not aware of this new Spark Driver and the job will fail immediately.
A typical case of this issue is when rolling-restart your NodeManagers. The Job will fail immediately and retries will fail as well.
Procedures to Recreate this Issue.

Submit a Long-running Spark-SQL query through beeline (with Spark on YARN, cluster mode)
Go to YARN ResourceManager UI, and identify the NodeManager where the Spark Driver is running
Kill the Spark Driver. E.g. kill the process or just restart the Server.

A new Spark Driver will be created but Kyuubi fails to find it.
Expected: Kyuubi is aware of the new Spark Driver IP, and the HA information is updated accordingly.
Affects Version(s)
1.6.1
Kyuubi Server Log Output
No response
Kyuubi Engine Log Output
No response
Kyuubi Server Configurations
# kyuubi.authentication   KERBEROS
kyuubi.authentication LDAP
kyuubi.authentication.ldap.base.dn ou=users,dc=mycompany,dc=com
kyuubi.authentication.ldap.url ldaps://ldapserver.mycompany.com:636
kyuubi.authentication.ldap.guidKey uid

kyuubi.kinit.principal<EMAIL_ADDRESS>kyuubi.kinit.keytab /opt/kyuubi/conf/hive.keytab

kyuubi.frontend.thrift.binary.bind.port 10009

kyuubi.ha.enabled true
kyuubi.ha.addresses zookeeper1.mycompany.com,zookeeper2.mycompany.com,zookeeper3.mycompany.com
kyuubi.ha.zookeeper.client.port 2181
kyuubi.ha.zookeeper.session.timeout 600000

kyuubi.ha.namespace=kyuubi_zk_onpremise
kyuubi.session.engine.initialize.timeout=180000

Kyuubi Engine Configurations
spark.acls.enable=true
spark.submit.deployMode=cluster
spark.admin.acls=
spark.driver.extraClassPath=
spark.driver.extraLibraryPath=/opt/cloudera/parcels/CDH/lib/hadoop/lib/native
spark.driver.memory=8g
spark.dynamicAllocation.enabled=true
spark.dynamicAllocation.initialExecutors=0
spark.dynamicAllocation.maxExecutors=100
spark.dynamicAllocation.minExecutors=0
spark.eventLog.dir=viewfs:///log/spark3
spark.eventLog.enabled=true
spark.executor.cores=4
spark.executor.extraJavaOptions=-XX:+UseNUMA
spark.executor.extraLibraryPath=/opt/cloudera/parcels/CDH/lib/hadoop/lib/native
spark.executor.memory=20g
spark.extraListeners=
spark.hadoop.dfs.client.datanode-restart.timeout=30
spark.hadoop.yarn.timeline-service.enabled=false
spark.io.compression.lz4.blockSize=128kb
spark.master=yarn
spark.pyspark.python=/opt/spark-env/bin/python
spark.shuffle.file.buffer=1m
spark.shuffle.io.backLog=8192
spark.shuffle.io.serverThreads=128
spark.shuffle.service.enabled=true
spark.shuffle.service.name=spark3_2_shuffle
spark.shuffle.service.port=7338
spark.shuffle.unsafe.file.output.buffer=5m
spark.sql.autoBroadcastJoinThreshold=-1
spark.sql.adaptive.autoBroadcastJoinThreshold=100m
spark.sql.adaptive.advisoryPartitionSizeInBytes=256m
spark.sql.adaptive.coalescePartitions.minPartitionSize=16m
spark.sql.adaptive.enabled=true
spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes=512m
spark.sql.adaptive.coalescePartitions.parallelismFirst=false
spark.sql.cbo.enabled=true
spark.sql.hive.caseSensitiveInferenceMode=NEVER_INFER
spark.sql.hive.convertMetastoreOrc=true
spark.sql.hive.metastore.jars=builtin
spark.sql.orc.compression.codec=zlib
spark.sql.orc.filterPushdown=true
spark.sql.orc.impl=native
spark.sql.parquet.compression.codec=gzip
spark.sql.queryExecutionListeners=
spark.sql.sources.partitionOverwriteMode=dynamic
spark.sql.statistics.fallBackToHdfs=true
spark.sql.streaming.streamingQueryListeners=
spark.sql.warehouse.dir=/apps/hive/warehouse
spark.ui.filters=org.apache.spark.deploy.yarn.YarnProxyRedirectFilter
spark.unsafe.sorter.spill.reader.buffer.size=1m
spark.yarn.dist.files=
spark.yarn.historyServer.address=sparkhistory.mycompany.com:18080
spark.yarn.historyServer.allowTracking=true
spark.yarn.am.extraLibraryPath=/opt/cloudera/parcels/CDH/lib/hadoop/lib/native
spark.extraListeners=com.hortonworks.spark.atlas.SparkAtlasEventTracker
spark.sql.queryExecutionListeners=com.hortonworks.spark.atlas.SparkAtlasEventTracker
spark.sql.streaming.streamingQueryListeners=com.hortonworks.spark.atlas.SparkAtlasStreamingQueryEventTracker
spark.files=/opt/spark/conf/atlas-application.properties

Additional context
I am new to Scala, but I am willing to contribute.
Are you willing to submit PR?

[X] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
[ ] No. I cannot submit a PR at this time.

Hi, thanks for raising this issue, ZK server will hold the engine node for a while when the client(it's spark driver in this case) connection is lost, I think https://github.com/apache/kyuubi/pull/3618 may be another solution for your case.

Hi, @pan3793, Thank you very much for your response.
#3618 looks very promising and I will test it soon!

Closing this issue for now.
After I checked the source code, I found it's a bit complicated to implement the feature in an idempotent way, especially when the Spark Driver is killed.
It would be too complicated to implement the logic from the Kyuubi side.

this bug does not fix by #3618 ,this issuse should get more care.

i use kyuubi-1.7 this bug still exists, and kyuubi-1.7 had merge the #3618


[google_maps_flutter] Relax Flutter version requirement to 0.11.9

This is the latest version currently available for pub health checks.
We should increase the requirement once pub updates its Flutter version
(to make sure we're running with the recent Skia texture fixes).

LGTM


Master branch with Swift 4 doesn't support Pod integration in ObjC project

ActiveLabel havn't handle*Tap methods after pod integration in objective c project.
 The error says that the handleXXXTap methods have no visible interface. I'm using Swift 4 for the project and the pod.


Save the best epoch in ModelCheckpoint

🚀 Feature
Add best_model_epoch (best_model_step) to the ModelCheckpoint callback.
Motivation
After training, it would be good to know during which epoch/step a model reached its min/max in the monitor metric.
Right now, it can be figured out by from the best_model_path by using a regex, which is rather inconvenient.
ckpt = ModelCheckpoint(...)
best_epoch = int(re.search(r'(?<=epoch\=)(\d+)(?=.ckpt)', ckpt.best_model_path).group(0))

Pitch
Add two additional attributes best_model_epoch and best_model_step  which are updated in _update_best_and_save alongside the existing best_xxx attributes.
Alternatives
Additional context

Our suggestion here is that you load the checkpoint and extract the epoch value
ckpt = torch.load("my_model_checkpoint.ckpt")
epoch = ckpt["epoch"]


Restructure the directory for proxy

Signed-off-by: Minkyu Park<EMAIL_ADDRESS>Problem
As discussed from #1420 this PR is to change the directory structure for proxy. This creates a proxy directory at top level and moves both backend and client proxy under it, naming backend and client respectively.
Solution
This creates a proxy directory at top level and moves both backend and client proxy under it, naming backend and client respectively.

Note: All schema changes require discussion. Please link the issue for context.

[ ] Your change modifies the core OpenLineage model
[ ] Your change modifies one or more OpenLineage facets

If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports S3 and GCS filesystem operations, tested with AWS EMR).
Checklist

[ ] You've signed-off your work
[ ] Your pull request title follows our guidelines
[ ] Your changes are accompanied by tests (if relevant)
[ ] Your change contains a small diff and is self-contained
[ ] You've updated any relevant documentation (if relevant)
[ ] You've updated the CHANGELOG.md with details about your change under the "Unreleased" section (if relevant, depending on the change, this may not be necessary)
[ ] You've versioned the core OpenLineage model or facets according to SchemaVer (if relevant)
[ ] You've added a header to source files (if relevant)

SPDX-License-Identifier: Apache-2.0
Copyright 2018-2022 contributors to the OpenLineage project

Codecov Report

Merging #1439 (05a4054) into main (3c6ece1) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##               main    #1439   +/-   ##
=========================================
  Coverage     79.61%   79.61%
  Complexity       74       74
=========================================
  Files            87       87
  Lines          4061     4061
  Branches         25       25
=========================================
  Hits           3233     3233
  Misses          794      794
  Partials         34       34

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

Looks great! But, you'll also want to update the proxy path for the backend in new-version.sh.

Thank you! Updated the PR to include that.


Added --emulate-network option to CLI options

I feel it is better to do it same as emulateDevice using env variables to have consistency


current main branch fails make -j distcheck due to missing dap4_test subdirectories in the autotools build files

When I build like:
ed@koko:~/netcdf-c$ export LD_LIBRARY_PATH=/usr/local/hdf5-1.12.2/lib
ed@koko:~/netcdf-c$ export CPPFLAGS="-I/usr/local/hdf5-1.12.2/include -I/usr/local/szip-2.1.1/include"
ed@koko:~/netcdf-c$ export LDFLAGS="-L/usr/local/hdf5-1.12.2/lib -L/usr/local/szip-2.1.1/lib"
ed@koko:~/netcdf-c$ export DISTCHECK_CONFIGURE_FLAGS=--disable-dap-remote-tests
ed@koko:~/netcdf-c$ ./configure --disable-dap-remote-tests

Then make -j check works. (Yea!)
But make -j distcheck does not work:
make[5]: Entering directory '/home/ed/netcdf-c/netcdf-c-4.9.1-development/_build/sub/dap4_test'
FAIL: test_meta.sh
FAIL: test_parse.sh
FAIL: test_raw.sh
FAIL: test_fillmismatch.sh
FAIL: test_data.sh

Note that I set DISTCHECK_CONFIGURE_FLAGS to --disable-dap-remote-tests, so should these tests even be running?

I also fails make distcheck, so this is not a parallel build/test issue.
When I look in the log for the failed tests I see:
../../../dap4_test/test_parse.sh: 27: cd: can't cd to /home/ed/netcdf-c/netcdf-c-4.9.1-development/_build/sub/dap4_test/baselinethredds
FAIL test_parse.sh (exit status: 2)

The problem here is the dap4_test/baseline (and perhaps other directories under dap4_test) does not have a Makefile.am.
If you just want to distribute the files in this directory for testing, we need a Makefile.am which lists all the files as EXTRA_DIST.
I will submit a PR and see if that fixes my distcheck...

...BTW it seems like this problem would also be present in the tarball release. That is, the tarball release, if constructed with make dist, will only contain the files listed in the Makefile.am files.
In fact, I see that make dist also breaks:
make[4]: Entering directory '/home/ed/netcdf-c/dap4_test'
 (cd baseline && make  top_distdir=../../netcdf-c-4.9.1-development distdir=../../netcdf-c-4.9.1-development/dap4_test/baseline \
     am__remove_distdir=: am__skip_length_check=: am__skip_mode_fix=: distdir)
make[5]: Entering directory '/home/ed/netcdf-c/dap4_test/baseline'
make[5]: *** No rule to make target 'distdir'.  Stop.
make[5]: Leaving directory '/home/ed/netcdf-c/dap4_test/baseline'
make[4]: *** [Makefile:1129: distdir-am] Error 1
make[4]: Leaving directory '/home/ed/netcdf-c/dap4_test'
make[3]: *** [Makefile:1125: distdir] Error 2
make[3]: Leaving directory '/home/ed/netcdf-c/dap4_test'
make[2]: *** [Makefile:873: distdir-am] Error 1
make[2]: Leaving directory '/home/ed/netcdf-c'
make[1]: *** [Makefile:867: distdir] Error 2
make[1]: Leaving directory '/home/ed/netcdf-c'
make: *** [Makefile:968: dist] Error 2
ed@koko:~/netcdf-c$

@DennisHeimbigner unfortunately we cannot list subdirectories in EXTRA_DIST, only files. ;-)

I have a fix to this but I have to turn off nczarr_test/run_scalar.sh for it to work. See #2545

I am doing a distcheck right now, and it looks like I can
put directories in EXTRA_DIST.
There was, however, an error in d4test_common.sh that specified
an incorrect path to one of those directories.

This path issue is fixed in the 4.9.1 wellspring branch and will be back propagated to main in due time. I'll take a look at doing that early, but no changes to fix it are needed now. I'll be back in town next Tuesday and will bring this fix to Main if nothing else.

Putting globs or directories in an Autotools file list is discouraged, but putting files in a subdirectory is perfectly allowed, and sometimes encouraged.

I am surprised; Autotools seems to handle directories with no problems.
What is the argument against them?

For wildcards/globs, there's this section of the documentation explaining the reasons; requesting tests in PRs and running tests in CI seem to cover most of their concerns.  The one exception is lack of portability to non-GNU make, which may not be a concern here.
I wasn't aware that Autotools accepted directories in EXTRA_DIST, but that gets described just before this bit of the documentation.  The Autotools developers also recommend against using directories in EXTRA_DIST due to the likelihood of picking up extraneous files, unless you add something to dist-hook to take the most likely extraneous files back out. If the release archive is created on CI rather than an active developer's computer, there are fewer places for extra files to creep into the directory.
In short: it looks like I failed to consider the Autotools advice in the context of CI.


'gax-google-pubsub-v1' and 'gax-google-logging-v2' wreck the 'google.protobuf' namespace

/cc @bjwatson, @tbetbetbe, @rok987
The failure only shows up when things are installed via easy_install (vs. pip).  The two packages each include and empty google/protobuf/__init__.py, without any namespace package declaration for it (or any purpose).  When installed in a particular order via easy_install / setup.py develop, the bogus google.protobuf package ends up higher on sys.path than the real google.protobuf, and mayhem ensues.
The fix would be to just drop them from the sdists altogether.

@geigerj What do you think? This seems easy to do as part of rolling new releases of these libraries for https://github.com/GoogleCloudPlatform/gcloud-python/issues/1979.

Yes, I will do this when I release the new libraries. I will also fix https://github.com/googleapis/toolkit/issues/239 next, which will stop us from generating these bogus packages.

@geigerj that would be great, thanks!

I've put up new versions of those packages without the google.protobuf package.

@geigerj Thank you!  I just verified that bin/easy_install gcloud-python in a fresh virtualenv pulls in gax-google-pubsub-v1 0.17.12 and gax-google-logging-v2 0.17.12, and that they do not break imports from google.protobuf.*.


[Sema] Remove unneeded uses of TVO_CanBindToInOut

In #15574, @CodaFi pointed out that @slavapestov added TVO_CanBindToInOut to the callers of createTypeVariable out of an "abundance of caution". This pull request removes TVO_CanBindToInOut as much as possible, without breaking the validation test suite. If this pull request is broken, then we probably need more tests to be added to the test suite.

@swift-ci please test

@swift-ci please test source compatibility

My only concern here is whether diagnostics regress as a result of our now failing to produce a solution for a constraint system as opposed to producing a solution and then emitting a diagnostic after.
I suspect we do not have good test coverage for those cases. Can you take a look at that, and add some tests that exercise diagnosing & in invalid positions where before your change we would have produced a solution?

I think that is potentially fixable by Fixes (that's probably the only viable way to go).

I also think it would be very helpful for future us if this would be documented somehow...

Hi @rudkx – I'm not clairvoyant, so if you know of new tests that need to be written, can you please elaborate? Thanks.

The tests here are particularly sensitive to changes in inout bindings.  I'm pleasantly surprised they weren't tripped by these changes.

Hi @codafi – Well… That's what happens when runs ninja swift-check-validation for each use of TVO_CanBindToInOut or lack thereof. :-)

How does one reproduce source compatibility failures? Specifically: ReactiveCocoa

@davezarzycki Looks like it's only ReactiveCocoa and it was caused by redeclaration fixes so consider it green for this case.

Super. Out of respect to the tentative "make reviewing mandatory" policy discussion, can somebody please bless this pull request with approval? Thanks!

I didn't have a specific test case in mind. Just a general concern that when I attempted to remove the flag in getTypeForPattern we produced worse diagnostics. Perhaps we're not going to hit that in these other cases because they involve a RHS with & somewhere, which will produce an InOutExpr with the flag set on the type variable.

@davezarzycki @rudkx I'm in favor of this change. If any diagnostics are discovered to have regressed later, we should fix how we diagnose inout. @xedin was looking at that. It's broken right now anyway.


Missing documentation link: Video reference in Updating HubPress

In the Updating HubPress README (https://github.com/HubPress/hubpress.io#updating-hubpress) it states, "To learn how to do this correctly (there’s a trick to it the first time you pull changes from upstream), you can watch the following video to learn the correct process." But there is no video embedded or video link.
p.s. I'm really psyched about HubPress!

Github do not deal well with the macro video.
Here is the link https://www.youtube.com/watch?v=KCylB780zSM

Maybe we should use the jekyll-asciidoc-quickstart to turn the readme into
a proper Asciidoc User Guide for HubPress. We could get stuff like a TOC
and proper video embeds. WDYT @Anthonny?
On Mon, Mar 23, 2015, 01:22 Anthonny Quérouil<EMAIL_ADDRESS>wrote:

Github do not deal well with the macro video.
Here is the link https://www.youtube.com/watch?v=KCylB780zSM
—
Reply to this email directly or view it on GitHub
https://github.com/HubPress/hubpress.io/issues/149#issuecomment-84630124
.

@anthonny - Thank you for the link!
@jaredmorgs - I know you weren't asking me, but projects that eat their own dogfood always get an extra cool point from me.  Not to mention, the subsequent forks and updated forks will have one more working example for reference purposes. :+1:

Original issue fixed.  Left open for possible replacement with proper Asciidoc version.

学习了

The one thing I don't want to do is mess up the workflow the other
community translators have with the existing README.
I think you can close this issue MetaSean, and I'll create another issue
specifically related to converting the README to a formalized User Guide.
On Mon, Mar 23, 2015, 08:57 MetaSean<EMAIL_ADDRESS>wrote:

Original issue fixed. Left open for possible replacement with proper
Asciidoc version.
—
Reply to this email directly or view it on GitHub
https://github.com/HubPress/hubpress.io/issues/149#issuecomment-84728840
.

@jaredmorgs jekyll-asciidoc-quickstart could be a relevant choice

#153 created to track the User Guide discussion.
On Tue, Mar 24, 2015, 20:27 Anthonny Quérouil<EMAIL_ADDRESS>wrote:

@jaredmorgs https://github.com/jaredmorgs jekyll-asciidoc-quickstart
could be a relevant choice
—
Reply to this email directly or view it on GitHub
https://github.com/HubPress/hubpress.io/issues/149#issuecomment-85442038
.

标题 不支持中文？

@eurusd  - I think you want issue #43

Since @jaredmorgs opened issue #153  :+1: , I'm closing this issue.

Thank you @metasean :+1:

@anthonny - No, thank you for starting HubPress!  I just wish I had more time, knowledge, or money so that I could do more to help move it forward.


Two problem :

Two problem :
1.I could not compile because in SplitflapTokens.swift :
Cannot invoke 'map' with an argument list of type '((CVarArgType) -> String)'
I use Xcode 7.0.1 , is that the reason I occur the problem?

after I delete the 'wrong' code, I run the demo ,but the animation is strange ：
sometime the low part was dismiss, why?

Thanks!

Hi,

It works perfectly with Xcode 7.1 and over, I can't test on my computer so I don't really know. Following  this link http://stackoverflow.com/questions/29953390/swift-compiler-error-cannot-invoke-map-with-an-argument-list-of-type you should add Int -> String in  inside the map block and see whether it works.

May be it is a bug. Sometimes when I update the texts too fast it can appears. I'll investigate on this.

Thank you


[QUERY] Creating a Blob container with an ImmutabilityPolicy

Query/Question
How do I create a Blob container with an ImmutabilityPolicy using the Azure Java SDK? There is no documentation regarding Immutable storage with the Java SDK.
Why is this not a Bug or a feature Request?
Some classes are there related to immutable storage, but there is no documentation.
Setup (please complete the following information if applicable):

OS: Windows 10
IDE : IntelliJ IDEA
azure-mgmt-storage 1.27.2

@xccc-msft  could you provide some guideline here? Thanks.

@alberto-alvarado-m As per the REST APIs provided by storage resource provider side, there are only BlobContainer_Create and ImmutabilityPolicy_CreateOrUpdate.
For your query, you will need to separate into below two steps when creating a blob container with immutability policy.

First, create blob container

storageManager.blobContainers().defineContainer(“containerName”)
                                <more required/desired configuration>
                                .create();

Second, create immutability policy and associate it with existing blob container created before

storageManager.blobContainers().defineImmutabilityPolicy(“policyName”)
                                .withExistingContainer(“resourceGroupName”, “accountName”, “containerName”)
                                <more required/desired configuration>
                                .create()

The withExistingContainer will help operate the association. Thanks.

Thanks, that's what I was looking for.

@alberto-alvarado-m As per the REST APIs provided by storage resource provider side, there are only BlobContainer_Create and ImmutabilityPolicy_CreateOrUpdate.
For your query, you will need to separate into below two steps when creating a blob container with immutability policy.

First, create blob container

storageManager.blobContainers().defineContainer(“containerName”)
                                <more required/desired configuration>
                                .create();

Second, create immutability policy and associate it with existing blob container created before

storageManager.blobContainers().defineImmutabilityPolicy(“policyName”)
                                .withExistingContainer(“resourceGroupName”, “accountName”, “containerName”)
                                <more required/desired configuration>
                                .create()

The withExistingContainer will help operate the association. Thanks.

Thanks, I did manage to be able to create a Blob container and its immutability policy with the examples you've provided.
Is it possible to lock the immutability policy and retrieve the immutability policy from a container with the Azure Java SDK?