pip install grpcio on Python 3.13 Alpine is unable to find precompiled version

What version of gRPC and what language are you using?
3.13 - From the python:3-alpine image
What operating system (Linux, Windows,...) and version?
Mac osx Sequoia 15.0 - M3 Pro - this may be a CPU Architecture problem - not tested in x86
What runtime / compiler are you using (e.g. python version or version of gcc)
None
What did you do?
docker run -ti python:3-alpine sh
pip install grpcio

What did you expect to see?
Install from the precompiled version - without the need for a compiler to be installed
What did you see instead?
❯ docker run -ti python:3-alpine sh
Unable to find image 'python:3-alpine' locally
3-alpine: Pulling from library/python
Digest: sha256:c38ead8bcf521573dad837d7ecfdebbc87792202e89953ba8b2b83a9c5a520b6
Status: Downloaded newer image for python:3-alpine
/ # pip install grpcio
Collecting grpcio
  Downloading grpcio-1.67.0.tar.gz (12.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.6/12.6 MB 8.5 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [36 lines of output]
      Traceback (most recent call last):
        File "/usr/local/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
          ~~~~^^
        File "/usr/local/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ~~~~^^^^^^^^^^^^^^^^^^^^^^^^
        File "/usr/local/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/tmp/pip-build-env-blm7dfdg/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 332, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=[])
                 ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-blm7dfdg/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 302, in _get_build_requires
          self.run_setup()
          ~~~~~~~~~~~~~~^^
        File "/tmp/pip-build-env-blm7dfdg/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 516, in run_setup
          super().run_setup(setup_script=setup_script)
          ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-blm7dfdg/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 318, in run_setup
          exec(code, locals())
          ~~~~^^^^^^^^^^^^^^^^
        File "<string>", line 274, in <module>
        File "<string>", line 218, in check_linker_need_libatomic
        File "/usr/local/lib/python3.13/subprocess.py", line 1036, in __init__
          self._execute_child(args, executable, preexec_fn, close_fds,
          ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                              pass_fds, cwd, env,
                              ^^^^^^^^^^^^^^^^^^^
          ...<5 lines>...
                              gid, gids, uid, umask,
                              ^^^^^^^^^^^^^^^^^^^^^^
                              start_new_session, process_group)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/usr/local/lib/python3.13/subprocess.py", line 1966, in _execute_child
          raise child_exception_type(errno_num, err_msg, err_filename)
      FileNotFoundError: [Errno 2] No such file or directory: 'c++'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
/ #

Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).
Anything else we should know about your project / environment?
I am on an Apple M3 Pro chip

yea - I just tested on an x86 and it works as expected. This is most certainly a CPU architecture issue.
root@homelab:~# docker run -ti python:3-alpine sh
Unable to find image 'python:3-alpine' locally
3-alpine: Pulling from library/python
43c4264eed91: Already exists
99a612635ae6: Pull complete
ea644e5aaa5c: Pull complete
ae04b12910c8: Pull complete
Digest: sha256:c38ead8bcf521573dad837d7ecfdebbc87792202e89953ba8b2b83a9c5a520b6
Status: Downloaded newer image for python:3-alpine
/ # pip install grpcio
Collecting grpcio
  Downloading grpcio-1.67.0-cp313-cp313-musllinux_1_1_x86_64.whl.metadata (3.9 kB)
Downloading grpcio-1.67.0-cp313-cp313-musllinux_1_1_x86_64.whl (6.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.1/6.1 MB 11.9 MB/s eta 0:00:00
Installing collected packages: grpcio
Successfully installed grpcio-1.67.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
/ #

Hi @jensenbox it does look like a CPU architecture issue as you described. I haven't tried on a Mac, but wasn't able to reproduce it on a Debian Linux x86_64 machine. While we further check the issue, can you try manually downloading the grpcio-1.67.0-cp313-cp313-musllinux_1_1_x86_64.whl file and installing it in the docker container.
wget https://files.pythonhosted.org/packages/9e/a8/96b3ef565791d7282c300c07c2a7080471311e7d5a239db15678aaac47eb/grpcio-1.67.0-cp313-cp313-musllinux_1_1_x86_64.whl
pip install grpcio-1.67.0-cp313-cp313-musllinux_1_1_x86_64.whl

@jensenbox  we need some dependency. use this
docker run -ti python:3-alpine sh
apk update
apk add build-base linux-headers protoc
pip install grpcio

@jensenbox we need some dependency. use this
docker run -ti python:3-alpine sh
apk update
apk add build-base linux-headers protoc
pip install grpcio

I can confirm that this worked perfectly - no compilation and a speedy install. Thank you!
Important
I was able to simplify the instructions to simply issue an apk update rather than having to install the other APKs:
christian@christian-desktop:/projects/core$ docker run -ti --rm --env PIP_ROOT_USER_ACTION=ignore python:3-alpine sh
Unable to find image 'python:3-alpine' locally
3-alpine: Pulling from library/python
43c4264eed91: Pull complete
99a612635ae6: Pull complete
ea644e5aaa5c: Pull complete
ae04b12910c8: Pull complete
Digest: sha256:c38ead8bcf521573dad837d7ecfdebbc87792202e89953ba8b2b83a9c5a520b6
Status: Downloaded newer image for python:3-alpine
/ # apk update
fetch https://dl-cdn.alpinelinux.org/alpine/v3.20/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.20/community/x86_64/APKINDEX.tar.gz
v3.20.3-226-ge6f221129e2 [https://dl-cdn.alpinelinux.org/alpine/v3.20/main]
v3.20.3-225-g23f01f9dae0 [https://dl-cdn.alpinelinux.org/alpine/v3.20/community]
OK: 24166 distinct packages available
/ # pip install --upgrade pip
Requirement already satisfied: pip in /usr/local/lib/python3.13/site-packages (24.2)
Collecting pip
  Downloading pip-24.3.1-py3-none-any.whl.metadata (3.7 kB)
Downloading pip-24.3.1-py3-none-any.whl (1.8 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 40.7 MB/s eta 0:00:00
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 24.2
    Uninstalling pip-24.2:
      Successfully uninstalled pip-24.2
Successfully installed pip-24.3.1
/ # pip install grpcio
Collecting grpcio
  Downloading grpcio-1.67.0-cp313-cp313-musllinux_1_1_x86_64.whl.metadata (3.9 kB)
Downloading grpcio-1.67.0-cp313-cp313-musllinux_1_1_x86_64.whl (6.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.1/6.1 MB 75.3 MB/s eta 0:00:00
Installing collected packages: grpcio
Successfully installed grpcio-1.67.0
/ #

So effectively the process is now:
docker run -ti --rm --env PIP_ROOT_USER_ACTION=ignore python:3-alpine sh
apk update
pip install grpcio


Allow mailaliases to be managed via hiera

Pull Request (PR) description
Allow mailaliases to be managed via hiera
This Pull Request (PR) fixes the following issues
n/a

I should have squashed the commits, didn't notice until I hit the merge button :(

I should have squashed the commits, didn't notice until I hit the merge button :(

sorry for the mess


Standardize informal treatment  and some grammar improvements

Plus, fix colorblindness translation.

[X] I have read freeCodeCamp's contribution guidelines.
[X] My pull request has a descriptive title (not a vague title like Update index.md)
[X] My pull request targets the master branch of freeCodeCamp.
[X] None of my changes are plagiarized from another source without proper attribution.
[X] My article does not contain shortened URLs or affiliate links.

If your pull request closes a GitHub issue, replace the XXXXX below with the issue number.
#XXXXX

Thanks!


[HOTFIX] fix 'checkbox' type for create iterpreter form

What is this PR for?
Not visible checkboxes in form to create new interpreter.
What type of PR is it?
Bug Fix
Screenshots (if appropriate)
before

after

Questions:

Does the licenses files need update? no
Is there breaking changes for older versions? no
Does this needs documentation? no

@tinkoff-dwh Thanks for the contribution.
Tested locally and works well. Merge if no more discussion.


tracker-miners: 3.5.3 -> 3.5.4

Description of changes

https://gitlab.gnome.org/GNOME/tracker-miners/-/compare/3.5.3...3.5.4
Includes improvements to the seccomp jail, and fixes for spurious SIGSYS positives, including the one encountered on my system https://github.com/the-furry-hubofeverything/hubble-systems/issues/14
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

It's running on my laptop, and I'm currently doing nixpkgs-review...

Result of nixpkgs-review run on x86_64-linux 1

27 packages built:

cinnamon.cinnamon-gsettings-overrides
cinnamon.nemo-fileroller
cinnamon.nemo-with-extensions
dropbox-cli
dropbox-cli.nautilusExtension
eiciel
eiciel.nautilusExtension
gnome-photos
gnome-photos.installedTests
gnome.file-roller
gnome.gnome-control-center
gnome.gnome-control-center.debug
gnome.gnome-terminal
gnome.nautilus
gnome.nautilus-python
gnome.nautilus-python.dev
gnome.nautilus-python.devdoc
gnome.nautilus-python.doc
gnome.nautilus.dev
gnome.nautilus.devdoc
gnomeExtensions.gtk4-desktop-icons-ng-ding
nautilus-open-any-terminal
nautilus-open-any-terminal.dist
pantheon.file-roller-contract
phosh
phosh-mobile-settings
tracker-miners

Bumped in
https://github.com/NixOS/nixpkgs/commit/e4271d0b7290dcd8707023194426eb4933e4b97a
On Wed, 22 Nov 2023, 09:37 Rodney Lorrimar, @.***>
wrote:

@.**** approved this pull request.
I have tested this PR and with v3.5.4 the fs miner seems to work as well
as it did before.
—
Reply to this email directly, view it on GitHub
https://github.com/NixOS/nixpkgs/pull/264715#pullrequestreview-1743828057,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAFMEY56NCHPM3CULF4OBILYFW2U5AVCNFSM6AAAAAA6YUBUASVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTONBTHAZDQMBVG4
.
You are receiving this because your review was requested.Message ID:
@.***>


Allow project directory paths with spaces and punctuation in the path

Many directories insist on having spaces = e.g. when Iain has shared files with me through OneDrive the obligatory have space such as "Sim, Iain - SegmentationFolders/..."

Hi Jose, I would put this as higher than low priority, I think. It is a pretty big limitation to the work flow - e.g. on Mac; an external drive with a space in the name counts as a space in the path.

OP's requested feature can cause issues with the underlying commandline interface and how different operating systems handle wild chars and escape the space character in the path. Due to the major changes that the requested feature requires to produce a bug free solution, the issue will be closed for now.


ROX-25320: initial conversion to OCI artifacts

Description
Based on main repo example: stackrox/stackrox#11616
Checklist

[ ] Investigated and inspected CI test results
[ ] Updated documentation accordingly

Automated testing

[ ] Added unit tests
[ ] Added integration tests
[ ] Added regression tests

If any of these don't apply, please comment below.
Testing Performed
TODO(replace-me)
Use this space to explain how you tested your PR, or, if you didn't test it, why you did not do so. (Valid reasons include "CI is sufficient" or "No testable changes")
In addition to reviewing your code, reviewers must also review your testing instructions, and make sure they are sufficient.
For more details, ref the Confluence page about this section.

The acs-enterprise-contract job in konflux seems to be failing but not for Collector (as far as I can tell from the json output) so I think we can ignore it, but it'd be good if someone from @stackrox/rhtap-maintainers verifies my assumption

The acs-enterprise-contract job in konflux seems to be failing but not for Collector (as far as I can tell from the json output) so I think we can ignore it, but it'd be good if someone from @stackrox/rhtap-maintainers verifies my assumption

yes, that is fine to ignore.


GH-3766: Update wasmi to 0.13.2

Closes #3766
Update wasmi to 0.13.2.
This wasmi version deprecates allocating MemoryInstance::get that always allocates and copies data back and forth between VM linear memory and the heap on the host.
With this commit host does not allocate data, but is deserializing data straight from a linear memory without allocating. Copies are made only when absolutely necessary.

do you have any comparative benchmarks, before & after?

As discussed initial results as measured with coremark benchmark with all the processed code has about 70% higher score. There are more stories in the backlog to create more detailed measurements of the execution.


Memoized values - am I doing this correctly?

I'm trying to create an optimised selector. The selector is shared amongst several components so it accepts a parameter from outside:
const defaultValue = {};

const useSingleValue = id => useStore(
  useCallback(state => state[id] || defaultValue, [id])
);

Is this the correct way to go about it?

thats quite clever, maybe too clever. ;-) but really i wouldn't worry about selectors. it's nothing at all, no expense, maybe we should even remove the entry in the github docs.

It's true that this is a really simple example. If the selector actually computed something complex, would this be the correct approach?

I have similar situation like a more complex implement, is it correct approach?
const { list, loading, actions } = useStore(
  state => {
    if (!byPlayer) {
      return {
        list: state.transactions,
        loading: state.loading,
        actions: state.actions
      }
    }
    return {
      list: state.transactionsByPlayer.data[byPlayer] || { total: 0 },
      loading: state.loading,
      actions: state.actions
    }
  },
  shadowEuqlityChecker
)

Basically, selectors should be lightweight as much as possible.
If you need heavy computation, please consider using useMemo.
const value = useStore(...);
const computedValue = useMemo(() => ..., [value]);

Closing this. Feel free to reopen for further discussion.

Seems like the docs were changed to encourage usage of useCallback? What changed?

@HarrisonB Are you referring to this note? https://github.com/react-spring/zustand#memoizing-selectors
Stable selectors (either with useCallback, or defining outside) will allow skipping unnecessary computation, which could improve performance.

@dai-shi, right, but then @drcmda is saying the opposite in this comment: https://github.com/react-spring/zustand/issues/106#issuecomment-617855715

Long story short, we discussed and changed our recommendation.
The main reason is we plan to use useMutableSource in v4, and it recommends stable selectors for optimization. So, we recommend it in v3 too, to prepare for v4.
v3 (as well as v2) optimizes with stable selectors too to some extent. It's trivial if the selectors are lightweight. Unnecessary computation is just about calling the selectors. If a selector is not referentially changed and the state is not changed, we skip calling the selector. It has nothing to do with re-renders. If a selector is very lightweight (ex. state => state.foo) skipping it is almost nothing. btw, useCallback is not zero cost, so in total we don't know which is better. The best is define the selector outside render, so we don't even need useCallback (if the selector doesn't depend on other values.)

Gotcha. Thanks for the thorough explanation!

Just thinking out loud here, I've been toying with the idea of proxies in environments that support it for simple state selectors, it'd be nice to possibly get it to work for nested selectors as well.  Maybe it's one way to prevent having to duplicate a set of selectors that mirrors the state object?
const memoizedSelectors = {};
const selector = new Proxy(memoizedSelectors, {
  get: function (target: any, prop: string) {
    // if the selector is not found, we create a new one
    // and cache the selector
    if (target[prop] === undefined) {
      target[prop] = (state: any) => state?.[prop];
    }
    return target[prop];
  },
});

// Then, instead of
// const berriesSelector = state => state.berries
// function Component() {
// // const berries = useStore(berriesSelector)
// const berries = useStore(selector.berries)
// ...
// }

we could use a WeakMap  to possibly allow for garbage collection but that seems like overkill since there really shouldn't be that many selectors.
However, if we're going to go down the proxies route, I would bet that we wouldn't even need selectors because
const { value } = selector

also triggers the proxy to be called and if we could register the component as a listener.  Then it gets close to what mobx is doing but without having to wrap everything in an observer.

Hi, nice try and thinking out loud.

const berries = useStore(selector.berries)

If I were to support getting just a property (non-nested), I'd use a custom hook.
const useStoreProp = (prop) => {
  const selector = useCallback((state) => state[prop], [prop])
  return useStore(selector)
}

const { value } = selector

Good point. It's almost what react-tracked does.

@dai-shi thanks that's a good idea, I might give the hook version a try next time -- we could always use something like a lodash or other just safe get to support a string version of the nested prop pretty easily so that should cover a lot of the cases.
Thanks the pointer to react-tracked I'll check it out.  It's hard to keep "track" of all of the nice state libraries you've got on the go!  Also thanks for sharing things like zustand and jotai as well 👍


Undocumented Breaking Changes in V2.0

@sksamuel  First and foremost -- thanks for the effort you put into this library....really appreciate the effort,
Migrating to version 2 -- I can't seem to get the syntax of the eviction listener to pass the swiggle red line check (SRLC) -- attempt multiple variations; however, seems the combination of  asCache and eviction lamba definition do mix well....simply passing the readme example produces the following

I only able to get reference to resolve explicitly specifying the generic <K, V> definition....and still breaks after the asCache extension function....any additional guidance to get this particular property to resolve correctly

After a little bit more digging -- guessing the signature now requires that we pass in a separate CoroutineScope for the eviction listener....am I under interpreting that correctly from this line?
https://github.com/sksamuel/aedile/blob/ef896798c1d81b2c8806120e7dd7b150c9dcae0e/aedile-core/src/main/kotlin/com/sksamuel/aedile/core/config.kt#L45

Thanks for flagging this up.
I missed updating those tests in 2.0.0.
Since the signatures clash from the builders in Caffeine and the extension functions, I've renamed them to withEvictionalListener and withRemovalListener, and those take the proper suspendable functions. They also no longer require a scope (you can provide one if desired).
I've released this as 2.0.1. Let me know if this all works for you.

Did you try withExpireAfter ?
    Caffeine.newBuilder()
        .maximumSize(2000)
        .initialCapacity(500)
        .withExpireAfter(loggerExpiry)
        .asCache<Int,UDate>()

@sksamuel Apologies for another ping, but is withExpireAfter a function in the current patched 2.0.1 version -- not seeing it in the code anywhere

Can you paste your builder code.

I have tried these two variation of the code (see below), searching this repo I wasn't able to find a code definition for withExpireAfter, hence my initial question...thanks again for the look.
private val loggerExpiry =
  object : Expiry<Int, UDate> {
    override fun expireAfterCreate(key: Int, value: UDate, currentTime: Long) =
      (value.asInstance - Clock.System.now()).inWholeNanoseconds

    override fun expireAfterUpdate(
      key: Int,
      value: UDate,
      currentTime: Long,
      currentDuration: Long,
    ) = currentDuration

    override fun expireAfterRead(
      key: Int?,
      value: UDate?,
      currentTime: Long,
      currentDuration: Long,
    ) = currentDuration
  }

private val loggerCache =
  Caffeine.newBuilder()
    .maximumSize(2000)
    .initialCapacity(500)
    .expireAfter(loggerExpiry)
    .asCache<Int, UDate>() _<-- Does Not Resolve_

private val loggerCache =
  Caffeine.newBuilder()
    .maximumSize(2000)
    .initialCapacity(500)
    .withExpireAfter(loggerExpiry) _<-- Does Not Resolve_
    .asCache<Int, UDate>()

Ok it was some issue with the variance on the type parameters. Can you update to 2.0.2 and try again ?

@sksamuel Will do -- will look out for the release -- thanks again for the prompt attention


Unexpected "CUDA error: the launch timed out and was terminated" or CUDA_ERROR_LAUNCH_TIMEOUT

Reporting a bug

[x] I have tried using the latest released version of Numba (most recent is
visible in the change log (https://github.com/numba/numba/blob/main/CHANGE_LOG).
[x] I have included a self contained code sample to reproduce the problem.
i.e. it's possible to run as 'python bug.py'.

I'm facing issues with CUDA kernel launch timeouts. The issue seems to be related to the tensor sizes passed to the CUDA kernel.
When attempting to minimize my reproduction example the issue disappeared, so here is my current minimal reproduction:
import math

import numba
import numba.cuda
import torch

@numba.njit
def numba_forward(xs, ys, freqs, sr: int):

    allpass_x1 = 0.0
    allpass_y1 = 0.0
    allpass_η = 0.0

    damping_decay = 3.0
    damping_brightness = 0.4
    damping_x1 = 0.0
    damping_x2 = 0.0
    damping_rho = 0.0

    for i in range(len(freqs)):
        freq = freqs[i]

        delay_length_frac = sr * (1 / freq) - 1

        # Compute allpass param
        delay_read_pos = i - delay_length_frac
        delay_read_idx = int(math.floor(delay_read_pos))

        alpha = 1 + delay_read_idx - delay_read_pos
        if alpha < 0.5:
            delay_read_idx = delay_read_idx + 1
            alpha += 1

        allpass_η = (1 - alpha) / (1 + alpha)

        if delay_read_idx >= 0:
            raw_delay_out = ys[delay_read_idx]
        else:
            raw_delay_out = 0.0

        # Allpass
        x = raw_delay_out

        y = allpass_x1 + allpass_η * (x - allpass_y1)
        allpass_x1 = x
        allpass_y1 = y

        allpass_out = y

        # Damping
        damping_rho = 0.001 ** (1 / (freq * damping_decay))

        x = allpass_out

        h0 = (1 + damping_brightness) * 0.5
        h1 = (1 - damping_brightness) * 0.25
        y = damping_rho * (h0 * damping_x1 + h1 * (x + damping_x2))
        damping_x2 = damping_x1
        damping_x1 = x

        damping_out = y

        if i < len(xs):
            x = xs[i]
        else:
            x = 0.0

        ys[i] = x + damping_out

@numba.cuda.jit
def numba_backward(xs, ys, freqs, n: int, sr: int, grad_ys, d_excitation):

    for j in range(n):
        xs[j] = 1.0
        if j > 0:
            xs[j - 1] = 0.0

        numba_forward(xs, ys, freqs, sr)

        for i in range(len(grad_ys)):
            d_excitation[j] += ys[i] * grad_ys[i]

device = torch.device("cuda:0")

use_working_values = False

if use_working_values:
    # This works
    k = 25
    n = 31670
else:
    # This doesn't
    k = 219
    n = 22126

xs = torch.ones((k,), device=device)
ys_tmp = torch.ones((n,), device=device)
freqs = torch.ones((n,), device=device)
sr = 22050
grad_ys = torch.ones((n,), device=device)
d_excitation = torch.zeros((k,), device=device)

print(xs.shape, ys_tmp.shape, freqs.shape, k, sr, grad_ys.shape, d_excitation.shape)

numba_backward[1, 1](  # type: ignore
    numba.cuda.as_cuda_array(xs),
    numba.cuda.as_cuda_array(ys_tmp),
    numba.cuda.as_cuda_array(freqs),
    k,
    sr,
    numba.cuda.as_cuda_array(grad_ys),
    numba.cuda.as_cuda_array(d_excitation),
)

# Usage of `d_excitation` is crucial to trigger the issue without using CUDA_LAUNCH_BLOCKING
print(d_excitation)

For the first pair of k/n values, the code works as expected. For the second pair the CUDA kernel launch seems to time out. The error can manifest itself in different tracebacks depending on whether CUDA_LAUNCH_BLOCKING is set.
Without CUDA_LAUNCH_BLOCKING the relevant traceback is:
  File "/tmp/debug_numba_launch_kernel_timeout.py", line 148, in <module>
    print(d_excitation)
  File "/home/me/.virtualenvs/ddsp_pytorch/lib/python3.10/site-packages/torch/_tensor.py", line 431, in __repr__
    return torch._tensor_str._str(self, tensor_contents=tensor_contents)
  File "/home/me/.virtualenvs/ddsp_pytorch/lib/python3.10/site-packages/torch/_tensor_str.py", line 664, in _str
    return _str_intern(self, tensor_contents=tensor_contents)
  File "/home/me/.virtualenvs/ddsp_pytorch/lib/python3.10/site-packages/torch/_tensor_str.py", line 595, in _str_intern
    tensor_str = _tensor_str(self, indent)
  File "/home/me/.virtualenvs/ddsp_pytorch/lib/python3.10/site-packages/torch/_tensor_str.py", line 347, in _tensor_str
    formatter = _Formatter(get_summarized_data(self) if summarize else self)
  File "/home/me/.virtualenvs/ddsp_pytorch/lib/python3.10/site-packages/torch/_tensor_str.py", line 138, in __init__
    tensor_view, torch.isfinite(tensor_view) & tensor_view.ne(0)
RuntimeError: CUDA error: the launch timed out and was terminated
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

With CUDA_LAUNCH_BLOCKING the relevant traceback becomes:
  File "/tmp/debug_numba_launch_kernel_timeout.py", line 112, in <module>
    numba_backward[1, 1](
  File "/home/me/.virtualenvs/ddsp_pytorch/lib/python3.10/site-packages/numba/cuda/dispatcher.py", line 539, in __call__
    return self.dispatcher.call(args, self.griddim, self.blockdim,
  File "/home/me/.virtualenvs/ddsp_pytorch/lib/python3.10/site-packages/numba/cuda/dispatcher.py", line 675, in call
    kernel.launch(args, griddim, blockdim, stream, sharedmem)
  File "/home/me/.virtualenvs/ddsp_pytorch/lib/python3.10/site-packages/numba/cuda/dispatcher.py", line 327, in launch
    driver.launch_kernel(cufunc.handle,
  File "/home/me/.virtualenvs/ddsp_pytorch/lib/python3.10/site-packages/numba/cuda/cudadrv/driver.py", line 2563, in launch_kernel
    driver.cuLaunchKernel(cufunc_handle,
  File "/home/me/.virtualenvs/ddsp_pytorch/lib/python3.10/site-packages/numba/cuda/cudadrv/driver.py", line 327, in safe_cuda_api_call
    self._check_ctypes_error(fname, retcode)
  File "/home/me/.virtualenvs/ddsp_pytorch/lib/python3.10/site-packages/numba/cuda/cudadrv/driver.py", line 395, in _check_ctypes_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [702] Call to cuLaunchKernel results in CUDA_ERROR_LAUNCH_TIMEOUT

Numba version: 0.58.1
CUDA version: 12.2
NVidia driver version: 535.129.03
GPU: NVIDIA GeForce GTX 980

Are you using the GPU for running a display as well? The watchdog timer may be enabled. For example on my system, if I run
$ python -c "from numba import cuda; cuda.detect()"

Found 1 CUDA devices
id 0         b'Quadro T1000'                              [SUPPORTED]
                      Compute Capability: 7.5
                           PCI Device ID: 0
                              PCI Bus ID: 1
                                    UUID: GPU-38dcb4d3-d143-f7c4-8548-d0aab805aba6
                                Watchdog: Enabled
             FP32/FP64 Performance Ratio: 32
Summary:
	1/1 devices are supported

Do you see that it is enabled on your setup?

Are you using the GPU for running a display as well?

Yes, this is just my main graphics card (NVIDIA GeForce GTX 980). I also noticed that when running with CUDA_LAUNCH_BLOCKING=1 my display doesn't really update any more while executing the reproduction code.

The watchdog timer may be enabled.

Yes, the snippet indicates that it is enabled.

OK - ideally you will need to use another GPU for the display, or disable X when running - this article outlines the situation and options in more detail: https://nvidia.custhelp.com/app/answers/detail/a_id/3029/~/using-cuda-and-x

Closing this as there does not appear to be an issue in Numba to resolve here.


feat(sql): preserve column order from query with indexmap

When using the select command in the sql plugin, the field order in each row isn't maintained due to the use of a hashmap. This update ensures that the field order remains consistent.
Example:
SELECT name, age FROM users;

Previously, the returned rows could have inconsistent order, such as
{"name": ..., "age": ...}

or
{"age": ..., "name": ...}

With this modification, the order will consistently follow the query, i.e., name followed by age.

Sure, added new changes file


Release 8.1.0

Hi, we got a release 8.1.0 PR https://github.com/detekt/sarif4k/pull/37, but in this repo there's no commit/tag/release/changelog/readme associated with it.

sorry - forgot to push the tag to the gradle plugin. its there now.

Did you also forget to push main?
https://github.com/dependency-check/dependency-check-gradle/tree/v8.1.0

Sorry about that. Fixed.  My normal routine for the release got messed up because the nexus server I publish to central through was having issues and timed out repeatedly...

No worries, only thing I wanted to see was what changed :)


fix: check to see if it exists before deleting

Signed-off-by: awwwd<EMAIL_ADDRESS>Fixes https://github.com/cello-proj/cello/issues/109

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Codecov Report

Merging #370 (4566905) into main (45f292b) will increase coverage by 0.16%.
The diff coverage is 100.00%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #370      +/-   ##
==========================================
+ Coverage   70.02%   70.18%   +0.16%
==========================================
  Files          11       11
  Lines        2042     2053      +11
==========================================
+ Hits         1430     1441      +11
  Misses        510      510
  Partials      102      102

Impacted Files
Coverage Δ

service/handlers.go
62.22% <100.00%> (+0.40%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 45f292b...4566905. Read the comment docs.


ws2811.h: No such file or directory on Raspberry Pi 4

Hi all, I followed the rpi_ws281x documentation, I compiled and installed the library (I have a libws2811.a in /usr/local/lib and I have a ws2811 folder with some .h files in /usr/local/include); I was abled to run the test succesfully too.
I downloaded the Go wrapper with:
go get github.com/rpi-ws281x/rpi-ws281x-go
I copied basic.go example in my project folder /home/pi/go/src/ws281x/main.go and then I've tried with:
go run main.go and go build but in both cases I get this error:
# github.com/jgarff/rpi_ws281x/golang/ws2811
In file included from ../../pkg/mod/github.com/jgarff/rpi_ws281x@v0.0.0-20201106090144-3a20f1bdd740/golang/ws2811/ws2811.go:38:
./ws2811.go.h:30:10: fatal error: ws2811.h: No such file or directory
 #include <ws2811.h>
          ^~~~~~~~~~
compilation terminated.

My Go version is:
go version go1.14.4 linux/arm
My Go env is:
GO111MODULE=""
GOARCH="arm"
GOBIN=""
GOCACHE="/home/pi/.cache/go-build"
GOENV="/home/pi/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="arm"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/pi/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_arm"
GCCGO="gccgo"
GOARM="6"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/pi/go/src/ws281x/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -marm -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build756845583=/tmp/go-build -gno-record-gcc-switches"

I would really like to be able to compile my code on this machine, I write and run my code on this specific hardware and I don't need cross-compile so I skipped the Docker compilation.
Is there something I missed? Could anyone please help me with this problem?

Hello. Can you please list all the files in your /usr/local/include ? (e.g. using find /usr/local/include).

Hello. Can you please list all the files in your /usr/local/include ? (e.g. using find /usr/local/include).

Hi and thank you for your reply.
Here is the output of find /usr/local/include:
/usr/local/include
/usr/local/include/ws2811
/usr/local/include/ws2811/rpihw.h
/usr/local/include/ws2811/pcm.h
/usr/local/include/ws2811/dma.h
/usr/local/include/ws2811/ws2811.h
/usr/local/include/ws2811/mailbox.h
/usr/local/include/ws2811/pwm.h
/usr/local/include/ws2811/gpio.h
/usr/local/include/ws2811/clk.h

Hi and thank you for your reply.
Here is the output of find /usr/local/include:
/usr/local/include
/usr/local/include/ws2811
/usr/local/include/ws2811/rpihw.h
/usr/local/include/ws2811/pcm.h
/usr/local/include/ws2811/dma.h
/usr/local/include/ws2811/ws2811.h
/usr/local/include/ws2811/mailbox.h
/usr/local/include/ws2811/pwm.h
/usr/local/include/ws2811/gpio.h
/usr/local/include/ws2811/clk.h

It looks good. What bothers me is this line :
# github.com/jgarff/rpi_ws281x/golang/ws2811

This is from an old version of the go-wrapper and this is no longer maintained.
The new wrapper is https://github.com/rpi-ws281x/rpi-ws281x-go
I guess that you used the basic.go from the  old wrapper (https://github.com/jgarff/rpi_ws281x/blob/master/golang/examples/basic.go) and this does no longer work. Perhaps we should ask the maintainer of the C library to remove this old wrapper.
Can you try with examples from the new wrapper (https://github.com/rpi-ws281x/rpi-ws281x-go/tree/master/examples) ?

It looks good. What bothers me is this line :
# github.com/jgarff/rpi_ws281x/golang/ws2811

This is from an old version of the go-wrapper and this is no longer maintained.
The new wrapper is https://github.com/rpi-ws281x/rpi-ws281x-go
I guess that you used the basic.go from the  old wrapper (https://github.com/jgarff/rpi_ws281x/blob/master/golang/examples/basic.go) and this does no longer work. Perhaps we should ask the maintainer of the C library to remove this old wrapper.
Can you try with examples from the new wrapper (https://github.com/rpi-ws281x/rpi-ws281x-go/tree/master/examples) ?

Now it works!
Oh god, I completely missed that examples folder.
Someone really should ask the maintainer of the C library to remove the old wrapper and maybe the language-specific examples; in this way anyone is forced to check the examples folder in the language-binding repo.

Now it works!
Oh god, I completely missed that examples folder.
Someone really should ask the maintainer of the C library to remove the old wrapper and maybe the language-specific examples; in this way anyone is forced to check the examples folder in the language-binding repo.


Error when selecting elements with inlines

https://user-images.githubusercontent.com/52297646/154511899-ebf5a817-8cf2-4793-b9dd-b721fa48231f.mp4

@ilya2204 thank you for your feedback.


Replace handshake and query command in sharding-scaling-mysql

For #4594 .

Pull Request Test Coverage Report for Build 10347

46 of 53 (86.79%)  changed or added relevant lines in 4 files are covered.
18 unchanged lines in 3 files lost coverage.
Overall coverage decreased (-0.1%) to 59.848%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

sharding-scaling/sharding-scaling-mysql/src/main/java/org/apache/shardingsphere/shardingscaling/mysql/binlog/MySQLNegotiateHandler.java
13
14
92.86%

sharding-scaling/sharding-scaling-mysql/src/main/java/org/apache/shardingsphere/shardingscaling/mysql/binlog/MySQLConnector.java
9
12
75.0%

shardingsphere-database-protocol/shardingsphere-database-protocol-mysql/src/main/java/org/apache/shardingsphere/database/protocol/mysql/packet/command/query/text/query/MySQLComQueryPacket.java
0
3
0.0%

Files with Coverage Reduction
New Missed Lines
%

sharding-orchestration/sharding-orchestration-core/sharding-orchestration-core-registrycenter/src/main/java/org/apache/shardingsphere/orchestration/core/registrycenter/util/IpUtils.java
3
76.0%

sharding-scaling/sharding-scaling-core/src/main/java/org/apache/shardingsphere/shardingscaling/core/synctask/history/HistoryDataSyncTask.java
7
76.47%

sharding-scaling/sharding-scaling-core/src/main/java/org/apache/shardingsphere/shardingscaling/core/execute/executor/channel/MemoryChannel.java
8
51.85%

Totals

Change from base Build 1105:
-0.1%

Covered Lines:
12531

Relevant Lines:
20938

💛  - Coveralls


demo files for OMA Standalone OrthoXML-converter are gone.

Whops, these should be in some place where people can see them right away. I'd still be in favor of having some demo-folder with some small example files to that end. Especially as "🐵  see, 🐵  do" is often the best way to learn how to run something 😂

@gedankenstuecke: they will be back :D

Ok, then I will not try to add them to avoid merge conflicts 😄
And let me know what you think of the reworked readme!

I put the files back in demo folder. Can you check if the oma_example.orthoxml and mapping file are correct?


"There are 0 connected RealSense devices" message

Required Info

Camera Model
ZR300

Firmware Version

Operating System & Version
Ubuntu 16.04

Kernel Version (Linux Only)
4.4.0-71-generic

Build System
Makefile

I get the message of 0 connected devices when I try to run the examples in /usr/local/bin such as "cpp-tutorial-1-depth". I did not have any issues when installing, however I don't see the camera in /dev/video* whenever I plug it in.
Has anyone faced this issue?
Thanks

I followed the installation instructions without major issues, I only had to update my kernel. After patching the uvcvideo kernel module I tried to run some example like "cpp-tutorial-1-depth", but I get the message saying 0 connected devices. When I plug the camera I cannot see it under /dev/video* . Has anyone faced this issue?

Could you please run dmesg -wT when the device is unplugged, plug it in and share the output?
This can determine if your Linux machine recognizes the connection of USB3 device or the problem is on a lower level.

The output of dmesg is
[Wed Apr  5 10:46:12 2017] usb 3-6: USB disconnect, device number 15
[Wed Apr  5 10:46:12 2017] usb 3-6.2: USB disconnect, device number 16
[Wed Apr  5 10:46:14 2017] usb 3-5: new high-speed USB device number 17 using xhci_hcd
[Wed Apr  5 10:46:14 2017] usb 3-5: New USB device found, idVendor=0451, idProduct=8027
[Wed Apr  5 10:46:14 2017] usb 3-5: New USB device strings: Mfr=0, Product=0, SerialNumber=1
[Wed Apr  5 10:46:14 2017] usb 3-5: SerialNumber:<PHONE_NUMBER>B7
[Wed Apr  5 10:46:14 2017] hub 3-5:1.0: USB hub found
[Wed Apr  5 10:46:14 2017] hub 3-5:1.0: 2 ports detected
[Wed Apr  5 10:46:14 2017] usb 3-5.2: new high-speed USB device number 18 using xhci_hcd
[Wed Apr  5 10:46:14 2017] usb 3-5.2: New USB device found, idVendor=04b4, idProduct=22c3
[Wed Apr  5 10:46:14 2017] usb 3-5.2: New USB device strings: Mfr=1, Product=6, SerialNumber=0
[Wed Apr  5 10:46:14 2017] usb 3-5.2: Product: CX3-WinUsb
[Wed Apr  5 10:46:14 2017] usb 3-5.2: Manufacturer: Cypress

It turned out it was my USB port, I went through all my USB ports, only one of them worked.

Hi,
I have the same eduard626 issue. I tried to change USB port but nothing changed. Could anyone help me?
Thanks.

Same issue too with the R200.
I got this message :
[Di Mär 20 14:02:45 2018] usb 4-1.2-port3: Cannot enable. Maybe the USB cable is bad?
when i plug the camera and dmesg -wT is running. Any idea on how to deal with it ?

R200 only supports USB3. Need to make sure if your USB connection is fine. Use "dmesg -wT" to see if it can be detected/recognized as a Super-Speed device. More of time, related to 1) the cable issue, 2) USB port or USB Hub capability.
Finding a USB3 qualified cable is a good approach.

Thanks it work now.
I changed my cable and used it on a different usb port (also didn't worked with the docker station).


🐛 Bug: package is listed as type: module even though it only exports .cjs files

Bug Report Checklist

[X] I have tried restarting my IDE and the issue persists.
[X] I have pulled the latest main branch of the repository.
[X] I have searched for related issues and found none that matched my issue.

Expected
// tsconfig.json
{
  "compilerOptions": {
    // ...
    "moduleResolution": "node16",
    // ...
  }
}

// package.json
{
  // ...
  "type": "commonjs"
  // ...
}

// file.ts
import * as foo from 'ts-api-utils';

Expected: this should work
Actual
The current file is a CommonJS module whose imports will produce 'require' calls; however, the referenced file is an ECMAScript module and cannot be imported with 'require'. Consider writing a dynamic 'import("ts-api-tools")' call instead.
  To convert this file to an ECMAScript module, change its file extension to '.mts' or create a local package.json file with `{ "type": "module" }`. ts(1479)

Additional Info
The root package.json is marked as type: module
https://github.com/JoshuaKGoldberg/ts-api-utils/blob/ba48dde317a7d88db22a1cba34bc820f92c5c9e3/package.json#L11
But the build only outputs one .cjs file<EMAIL_ADDRESS>It also lists an import path, but such a path is not included in the released package:
https://github.com/JoshuaKGoldberg/ts-api-utils/blob/ba48dde317a7d88db22a1cba34bc820f92c5c9e3/package.json#L15
I think the former is correct - but i guess that TS doesn't accept it? Is it a problem with this module or the consuming TS config?

Ah, this is a bug yeah. #37 should fix it. Thanks for reporting!

https://twitter.com/bradzacher/status/1628248371143274498
Okay doing some investigation, I've figured out how this package is supposed to be setup if it's got dual exports.
What type: "module" does is tells consumers "all ambiguous files covered by the package.json should be treated as .mjs files". TS mirrors this same behaviour locally developing .ts files, and externally with the .d.ts files.
Currently the package is configured with package.exports['./'].types = "./lib/index.d.ts"
By declaring type: "module", you're telling TS that ./lib/index.d.ts is ESM - i.e. it is supposed to be ./lib/index.d.mts -> meaning it is not allowed to be required from a CJS file.
This is why we're getting the TS error in @typescript-eslint. We're a CJS package, so TS is blocking us from requireing an ESM file - which is technically correct.
Sadly in this instance TS blocks hard here because (I believe) there are no types for the CJS export. It just sees ESM types, so it essentially ignores the CJS file (@andrewbranch is this the correct way to phrase it?)
So how can this be fixed?
(1) add a separate .d.cts def file for the commonjs export.
Ideally it should use export =, but ultimately just exporting a new file with exactly the same content, just with the correct extension should be "okay".
(2) fix the package.json config
(2.a) One option is to remove the exports.types field altogether and just add the relevant .d.ts/.d.cts files - TS is smart enough to find the correct files (it's what andrew recommends). But personally I like to be explicit, so I'd personally go with (b)...
(2.b) update the package.json config to:
	"type": "module",
	"exports": {
		".": {
			"types": {
				"import": "./lib/index.d.ts",
				"require": "./lib/index.d.cts"
			},
			"import": "./lib/index.js",
			"require": "./lib/index.cjs"
		}
	},

(TIL you can nest the condition within the types condition!)
This will explicitly tell TS where to find the types for each of exported files.
(3) [optional] name the files to be explicitly .mts.
You don't need to do this, but it's a nice thing to do to remove ambiguity in the resolution everywhere. Completely optional aesthetic step.

Ahh, I was having this same issue a little while back in my other libraries; I fixed it by implementing option 2b there.
I wasn't noticing this issue in this project so I assume somewhere between TS 4.7 and the current 4.9, they fixed this so that .d.ts files could be treated as either cjs or mjs based on the context of what file was responsible from them being loaded. I guess I was wrong and this issue still remains.

I tested 2.b by manually modifying the module within my @typescript-eslint repo and it worked as expected.
I changed the export { ... } to export = { ... }, and it meant I couldn't do import * as tools from 'ts-api-utils' any more (ofc), but other than that - worked fine, again.


Simplify build process for portable builds

The current process for building portable files involves flattening several files into a tar and transferring to another server via HTTP to kick off an external build. This setup works well for a small team with a permanent build setup, but would be confusing for an open source contributor. The build scripts should be consolidated and simplified so they can be run directly from the repo, respectively on a Linux, Mac, or Windows system. Also, update build instructions on how to build portable on the above 3 platforms.

Pavel is currently working this on my team.

I'll test this one and then close

Tested and closing


Importer can't download file from AWS

The user is using a server-side script to upload files to CartoDB using Import API.
Several of those imports fails when the importer tries to download the file using the AWS URL. That same URL could be used afterwards to download the same file.
Context
The problem reproduces with random files, which have been manually uploaded later without any problem.
This is the Import API request to check the status of a failed import:
ubuntu@****:~/user-tools/london-import$ curl -v 'https://username.cartodb.com/api/v1/imports//888adc31-827f-45cc-a2bb-b834923f52d5?api_key=****'

Response
{
	"id":"888adc31-827f-45cc-a2bb-b834923f52d5",
	"user_id":"****",
	"table_id":null,
	"data_type":"url",
	"table_name":null,
	"state":"failure",
	"error_code":1001,
	"queue_id":"888adc31-827f-45cc-a2bb-b834923f52d5",
	"tables_created_count":null,
	"synchronization_id":null,
	"type_guessing":true,
	"quoted_fields_guessing":true,
	"content_guessing":false,
	"create_visualization":false,
	"visualization_id":null,
	"user_defined_limits":"{\"twitter_credits_limit\":0}",
	"get_error_text":{"title":"Download error",
	"what_about":"The remote URL returned an error. Please verify your file is available at that URL.",
	"source":"user"},
	"display_name":"x23245593x.zip",
	"success":false,
	"original_url":"http://s3.amazonaws.com/com.cartodb.imports.production/c0535bc3d73df3394e18/x23245593x.zip?AWSAccessKeyId=****&Expires=1460135290&Signature=****",
	"warnings":null,
	"is_raster":false}

That same URL is used afterwards to download the file, with success:
ubuntu@****:~/user-tools/london-import$ wget -O /tmp/z.zip 'http://s3.amazonaws.com/com.cartodb.imports.production/c0535bc3d73df3394e18/x23245593x.zip?AWSAccessKeyId=****&Expires=1460135290&Signature=****'

HTTP request sent, awaiting response... 200 OK
Length: 2281322 (2.2M) []
Saving to: '/tmp/z.zip'

100%[==>] 2,281,322   2.11MB/s   in 1.0s

2016-04-08 15:16:19 (2.11 MB/s) - '/tmp/z.zip' saved [2281322/2281322]

ubuntu@ip-172-31-32-234:~/user-tools/london-import$ cd /tmp/z.zip
bash: cd: /tmp/z.zip: Not a directory
ubuntu@ip-172-31-32-234:~/user-tools/london-import$ cd /tmp
ubuntu@ip-172-31-32-234:/tmp$ unzip z.zip
Archive:  z.zip
  inflating: base/cme9bno0/breaks/x23245593x.dbf
  inflating: base/cme9bno0/breaks/x23245593x.prj
  inflating: base/cme9bno0/breaks/x23245593x.shp
  inflating: base/cme9bno0/breaks/x23245593x.shx
ubuntu@****:/tmp$

The error log for that import shows:
15:09:31 UTC: Errored importing data:
15:09:31 UTC: CartoDB::Importer2::DownloadError: DOWNLOAD ERROR: Code:20

Current result
The file is not downloaded from AWS, although it is available and could be downloaded as the logs show above. The same file(s) that randomly fail could be imported later with no problem.
Expected result
The importer should get the file from the AWS URL and continue the import process as usual

Logs are simplified and anonymised, please let me know if you need further info

Issue moved to CartoDB/cartodb-management #4435 via ZenHub.io


Fixes #706

Saves the continuous rendering state if one is called before mapView gets initialized.
On the mapView initialize callback sets the appropriate rendering mode if required.

@hjanetzek @blair1618 can you guys give this a quick look .. and I can cut a new release (call it 0.2.1-android) for our sdk.

LGTM. though I think #708 is not a change of the public API either. @blair1618 could you check?

Closing in favor of https://github.com/tangrams/tangram-es/pull/708, which looks good to me.


Misleading variable name description for DCM

Describe the bug
A variable for direction cosine matrix from Body frame to Platform frame should be defined as dcm_PB.
In coarseSunSensor.h, line 40:
    Eigen::Matrix3d     dcm_PB;                 //!< [-] DCM from platform frame P to body frame B
Here, either the variable name is incorrect or the comment next to it.
From the calculations in coarseSunSensor.cpp, it looks like the comment is incorrect.
There is no bug in the code, but the comment is misleading.

Mm, interesting.  I'll take a look at this.


Private timeline hides cert solutions

If you set the timeline to private and open one of your certifications, all the solutions will be hidden:

Related: https://github.com/freeCodeCamp/freeCodeCamp/issues/40403#issuecomment-779957353

I am closing this in favour of the other issue because the fix is most likely to be in the same area. Tom, please feel to carry over your report if the other issue does not cover this.

@raisedadead We decided to go a different route: https://github.com/freeCodeCamp/freeCodeCamp/pull/45562
But we have not actually discussed what that route is: https://github.com/freeCodeCamp/freeCodeCamp/issues/45563

Ok - lets' discuss this over a call sometime early next week.

In addition to the discussed "404" page or whatever we come up with, we should add this next to the timeline setting:


Print Keypad Combinations using Recursion

Have you read the Contributing Guidelines on Pull Requests?
YES!
Description
Add the code to print the keypad Combinations using Recursion in C++
Checklist

[x] I've read the contribution guidelines.
[x] I've checked the issue list before deciding what to submit.
[x] I've edited the README.md and link to my code.

Related Issues or Pull Requests
Solve Issue : #3545

I don't think that this program suits the domain of this repo. You can come up with some better algorithmic problem.


Memory pre-loading

The following issue came up during discussions related to the implementation of the number theoretic transform (NTT).
In many situations, it is required to have arrays of large sizes loaded into memory before the actual program starts running. This is the situation for example when doing NTT where a large array containing roots of unity is needed during the iterations of the algorithm. Clearly these constants should reside in memory as they are, most of the time, used several times during the execution of any useful program.
The question that poses itself then is how to do this in the most efficient+secure way. Some ideas that came up during the discussion:

One might augment the current mechanism used to load advice inputs in order to target the above situations in an efficient manner.
The memory trace can be pre-populated with the data by the prover before the start of the execution. This should be better, I believe, than the previous proposal but it is not immediately clear, to me at least, how we can insure soundness with this modification  while still using the methodology we currently use for insuring soundness i.e. without any additional permutation-checks machinery.

I am thinking these could be two separate thing. Basically, I think we'd need to support the first approach regardless of whether we support the second - so, might as well figure out what's the best way to do that.
First, to set the baseline. We could write constants to the memory using push operations. If we have 1K constants to write, this would take close to 3K VM cycles (1K cycles to push the value onto the stack, 1K cycles to pop them off the stack, and probably close to 1K cycles to save them to memory).
Theoretically, using the advice tape, we could speed this up significantly. The mechanism would work like this:

Hash the inputs outside of the VM.
Push the hash of the inputs onto the stack.
Then read the inputs from the advice tape (sing loadw.adv instruction) and hash them sequentially (using rpperm instruction) while saving the read inputs to memory.
Once all the inputs have been read and hashed, compare the computed hash with the hash pushed onto the stack in step 2. If the hashes are the same, we know that the memory now has the values we wanted to put there.

I've spent a little bit of time thinking about how this can be done with the current instructions, and unfortunately, practical efficiency is probably not going to be great. The main reason is that to absorb new elements into the hasher state we need to add them to the current state (rather than just replacing them). Just this step will require 8 additions to absorb 8 new elements. So, I wouldn't be surprised if we end up with needing more than 3K cycles with this approach.
But, we could add a specialized instruction to help with this task, and that could make things significantly faster. Here is one idea:
Imagine we have and instruction pipe_adv which does the following:

Reads 4 elements from the advice tape and adds them to the top 4 elements of the stack (element-wise).
Writes the read elements into the memory address specified by the 13th element of the stack.
Increments the 13th element of the stack.

The transition diagram for this instruction could look like so:
[C, B, A, addr, ...] -> [C + D, B, A, addr + 1, ...]
D -> mem[addr]

In the above:

C, B, and A are words (4 elements each) on the top of the stack.
D is the word read from the advice tape

Then, using this instruction, we could use the following sequence to absorb 8 elements into hasher state and to write them to memory at the same time:
swapw
pipe_adv
swapw
pipe_adv
rpperm

Thus, with 5 cycles we'd process 8 field elements, and the overall number of cycles needed to put 1K elements to memory should be slightly over 600 (or 5x improvement over the baseline).
We could go a step further and define pipe_adv as follows:
[C, B, A, addr, ...] -> [C + E, B + D, A, addr + 2, ...]
D -> mem[addr]
E -> mem[addr + 1]

In this case, the following sequence of instructions would process 8 elements:
pipe_adv
rpperm

And then we need under 300 cycles to move all 1K elements to memory (i.e., a 10x improvement over the baseline).
Of course, the above is a very specialized instruction and the question is whether it is worth having, or whether there are other approaches which could achieve the same result as efficiently.

Thank you @bobbinth for the nice description!
Two points, which are more of curiosities, that I have:

Where does the constraint of needing to add the element, to be hashed, instead of just replacing them come from?
Is there a mechanism of extending the stack "indefinitely" with the constants before the start? And can such a mechanism be leveraged/modified in the context of the current issue?

Where does the constraint of needing to add the element, to be hashed, instead of just replacing them come from?

This comes from Rescue Prime specifications. I wonder if this can be changed without the loss of security.

Is there a mechanism of extending the stack "indefinitely" with the constants before the start? And can such a mechanism be leveraged/modified in the context of the current issue?

Yes, this is possible and, I believe, @itzmeanjan actually implemented something like this for secp256k1 point multiplication. However, this will not work in all cases - so, we still need to find an efficient for moving data from the advice tape to memory.
Another thing that comes to mind: we probably also need a revers of these operation: computing a sequential hash of some memory region. This would be useful, for example, if we want to return a lot of data as output. This data could be written to memory, then hash of this data could be computed, and then this hash could be returned via the stack.
So, we'll probably need another operation for this: maybe something like pipe_mem.

This comes from Rescue Prime specifications. I wonder if this can be changed without the loss of security.

The sponge construction does allow for what is called "overwrite mode" as per section 6.2 in the reference work https://keccak.team/files/SpongeDuplex.pdf . So I don't see why it shouldn't work in our setting.
The question then is: would this simplification make your first proposal to use advice tape, without new instructions, competitive enough?

If that is possible, that would simplify things quite a bit (and not only here). I think we probably would still want specialized instructions as I think this sequential "unhashing" will be a very heavily used operation in the VM - so, we'd want to make it as efficient as possible.

A more concrete description of what we could do here. Suggestions on better names for operations/instructions are welcome.
Add PIPE operation and related assembly instruction
As described above, the PIPE operation would move 8 elements from the head of the advice tape into memory via the stack. The semantics of the operation are as follows:
Denoting stack registers as $s_0, ..., s_{15}$ with $s_0$ being at the top of the stack, and advice tape items as $t_0, ..., t_7$, with $t_0$ being at the head of the tape, the operation induces the following stack transition:
$$
s_i' = s_i + t_{7 - i} \text{ for } i \in 0, ..., 7
$$
$$
s_i' = s_i  \text{ for } i \in 8, ..., 11
$$
$$
s_{12}' = s_{12} + 2
$$
To describe the above informally:

We add 8 items at the head of the advice tape to the top 8 elements of the stack. This addition happens in "stack order" - i.e., $t_0$ is added to $s_7$, $t_1$ is added to $s_6$ etc. The added items are removed from the tape.
We copy over the values of the following 4 stack elements to the next row.
We increment the 13th stack element by 2.

In addition to the above, we update the memory as follows:

$mem[s_{12}] \leftarrow [ s_{7}' - s_{7}, s_{6}' - s_{6}, s_{5}' - s_{5}, s_{4}' - s_{4} ]$
$mem[s_{12} + 1] \leftarrow [ s_{3}' - s_{3}, s_{2}' - s_{2}, s_{1}' - s_{1}, s_{0}' - s_{0} ]$

Basically, we write $[t_0, t_1, t_2, t_3]$ into memory address specified by $s_{12}$, and $[t_4, t_5, t_6, t_7]$ into the memory address specified by $s_{12} + 1$. This does require two memory operations (two writes), and thus, we need to use an opcode with a low degree. I think degree 6 opcode should work here - so, we can probably use 1010100.
The assembly instruction to invoke this operation could be adv_pipe.
Note: this procedure will change slightly after we migrate to RPO hash function (#474). The main difference will be that we would be able to replace field elements at the top of the stack rather than add into them.
Add MSTREAM operation and related assembly instruction
The MSTREAM operation would add 8 elements from adjacent memory addresses to the elements at the top of the stack. The semantics of the operation are as follows (using the same notation as above):
$$
s_{7 - i}' = s_{7 - i} + mem[s_{12}][i] \text{ for } i \in 0, ..., 3
$$
$$
s_{3 - i}' = s_{3 - i} + mem[s_{12 + 1}][i] \text{ for } i \in 0, ..., 3
$$
$$
s_i' = s_i  \text{ for } i \in 8, ..., 11
$$
$$
s_{12}' = s_{12} + 2
$$
To describe the above informally:

We take the word at memory address specified by $s_{12}$ and add its elements to the second word from the top of the stack.
We take the word at memory address specified by $s_{12} + 1$ and add its elements to the word at the top of the stack.
We copy over the values of stack elements $s_8, ..., s_{11}$ to the next row.
We increment the 13th stack element by 2.

This requires two memory operations (two reads), and thus, we need to use an opcode with a low degree. I think degree 6 opcode should work here - so, we can probably use 1010110.
The assembly instruction to invoke this operation could be mem_stream.
Add memory advice injector decorator
The purpose of this decorator is to copy a region of memory into the advice provides and put it under the specified key. The assembly instruction for this decorator could be adv.mem.[start addr].[num word].
For example, adv.mem.4.3 would take 3 words from memory at addresses 4, 5, and 6, and would put them into the advice provider under the key specified by the top 4 elements of the stack.
To support this operation, in addition to the decorator itself, we'll need to add a method to the advice provider to support inserting into the value map. This could look something like this:
pub fn insert_into_map(&mut self, key: Word, values: Vec<Felt>) -> Result<(), ExecutionError>

We could also support other variants of this decorator (though, this support could also be added later on). For example, mem.adv could take both the start address and number of words from the stack as well.

Closed by #486, #488, and 491.


(SERVER-2251) Add '--verbose' flag to 'puppetserver ca' cli tool

Reformat the output of the URL for the logger in verbose mode.

Can you update the commit message to be more specific to this change? Something like "Print URL string instead of raw Ruby object"?

Can you update the commit message to be more specific to this change? Something like "Print URL string instead of raw Ruby object"? You can either keep the ticket number or tag it as "(maint)".

I tried 😢

@Tu2607 you should be able to do a git commit --amend and then force push to your branch git push -f origin HEAD


[Bug] [JDBC] Connection leak when exception occurs

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened
[JDBC] Connection leak when exception occurs

SeaTunnel Version
dev
SeaTunnel Config
-

Running Command
-

Error Exception
-

Zeta or Flink or Spark Version
No response
Java or Scala Version
No response
Screenshots
No response
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

please assign me

The new multi-table write feature changes the way connections are managed, and the connection leak issue has been resolved in the dev branch.


Placeholder of TextViewRowFormer doesn't handle multiple lines

Hello,
I have a long placeholder for a TextViewRowFormer but I can't show it on multiple lines unfortunately.
Is there something to set to handle this case?

Hey @BabyAzerty, have you found a solution?

@ZacharyKhan Yes, I ended up creating my own cell
import UIKit
import Eureka

class DescriptionEurekaCell: Cell<String>, CellType {

    @IBOutlet weak var textView: UITextView!

    override func setup() {
        super.setup()
        textView.text = row.value ?? ""
        textView.sizeToFit()
        textView.isSelectable = true
        textView.dataDetectorTypes = .all
    }

    override func update() {
        super.update()
        textView.text = row.value ?? ""
        textView.sizeToFit()
    }
}

final class DescriptionEurekaRow: Row<DescriptionEurekaCell>, RowType {
    required public init(tag: String?) {
        super.init(tag: tag)
        cellProvider = CellProvider<DescriptionEurekaCell>(nibName: "DescriptionEurekaCell")
    }
}

And you just have to create a xib with a single uitableviewcell view + a uitextview inside it (and link them to the DescriptionEurekaCell).


Comparing newly created class with an already existed class structures

Hi,
I'm making new classes using a set of json string responses and saving the .java class files.
I need to compare the json response structure with an already created .java file (if exists) every time before making a class of that json string.
Does jsonschema2pojo have an inner tool to do this comparison? In the absence of such a tool, I thought of try mapping (by Jackson Object mapper) the json response to the already existed class, if the fields don't match and Object mapper throws an error (hopefully), Then we could continue with the task of replacing this new file with the older version .java file.
Something like this:
	JCodeModel codeModel = new JCodeModel();

	GenerationConfig config = new DefaultGenerationConfig() {

		@Override
		public SourceType getSourceType() {
			return SourceType.JSON;
		}
	};

	SchemaMapper mapper = new SchemaMapper(
			new RuleFactory(config, new Jackson2Annotator(config), new SchemaStore()), new SchemaGenerator());

	mapper.generate(codeModel, action, packageName, apiResultString);

	codeModel.build(Files.createTempDirectory("test").toFile());

	// Compile the .java file ...

	try {
		objectMapper.reader(apiResultAsString, CreatedClass.class);

		//Remove the CreatedClassTemp.java file since the old class apparently matches the current json response structure

	} catch (Exception e) {

		// Replace CreatedClass.java file with the new file CreatedClassTemp.java
	}

However, This has an overhead while needs to create and compile a temporary class, in every attempt.
What do you suggest?
Thanks

There's nothing in this project that will do this for you. Sorry, this has nothing to do with this project at this point, maybe talk this through with a colleague?

Thanks for the response Anyway :)


KodaTree minimal reference data snapshot

We are playing with the idea of having a static snapshot to probably avoid cases of what's happening due #1513 and this could unpack at the client way faster than handling each network per se, using IPFS addressing system for this.
It would be needed in cases like if user visits collection, we can have linked all NFT in that collections, so what happens you can get everything.
At some point, SubSquid might help solve this but still adds one round-trip per request and hence static immutable NFTs won't change much this static mapping could be faster on clients locally.
The purpose of the tree would be super-charge fetching data about items, collections &c as for binary items we would have #1515

Prob already solved by stellate


⚠ Bofrost.es has degraded performance

In 5d967aa, Bofrost.es (https://www.bofrost.es) experienced degraded performance:

HTTP code: 200
Response time: 1642 ms

Resolved: Bofrost.es performance has improved in 1a18c8c after 8 minutes.


Documentation for 'Fop native failures due to pdfbox 3 upgrade' #1

fugerit79/camel-quarkus/issues/1
#5244
#5234
This PR's goal is to
Fix #6743
And document on camel-quarkus-fop limitations page the fact that native profile is currenlty disabled.
[X] An issue should be filed for the change unless this is a trivial change (fixing a typo or similar). One issue should ideally be fixed by not more than one commit and the other way round, each commit should fix just one issue, without pulling in other changes.
[X] Each commit in the pull request should have a meaningful and properly spelled subject line and body. Copying the title of the associated issue is typically enough. Please include the issue number in the commit message prefixed by #.
[X] The pull request description should explain what the pull request does, how, and why. If the info is available in the associated issue or some other external document, a link is enough.
[X] Phrases like Fix # or Fixes # will auto-close the named issue upon merging the pull request. Using them is typically a good idea.
[ ] Please run mvn process-resources -Pformat (and amend the changes if necessary) before sending the pull request.
[X] Contributor guide is your good friend: https://camel.apache.org/camel-quarkus/latest/contributor-guide.html

Thanks @fugerit79 - can you run the following command locally to regen the docs module as well.
./mvnw -f extensions/fop/runtime/pom.xml clean install

Thanks @zhfeng , the docs module has been generated.
Is there anything else I can do?

This may actually not be needed in the next release. FOP in Camel 4.9.0 will use pdfbox 3.x.

@jamesnetherton as you think, but I lost some time before understanding why the native profile was not working.
Is there any due date for the release? And are we sure the native profile will work again for camel-quarkus-fop?
If there are no proof that the native build will work again for sure, I would rather update the documentation and then update it again.
So I guess would better to document the issue right now.

And are we sure the native profile will work again for camel-quarkus-fop

I need to investigate it. But I don't see why it can't be made to work when we have the correct pdfbox version.
Anyway, I'll this for now until it's fixed.


cmd/go: 1.20 broke buildvcs inside bare Git repositories with worktrees

What version of Go are you using (go version)?

$ go version
go version go1.20.2 linux/arm64

Does this issue reproduce with the latest release?
Yes, this issue was introduced in Go 1.20 release.
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/home/tie/.cache/go-build"
GOENV="/home/tie/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS="-trimpath"
GOHOSTARCH="arm64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/tie/go/pkg/mod"
GONOPROXY="go.pact.im/private"
GONOSUMDB="go.pact.im/private"
GOOS="linux"
GOPATH="/home/tie/go"
GOPRIVATE="go.pact.im/private"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/nix/store/hi9hs61z09yi1c9v5paw4273cbyhf51w-go-1.20.2/share/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/nix/store/hi9hs61z09yi1c9v5paw4273cbyhf51w-go-1.20.2/share/go/pkg/tool/linux_arm64"
GOVCS=""
GOVERSION="go1.20.2"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="0"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -fno-caret-diagnostics -Qunused-arguments -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build711157649=/tmp/go-build -gno-record-gcc-switches"

What did you do?

go20-buildvcs-issue.sh
#!/bin/sh
set -eu

temp=$(mktemp -d)
cleanup() {
        rm -r -f -- "$temp"
}
trap cleanup EXIT

cd "$temp"

cat >go.mod <<EOF
module test
EOF

cat >main.go <<EOF
package main

import "fmt"

func main() {
        fmt.Println("OK")
}
EOF

(
        mkdir source && cd source
        git init --quiet
        cp ../go.mod ../main.go .
        git add go.mod main.go
        git commit --quiet --message 'initial commit'
)

gobuild() {
        go build -o out
        go version -m out
        ./out && rm out
}

go version

(
        echo && echo "Git repository"
        git clone --quiet ./source good && cd good
        gobuild
)
(
        echo && echo "Git repository with nested worktree"
        git clone --quiet ./source nested-worktree && cd nested-worktree
        git worktree add --quiet nested && cd nested
        gobuild
)
(
        echo && echo "Bare Git repository with worktree"
        git clone --quiet --bare ./source bare/.git && cd bare
        git worktree add --quiet main && cd main
        gobuild
)

Go 1.19
$ nix shell nixpkgs#go_1_19 -c sh go20-buildvcs-issue.sh
go version go1.19.6 linux/arm64

Git repository
out: go1.19.6
        path    test
        mod     test    (devel)
        build   -compiler=gc
        build   -trimpath=true
        build   CGO_ENABLED=0
        build   GOARCH=arm64
        build   GOOS=linux
        build   vcs=git
        build   vcs.revision=043649dd3ee451e187991c8ce7d54934b2d47cda
        build   vcs.time=2023-03-16T03:13:43Z
        build   vcs.modified=false
OK

Git repository with nested worktree
out: go1.19.6
        path    test
        mod     test    (devel)
        build   -compiler=gc
        build   -trimpath=true
        build   CGO_ENABLED=0
        build   GOARCH=arm64
        build   GOOS=linux
        build   vcs=git
        build   vcs.revision=043649dd3ee451e187991c8ce7d54934b2d47cda
        build   vcs.time=2023-03-16T03:13:43Z
        build   vcs.modified=false
OK

Bare Git repository with worktree
out: go1.19.6
        path    test
        mod     test    (devel)
        build   -compiler=gc
        build   -trimpath=true
        build   CGO_ENABLED=0
        build   GOARCH=arm64
        build   GOOS=linux
        build   vcs=git
        build   vcs.revision=043649dd3ee451e187991c8ce7d54934b2d47cda
        build   vcs.time=2023-03-16T03:13:43Z
        build   vcs.modified=false
OK

Go 1.20
$ nix shell nixpkgs#go_1_20 -c sh go20-buildvcs-issue.sh
go version go1.20.2 linux/arm64

Git repository
out: go1.20.2
        path    test
        mod     test    (devel)
        build   -buildmode=exe
        build   -compiler=gc
        build   -trimpath=true
        build   CGO_ENABLED=0
        build   GOARCH=arm64
        build   GOOS=linux
        build   vcs=git
        build   vcs.revision=fc4846bf2f91ae1065294ddb0d1696e58a9cf850
        build   vcs.time=2023-03-16T03:17:37Z
        build   vcs.modified=false
OK

Git repository with nested worktree
out: go1.20.2
        path    test
        mod     test    (devel)
        build   -buildmode=exe
        build   -compiler=gc
        build   -trimpath=true
        build   CGO_ENABLED=0
        build   GOARCH=arm64
        build   GOOS=linux
        build   vcs=git
        build   vcs.revision=fc4846bf2f91ae1065294ddb0d1696e58a9cf850
        build   vcs.time=2023-03-16T03:17:37Z
        build   vcs.modified=true
OK

Bare Git repository with worktree
error obtaining VCS status: exit status 128
        Use -buildvcs=false to disable VCS stamping.

Notice that Go 1.19 works fine with this setup.
A shorter reproducer for this bug:

go20-buildvcs-issue.sh
#!/bin/sh
set -eu

temp=$(mktemp -d)
cleanup() {
        rm -r -f -- "$temp"
}
trap cleanup EXIT

cd "$temp"
git init --quiet
echo -n 'module test' >go.mod
echo -n 'package main;func main(){}' >main.go
git add go.mod main.go
git commit --quiet --message 'initial commit'
git config core.bare true && rm go.mod main.go
git worktree add --quiet main && cd main
go build -o out
go version -m out

Also note that this is not an issue with Go version in Nixpkgs, the examples above use nix just to avoid installing different Go versions.
What did you expect to see?
All builds succeed.
What did you see instead?
Go 1.20 runs git status --porcelain outside of the Git work tree, the command fails with fatal: this operation must be run in a work tree error.

Seems to be introduced in bb3965695d3bc3586fba55c7b8d5e8a56cd4c0c9 that added checks for file mode when searching for repository root.

For reference, in Git work trees, .git is a file containing gitdir: <path-to-root>/.git/worktrees/<name> (although the path is absolute by default, Git works with relative paths too).

I’m not really sure if bb3965695d3bc3586fba55c7b8d5e8a56cd4c0c9 was the right way to fix the case in #53640, which had a broken Git repository in the first place (since only the submodule was copied to the Docker container).

I still think the fix for #58218 will do what you want
CL 463849

Oh, right, sorry for not looking up existing issues! I known that work trees are somewhat an advanced Git feature and are likely used by a minority of users, and since this wasn’t fixed yet I’ve assumed that nobody reported this issue 😅
Will the fix be eventually backported to 1.20 release?

cc @bcmills @mvdan

Yes, this is a duplicate. And I would say we would backport the revert, given that it's a regression and the revert should be relatively harmless.


Log to stdout instead of stderr

Dear Jarek,
the logging output currently goes to STDERR and not to STDOUT
Could you change that, so that logging output goes to STDOUT by default or that it would be possible to change it to STDOUT by configuration / parameter?
Background is that I want to run this as a cron job.
The place to change that technically is in sync.py, line 73, call to logging.basicConfig (set parameter "stream" to "sys.stdout").
Best regards,
Michael

Hi Michael,
I agree, perhaps the way how the output is written is not very consistent. But for cronjob, typical way of handling this problem is to redirect STDERR to STDOUT and then write to a log, e.g.
jhartman@wieloryb:~$ crontab -l

00 10 * * * ( . ~/.profile; cd /home/jhartman/scripts/withings-garmin-v2/; date; ./sync.py --garmin-username=jaroslaw.hartman --trainerroad-user=jhartman 2>&1 ) >> /var/log/withings.log

(note 2>&1 in the line)
Take a look if this will work for you.
Best regards,
Jarek

Dear Jarek,
Thank you, that would certainly work.
But I think with this cron job configuration you don't get automatically informed when something goes wrong (exception).
But on the other hand one will probably notice sooner or later whether synchronization has stopped working, so an automatic notification is not really required.
For the time being I've just modified my local installation to log to STDOUT. Maybe I'll have a look at a pull request / fork some time later.
So I'm closing this ticket, thanks again for your quick reply!
Best regards,
Michael


Frontend rewrite using Angular

Hi @josephdadams ,
This is such a cool project, I really like it. Thanks for creating it.
However, I found some things which could be improved (like missing help texts or bootstrap classes for input fields, chat issues on layout mobile devices, ...). I'd be happy to work on all that.
But as the UI is already quite complex with elements constantly being created with HTML strings and then added to the DOM, I thought that maybe using a frontend framework would reduce complexity and improve maintainability of the code?
I'm personally quite familiar with Angular, so if I did a frontend rewrite using Angular, would you be interested in a PR?

A second step could be rewriting the backend with TypeScript. This way we could create a class for each Source and get rid of all the switch-case logic.

Sure! Happy to have you contribute!


Rename “contexts” to something more meaningful

It should not be necessary to explain what “contexts” means, what it contains.

viewportsData ?

Contexts contain also screen density, not only viewports.
Maybe viewportsAndDensities?

ScreenDensity is related to viewport and not to screen because it's Also related to zoom. So, width, height and density are data about viewport.

That's right if you consider the viewport as physical pixels, but we need to consider CSS viewport and density separately.
One reason is that we might want to compress more images meant for densities above 2dppx: #64

Let's go for stats


Certain project details fields are empty intermittently

Actual: description and other fields are empty

Expected: fields should be populated. The service returns them, as the screenshot proves.
After switching to the details page for a different project, and then switching back, the description is actually displayed so I'm inclined to think that there is a race at play here:

Hi @hannes-ucsc , it looks like the root cause of this defect is in the area where we format the project's organ value/s for display. We are currently assuming there is always a value for organ but this is not always the case. We get a JS error when we attempt to manipulate an undefined organ value, which in turn prevents subsequent formatting of the project values to occur (for example, project description).
This defect will be fixed once I update this area of the code to (re)use the mappers I created for #660.
I am unsure however, about the intermittent behavior you mentioned.
Looking at your screenshot of the project detail page above, it has an organ value of decidua, blood, placenta.
Looking at the project in production right now, it appears to have no value for organ.
Here is a screenshot of the project response in the data table:

Here is a screenshot of the project response on the detail page:

The write to and read from our local store is atomic. That is, the selected project is updated completely or not at all: there is no partial update of the project. Is it possible that the server is intermittently returning a different value for organ?

Not sure what you are asking, @MillenniumFalconMechanic. Looks like you initially though Azul was intermittently returning different results for the same project detail URL but later retracted that observation? To reproduce the intermittent nature of this bug, follow these steps:
On the Projects tab, click on a project that has a value other than Unspecified in the Organ column. Hit Back button.  Click on a project with Unspecified in the Organ column. The project description will be displayed but the organ will be displayed as the value from the previously displayed project. Hit Refresh. The project details page reverts to not displaying the project description.

We are currently assuming there is always a value for organ but this is not the case.

I just want to clarify that according to the metadata standard, there is always an organ for all projects. The complexity comes from displaying organ for specimens and organ_model for cell lines and organoids. Cell lines and organoids do in fact have their own organ metadata. There is a ticket in Azul about this topic, maybe @zperova can find/post it here?

@MillenniumFalconMechanic I wonder whether this issue is related with the recent changes in Azul to capture both organ for specimens and organ_model for cell lines and organoids:
https://github.com/DataBiosphere/azul/issues/1022
https://github.com/DataBiosphere/azul/issues/977

Hi @hannes-ucsc, that is correct, I retracted my question regarding the Azul response after I was able to consistently reproduce the problem. The root cause remains unchanged: there is a JavaScript error on this page when we encounter a project with no key for organ.
The steps to reproduce are:

Navigate to projects tab
Select a project with a specified value for organ
Navigate back to the projects table
Select a project with a value of "Undefined" for organ

At this point, you can see the project page is displaying a mix of the two projects you have selected.

@malloryfreeberg

Cell lines and organoids do in fact have their own organ metadata.

Not all projects in production, do. Pe'er doesn't have model_organ on cell_lines. But that's special because you're still planning to reingest it, correct?
There are other projects in production that also display Unspecified in the Organ column and that must have a different reason. Azul returns the model organ correctly under hits[*].samples.modelOrgan for those projects.
@danielsotirhos will be exposing the effective organ as part of hits[*].samples.effectiveOrgan in DataBiosphere/azul#1022 which @zperova linked to above. IF Data Browser only wants to display a single value in a single organ column (and I am not sure that's advisable and have argued against ad nauseam before), Data Browser should use hits[*].samples.effectiveOrgan.

@hannes-ucsc I can see 4 dataseta that have Unspecified in the Organ model in the Browser. I looked at the metadata and we have the corresponding fields filled in (including in the Pe'er dataset, which indeed needs to be re-ingested but that should not matter for displaying the Organ model).
The Organ column should be populated from cell_line.model_organ.text (or cell_line.model_organ.ontology_label, I forgot which one you use at the moment) for the following datasets:

Single-cell RNA-seq analysis  throughout a 125-day differentiation protocol that converted H1 human embryonic stem cells to a variety of ventrally-derived cell types.
Comparison, calibration, and benchmarking of high-throughput single cell RNA-Seq techniques for unbiased cell-type classification
Profiling of CD34+ cells from human bone marrow to understand hematopoiesis (Pe'er)

The Organ column should be populated from organoid.model_organ.text (or organoid.model_organ.ontology_label) for Assessing the relevance of organoids to model inter-individual variation.
Could you please confirm that this is indeed the case? Otherwise, I do not understand why it is displayed as Unspecified.

@zperova: Looking at Pe'er. I took one example bundle (11be0dbd-ad77-49b9-8c46-ebb455e88eea.2018-12-05T172232.264458Z) and its cell_line_0.json does not have an model_organ:
{
        "describedBy": "https://schema.humancellatlas.org/type/biomaterial/9.0.0/cell_line",
        "schema_type": "biomaterial",
        "biomaterial_core": {
            "biomaterial_id": "HS_BM_1_cell_line",
            "biomaterial_name": "Bone Marrow CD34+ stem/progenitor cells",
            "ncbi_taxon_id": [
                9606
            ]
        },
        "supplier": "AllCells, LLC",
        "catalog_number": "ABM022F",
        "catalog_url": "https://www.allcells.com/products/bone-marrow-cd34-stem-progenitor-cells",
        "cell_line_type": "stem cell",
        "cell_type": {
            "text": "CD34-positive, CD38-negative hematopoietic stem cell",
            "ontology": "CL:0001024"
        },
        "date_established": "2016-10-12T00:00:00Z",
        "genus_species": [
            {
                "text": "Homo sapiens",
                "ontology": "NCBITaxon:9606"
            }
        ],
        "provenance": {
            "document_id": "b95bd00c-1c00-4f57-820f-f3f431e63377",
            "submission_date": "2018-12-04T19:02:13.661Z",
            "update_date": "2018-12-04T19:02:25.327Z"
        }
    }

So for Pe'er the ball is in Metadata/Ingest's court, as far as I can tell.

@hannes-ucsc you are right - I checked the spreadsheet that is updated for re-ingestion, not the one that corresponds to the dataset currently in the Data Browser. Once re-ingested this will be fixed.

HPSI human cerebral organoids: Azul returns hits[*].samples[0].modelOrgan as brain but Data Browser does not display it.
Drop-seq, DroNc-seq, Fluidigm C1 Comparison: Azul returns hits[*].samples[0].modelOrgan as stem cell but Data Browser does not display it.
Single cell RNAseq characterization of cell types produced over time in an in vitro model of human inhibitory interneuron differentiation: Azul returns hits[*].samples[0].modelOrgan as brain but Data Browser does not display it.

Hmm, it is interesting that the Organ is not displayed at all in the Project view, however in the Project Browser it is displayed as Unspecified - should not these be pulled from the same place?

Hi @zperova, I believe this is due to the defect @hannes-ucsc mentioned originally. I should have the fix, and corresponding tests, ready to push today. Going forward, project data displayed in the data table should always match the project data listed on the corresponding project detail pages.

This update for this is deployed to dev and integration. Will move this to staging on the 6/12 staging deploy.

The update for this is deployed to staging.

@zperova @malloryfreeberg, @hannes-ucsc an update for this ticket has been deployed to production.

The project detail page no longer errors if the organ is unspecified. We are now more resilient to variations on missing data "null", [], [null] etc. or just no key for the property in the json.

In all cases the values displayed on the project detail page should match the values displayed on the projects' row in the projects tab. There was some inconsistency in displaying blank or Unspecified or "--". Now the only indicator used is "Unspecified".

We are still not displaying model organ on the project tab or the project detail page.  We will add this first to the detail page and then to the project table once we are able to horizontal scroll the table as the table is quite full and we don't have room for new columns.

@NoopDog thank you for the update. I do not think Model organ should be a separate column - instead renaming the Organ column to Organ/Model organ should be sufficient as it will be either an organ in case of primary tissue or a model organ in case of organoids and cell lines. Since there is a Sample type column right next to Organ it will be obvious to the consumer whether the value refers to an organ or a model organ. Should I make a separate ticket for that @NoopDog ?

@NoopDog Consider displaying both fields in a single column and annotating model organ values:

Note how the two values are wrapped allowing the column to remain narrow. Horizontal scrolling is a usability nightmare IMO.

I agree with @hannes-ucsc about the horizontal scrolling. More against having a Model organ as a separate column is that in most of the cases (primary tissue samples) it will be empty. We have to keep in mind that once we get imaging datasets in we will have to add imaging related columns or repurpose existing ones to serve both sequencing and imaging metadata. I am very happy to discuss this in more details.

Ok sure, I agree that horizontal scroll can be a pain and we have avoided it so far. We do have many more columns to add so we need some approach to handle. We can investigate alternatives to the horizontal scrolling. Keep in mind that we need to fit on a 13inch mac screen which i believe is only  1280 pixels or so.  In any case, we can explore a more card like "row" that can get taller.
We can try the combined column for the Organ/Model organ in this case. We have already a ticket to add model organ. I can just add to that to combine it with the organ.

Keep in mind that we need to fit on a 13inch mac screen which i believe is only 1280 pixels or so.

Where does that requirement originate from?