Improves Python History call performance

Description

Improves performance of creating a new DataFrame representing the data contained within Slice.
We use Apache Arrow to construct and provide data to the Python side without Boxing of values or having to go through the Python.NET layer.
Each DataFrame that is subsequently created will be faster than the first one, since it won't require any reallocations.
However, this comes at the cost of increased memory usage. Memory usage benchmarking shows that the Arrow implementation can be more memory efficient when constructing smaller DataFrames. However, for large requests, this implementation will use more memory than the previous implementation. Graphs will be provided on Monday.
For large DataFrames, this PR will improve performance as great as >= 2x from the previous implementation.
We use the gsalaz98.Unofficial.Apache.Arrow NuGet package temporarily to resolve a dependency issue that arises from the use of protobuf-net. The common dependency both share that is problematic is the System.Buffers  and System.Memory. The csproj file was adjusted in the Arrow project to build the same targets as protobuf-net, which resolved that dependency issues. No functionality changes have been made to the Arrow project. Let's figure out if this is the path we want to go, and namespace it under QC if we do choose to go this route instead of pointing at my NuGet package.
Also, we add pyarrow to the DockerfileLeanFoundation in this PR.
As if this wasn't enough, we double dip with this PR by fixing #4297 which prevented us from requesting Tick history 😃
Benchmarks:
PR (Options + Research R4-12 node) -- 2014-06-06 to 2014-06-09, all option chains:
72.818 seconds for 528450 rows x 16 cols == 116k data points per second
71.505 seconds for 528450 rows x 16 cols == 118k data points per second
73.518 seconds for 528450 rows x 16 cols == 115k data points per second

Master (Options + Research R4-12 node) -- 2014-06-06 to 2014-06-09, all option chains:
155.667 seconds for 528450 rows x 16 cols == 54k data points per second
156.491 seconds for 528450 rows x 16 cols == 54k data points per second
166.445 seconds for 528450 rows x 16 cols == 51k data points per second

Master (Equity + Research R4-12 node, Resolution.Minute) - 1 month:
0.984 seconds for 7800 rows x 15 cols == 119k data points per second
0.955 seconds for 7800 rows x 15 cols == 123k data points per second
0.969 seconds for 7800 rows x 15 cols == 120k data points per second

PR (Equity + Research R4-12 node, Resolution.Minute) - 1 month:
0.750 seconds for 7800 rows x 15 cols == 156k data points per second
0.740 seconds for 7800 rows x 15 cols == 158k data points per second
0.890 seconds for 7800 rows x 15 cols == 131k data points per second

Master (Forex + Research R4-12 node, Resolution.Minute) - 1 month:
2.316 seconds for 29220 rows x 12 cols == 151k data points per second
2.306 seconds for 29220 rows x 12 cols == 152k data points per second
2.675 seconds for 29220 rows x 12 cols == 131k data points per second

PR (Forex + Research R4-12 node, Resolution.Minute) - 1 month:
1.462 seconds for 29220 rows x 12 cols == 240k data points per second
1.623 seconds for 29220 rows x 12 cols == 216k data points per second
1.526 seconds for 29220 rows x 12 cols == 230k data points per second

Master (Forex + Research R4-12 node, Resolution.Hour) - 1 month:
1.335 seconds for 487 rows x 12 columns == 4377 data points per second
1.261 seconds for 487 rows x 12 columns == 4634 data points per second
1.294 seconds for 487 rows x 12 columns == 4516 data points per second

PR (Forex + Research R4-12 node, Resolution.Hour) - 1 month:
1.107 seconds for 487 rows x 12 columns == 5279 data points per second
1.225 seconds for 487 rows x 12 columns == 4770 data points per second
1.438 seconds for 487 rows x 12 columns == 4063 data points per second

These benchmarks are also indicative of the performance for History calls in Backtesting. Extensive testing revealed that there's no difference between Backtesting and Research performance gains.
Related Issue

Closes #4297
Motivation and Context

Improves research experience
Requires Documentation Change

No
How Has This Been Tested?

New unit tests covering different kinds of DataFrames and asserting History's behavior for Python
Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[x] Refactor (non-breaking change which improves implementation)
[x] Performance (non-breaking change which improves performance. Please add associated performance test and results)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Non-functional change (xml comments/documentation/etc)

Checklist:

[x] My code follows the code style of this project.
[x] I have read the CONTRIBUTING document.
[x] I have added tests to cover my changes.
[x] All new and existing tests passed.
[ ] My branch follows the naming convention bug-<issue#>-<description> or feature-<issue#>-<description>

Sorry for the late review here! 🙏
The PR using Apache 3 throws Unexpected null field Field.children arrow. Executed the following tests using the previous commit that used using <PackageReference Include="gsalaz98.Unofficial.Apache.Arrow" Version="2.0.2-SNAPSHOT" />
Test HistoryRequestBenchmark.py
        self.SetStartDate(2010, 1, 1)
        self.SetEndDate(2015, 12, 12)

TEST 1  self.History([self.symbol], 60, Resolution.Minute)
master
53.55 seconds at 25k data points per second. Processing total of 1,354,951 data points.
53.94 seconds at 25k data points per second. Processing total of 1,354,951 data points.
53.47 seconds at 25k data points per second. Processing total of 1,354,951 data points.
PR not rebased - pythonNet 1.0.30 & old framework
70.67 seconds at 19k data points per second. Processing total of 1,354,830 data points
PR rebased at https://github.com/Martin-Molinero/Lean/pull/new/feature-python-dataframe-performance-rebased
71.21 seconds at 19k data points per second. Processing total of 1,354,830 data points.
70.74 seconds at 19k data points per second. Processing total of 1,354,830 data points.
TEST 2 - self.History([self.symbol], 600, Resolution.Minute)
master
171.53 seconds at 17k data points per second. Processing total of 2,971,491 data points.
172.15 seconds at 17k data points per second. Processing total of 2,971,491 data points.
PR rebased at https://github.com/Martin-Molinero/Lean/pull/new/feature-python-dataframe-performance-rebased
181.28 seconds at 16k data points per second. Processing total of 2,971,491 data points.
182.66 seconds at 16k data points per second. Processing total of 2,971,491 data points.
Test 2 - self.History([self.symbol], 6000, Resolution.Minute) Start: 1/1/2010 Stop: 1/1/2011
master
238.40 seconds at 14k data points per second. Processing total of 3,221,905 data points.
PR rebased:
230.79 seconds at 14k data points per second. Processing total of 3,221,905 data points.
Profiled call stack of PR test 1

Profiled call stack of master test 1

Closing for now


Find a linter rule to forbid empty catch blocks.

From https://github.com/aws-amplify/samsara-cli/pull/556#discussion_r1380608467 .

Did a little more research on this, and I'm not sure this is possible using AST selectors. The source code for the no-empty lint rule is here and it's looking at the source code to determine if there are comments rather that just looking at the AST

Fixed by https://github.com/aws-amplify/amplify-backend/pull/915


fix the navbar not visible on ipad

📑 Description
Fix the navbar button not display on Ipad
now

after

Status

[ ] Not Completed
[x] Completed

✅ Checks

[x] My pull request adheres to the code style of this project
[ ] My code requires changes to the documentation
[ ] I have updated the documentation as required
[ ] I have checked the page with https://validator.unl.edu/
[x] All the tests have passed
[x] My pull request is based on the latest commit (not the npm version).

ℹ Additional Information

@gi4no Can you have a look at this issue? I think it is related to this PR.

Using a mobile, click the component link and try to open a side bar.
It doesn't show the component menu any more. It shows the main menu.


Provide a 'global' PropertyChanged event (e.g. at the Consumer)

In order to get notified about a parameter value change, you must today subscribe to the PropertyChanged event of that specific Parameter instance.
This individual event subscription is fine and works perfect.
But in case you are interested in ANY parameter value change (e.g. because your are implementing a general viewer/consumer app), you must subscribe to ALL parameter PropertyChanged events - which might be hundreds or thousand of events to subscribe to (depending on the provider you are connected to).
This seems in some cases quite uncomfortable and cumbersome.
Is there a possibility to provide a single/global PropertyChanged event (e.g. at the Consumer class level) which is being raised, in case ANY parameter is changing. The PropertyChangedEventArgs PropertyName  member in this case contain the full path of the related Parameter being changed?!
Many Thanks in advance,
Bernd

Is there a possibility to provide a single/global PropertyChanged event (e.g. at the Consumer class level) which is being raised, in case ANY parameter is changing. The PropertyChangedEventArgs PropertyName member in this case contain the full path of the related Parameter being changed?!

Can you elaborate why you need this? What is the real-world use case? I'm asking because this could difficult to implement efficiently.

I tried to elaborate this already above... but again:
Think of implementing a full and generic Ember+ Tree Viewer, which displays all nodes and parameter leafs including its parameter values.
You of course want(need) to update the UI in case a parameter value changes.
Today you would need to subscribe to ALL parameter PropertyChanged events - which might be hundreds or thousand of events to subscribe to (depending on the provider you are connected to).
This is quite inefficient.
If there would be a single, central, global (however you name it) event on the consumer level, which tells you:
a) the full path of the element/parameter which has been changed
or
b) gives you the related IParameter already
-> you could very easily identify the changed parameter and update the UI accordingly.
This simply saves and eases the implementation of a consumer app.
Here is another scenario/use-case:
Assume you implement a consumer app, which needs to trigger certain logical actions in a connected radio automation or play-out system. This consumer app should be flexible and dynamic and be able to connect to ANY Ember+ provider is a very generic fashion (meaning you do NOT want to 'hard-code' anything for a specific provider).
This means you MUST use the dynamic mode to mirror the providers ember schema/tree/nodes.
Now your consumer app needs to implement in a very generic way rules to trigger the above mentioned actions based on parameter changes.
In order to implement something like this: you would need a central, global event at the consumer level which gets raised, whenever ANY parameter value changes.
Else, the only solution is to again subscriber to ALL existing parameters... which is again very inefficient.

Think of implementing a full and generic Ember+ Tree Viewer, which displays all nodes and parameter leafs including its parameter values.
You of course want(need) to update the UI in case a parameter value changes.

Agreed.

Today you would need to subscribe to ALL parameter PropertyChanged events - which might be hundreds or thousand of events to subscribe to (depending on the provider you are connected to).
This is quite inefficient.

What GUI technology are you using?

Assume you implement a consumer app, which needs to trigger certain logical actions in a connected radio automation or play-out system. This consumer app should be flexible and dynamic and be able to connect to ANY Ember+ provider is a very generic fashion (meaning you do NOT want to 'hard-code' anything for a specific provider).
This means you MUST use the dynamic mode to mirror the providers ember schema/tree/nodes.
Now your consumer app needs to implement in a very generic way rules to trigger the above mentioned actions based on parameter changes.

Ok, IIUC, you are saying that you have one or more sets of parameters. Whenever one parameter of such a set changes, then you need to process the values of all parameters in the set to produce some kind of output, correct?

What GUI technology are you using?

C#: further I am using the DevExpress lib in my UI.
If you want I can send you a sample project of a simple viewer.

Ok, IIUC, you are saying that you have one or more sets of parameters. Whenever one parameter of such a set changes, then you need to process the values of all parameters in the set to produce some kind of output, correct?

I am not sure where you are heading, as no, I don't necessarily have a predefined set of parameters. The mapping of ANY parameter value change should be dynamic. Meaning even at runtime of the application new mapping rules should be added or modified;as such, a global event would simply be optimal.

C#: further I am using the DevExpress lib in my UI.

Ok, DevExpress supports data binding, correct? If you have DevExpress UI elements bind directly to IParameter.Value you don't need to bother with subscribing/unsubscribing yourself, right?

I am not sure where you are heading, as no, I don't necessarily have a predefined set of parameters. The mapping of ANY parameter value change should be dynamic. Meaning even at runtime of the application new mapping rules should be added or modified;

That's fine, at runtime you'd still have one or more sets of parameters, as described before. It doesn't matter whether the number of parameters in a given set changes dynamically or not.

as such, a global event would simply be optimal.

I would argue that it depends, as always. If you have only a single set of parameters and you absolutely need to recalculate everything whenever the value of any of those parameters changes, then yes, a mechanism as the one you're proposing is probably a good idea. OTOH, if you have many independent sets of parameters, I would think it's probably more efficient to subscribe the calculation method for each set to only the PropertyChanged events of the parameters that belong to the set, right?
Sure, you do have to write the code to subscribe and unsubscribe from the different parameters as needed, but such a generic system would do that in a single place anyway, right?

DevExpress supports data binding, correct?

Yes, but I don't want to use that, as it comes with certain limits, which are much easier to overcome without using data binding (especially for larger trees).
In the end it is a philosophical discussion.
If it is too hard/complex to implement on your side, then don't do it. There is no issue with doing so!
I just thought, that it might be easy to implement on your side; e.g. as you already raise an event on a single IParameter; you might simply propagate that to the related IConsumer level...

Yes, but I don't want to use that, as it comes with certain limits, which are much easier to overcome without using data binding (especially for larger trees).

Ok, fair enough.

I just thought, that it might be easy to implement on your side; e.g. as you already raise an event on a single IParameter; you might simply propagate that to the related IConsumer level...

Raising an event on the parameter level can be implemented such that you basically only have a measurable cost if there's a handler subscribed to the event. Once you start propagating such events up the tree, that is no longer possible. I have the suspicion that it's going to be rather inefficient, if you consider the fact that some providers change thousands of parameters in a single message.  Let me think about this some more, it seems there should be a way to implement this such that there is no cost as long as no handler is subscribed to the event.
BTW, given the "some providers change thousands of parameters in a single message" scenario above, you'd get thousands of change messages with said event. Would that be what you want?

BTW, given the "some providers change thousands of parameters in a single message" scenario above, you'd get thousands of change messages with said event. Would that be what you want?

I am afraid yes, as I would also get thousands of change messages when subscribed to the individual parameters events ;-)
However, while thinking of it, if there might be an IConsumer property to define this, that would be best. E.g. something like bool RaiseIndividualChangeEvents.
If that's set to false, only one event per received Glow message would be raised (its event args properties might be set to null in that case).
... just thinking loud ;-)


error: invalid command 'bdist_wheel'

uv 0.2.6
when installing pyyaml==5.4.1
I get:
Resolved 126 packages in 1.66s
error: Failed to download distributions
  Caused by: Failed to fetch wheel: pyyaml==5.4.1
  Caused by: Failed to build: `pyyaml==5.4.1`
  Caused by: Build backend failed to build wheel through `build_wheel()` with exit status: 1
--- stdout:

--- stderr:
usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: -c --help [cmd1 cmd2 ...]
   or: -c --help-commands
   or: -c cmd --help

error: invalid command 'bdist_wheel'
---
  Caused by: This error likely indicates that you need to `uv pip install wheel` into the build environment for pyyaml==5.4.1

If it knows what its missing, why doesn't it install that package?

Ah, it's package's job to declare its build dependencies. The "Caused by" at the end is us trying to provide a helpful hint for how to fix it, but we can't know that that's actually the problem, and it would be spec incompliant for us to make assumptions about it.

Related https://github.com/astral-sh/uv/issues/2252#issuecomment-1985287074
You could resolve this by using --no-build-isolation and installing the requisite build dependencies.
@konstin did you forget to make sure this special case message was only raised under --no-build-isolation? There's not a pull request linked in https://github.com/astral-sh/uv/issues/2342. I think the message is not helpful outside that context because installing wheel won't help.

I think it's helpful, but perhaps should be rephrased? It's way more helpful to know that the wheel package is missing than that bdist_wheel is an invalid command.

We indeed attach the context uncoditionally and lack the build isolation check. This shouldn't be the problem here, pyyaml 5.4.1 correctly declares a dependency on wheel (https://inspector.pypi.io/project/pyyaml/5.4.1/packages/a0/a4/d63f2d7597e1a4b55aa3b4d6c5b029991d3b824b5bd331af8d4ab1ed687d/PyYAML-5.4.1.tar.gz/PyYAML-5.4.1/pyproject.toml).
Can you share the commands to reproduce the error? It installs successfully for me on older python and on newer pythons i get a different error (windows and ubuntu):
$ uv venv -p 3.12
$ uv pip install pyyaml==5.4.1
Resolved 1 package in 280ms
error: Failed to download distributions
  Caused by: Failed to fetch wheel: pyyaml==5.4.1
  Caused by: Failed to build: `pyyaml==5.4.1`
  Caused by: Build backend failed to determine extra requires with `build_wheel()` with exit code: 1
--- stdout:
running egg_info
writing lib3\PyYAML.egg-info\PKG-INFO
writing dependency_links to lib3\PyYAML.egg-info\dependency_links.txt
writing top-level names to lib3\PyYAML.egg-info\top_level.txt
--- stderr:
Traceback (most recent call last):
  File "<string>", line 14, in <module>
  File "C:\Users\Konstantin\AppData\Local\uv\cache\.tmpTnVWP3\.venv\Lib\site-packages\setuptools\build_meta.py", line 325, in get_requires_for_build_wheel
    return self._get_build_requires(config_settings, requirements=['wheel'])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Konstantin\AppData\Local\uv\cache\.tmpTnVWP3\.venv\Lib\site-packages\setuptools\build_meta.py", line 295, in _get_build_requires
    self.run_setup()
  File "C:\Users\Konstantin\AppData\Local\uv\cache\.tmpTnVWP3\.venv\Lib\site-packages\setuptools\build_meta.py", line 311, in run_setup
    exec(code, locals())
  File "<string>", line 271, in <module>
  File "C:\Users\Konstantin\AppData\Local\uv\cache\.tmpTnVWP3\.venv\Lib\site-packages\setuptools\__init__.py", line 103, in setup
    return distutils.core.setup(**attrs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Konstantin\AppData\Local\uv\cache\.tmpTnVWP3\.venv\Lib\site-packages\setuptools\_distutils\core.py", line 184, in setup
    return run_commands(dist)
           ^^^^^^^^^^^^^^^^^^
  File "C:\Users\Konstantin\AppData\Local\uv\cache\.tmpTnVWP3\.venv\Lib\site-packages\setuptools\_distutils\core.py", line 200, in run_commands
    dist.run_commands()
  File "C:\Users\Konstantin\AppData\Local\uv\cache\.tmpTnVWP3\.venv\Lib\site-packages\setuptools\_distutils\dist.py", line 969, in run_commands
    self.run_command(cmd)
  File "C:\Users\Konstantin\AppData\Local\uv\cache\.tmpTnVWP3\.venv\Lib\site-packages\setuptools\dist.py", line 968, in run_command
    super().run_command(command)
  File "C:\Users\Konstantin\AppData\Local\uv\cache\.tmpTnVWP3\.venv\Lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
    cmd_obj.run()
  File "C:\Users\Konstantin\AppData\Local\uv\cache\.tmpTnVWP3\.venv\Lib\site-packages\setuptools\command\egg_info.py", line 321, in run
    self.find_sources()
  File "C:\Users\Konstantin\AppData\Local\uv\cache\.tmpTnVWP3\.venv\Lib\site-packages\setuptools\command\egg_info.py", line 329, in find_sources
    mm.run()
  File "C:\Users\Konstantin\AppData\Local\uv\cache\.tmpTnVWP3\.venv\Lib\site-packages\setuptools\command\egg_info.py", line 550, in run
    self.add_defaults()
  File "C:\Users\Konstantin\AppData\Local\uv\cache\.tmpTnVWP3\.venv\Lib\site-packages\setuptools\command\egg_info.py", line 588, in add_defaults
    sdist.add_defaults(self)
  File "C:\Users\Konstantin\AppData\Local\uv\cache\.tmpTnVWP3\.venv\Lib\site-packages\setuptools\command\sdist.py", line 102, in add_defaults
    super().add_defaults()
  File "C:\Users\Konstantin\AppData\Local\uv\cache\.tmpTnVWP3\.venv\Lib\site-packages\setuptools\_distutils\command\sdist.py", line 250, in add_defaults
    self._add_defaults_ext()
  File "C:\Users\Konstantin\AppData\Local\uv\cache\.tmpTnVWP3\.venv\Lib\site-packages\setuptools\_distutils\command\sdist.py", line 335, in _add_defaults_ext
    self.filelist.extend(build_ext.get_source_files())
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 201, in get_source_files
  File "C:\Users\Konstantin\AppData\Local\uv\cache\.tmpTnVWP3\.venv\Lib\site-packages\setuptools\_distutils\cmd.py", line 107, in __getattr__
    raise AttributeError(attr)
AttributeError: cython_sources
---

well, I'm just using nixos which makes 0 assumtions about the things that are installed. its absolute barebones python, no global packages. the only binary I got is python3.10
then I just do uv pip install pyyaml==5.4.1

Yeah, it's the fault of the package at that specific version. It looks like you need to use PyYAML 6.0.1 in order to have a working install.

There is effectively nothing that uv can do here. That version of the package does not declare the correct build dependencies. So you either need to install them yourself out-of-band (uv pip install wheel, then uv pip install --no-build-isolation pyyaml==5.4.1) or update to a newer version.

(Adding to the existing ticket because I think it's the same issue; if it's actually a different issue I'm happy to file a new ticket.)
I switched a project from pip to uv. It worked for a little while, but then this problem broke the build:
root@27436ccb1f5b:/srv# pip install --upgrade pip setuptools uv wheel
root@27436ccb1f5b:/srv# uv pip install --no-cache --reinstall --system --verbose biopython==1.77
DEBUG uv 0.2.13
DEBUG Searching for Python interpreter in search path
DEBUG Found cpython 3.10.12 at `/usr/local/bin/python3` (search path)
DEBUG Using Python 3.10.12 environment at /usr/local/bin/python3
DEBUG Acquired lock for `/usr/local`
DEBUG Using request timeout of 30s
DEBUG Solving with installed Python version: 3.10.12
DEBUG Adding direct dependency: biopython==1.77
DEBUG No cache entry for: https://pypi.org/simple/biopython/
DEBUG Searching for a compatible version of biopython (==1.77)
DEBUG Selecting: biopython==1.77 (biopython-1.77.tar.gz)
DEBUG No cache entry for: https://files.pythonhosted.org/packages/4f/47/323e23c427b3246cc092a767f1868396a47ae05a7a135ce828600e7fe177/biopython-1.77-cp36-cp36m-macosx_10_9_x86_64.whl.metadata
DEBUG Adding transitive dependency for biopython==1.77: numpy*
DEBUG No cache entry for: https://pypi.org/simple/numpy/
DEBUG Searching for a compatible version of numpy (*)
DEBUG Selecting: numpy==2.0.0 (numpy-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl)
DEBUG No cache entry for: https://files.pythonhosted.org/packages/d6/a8/6a2419c40c7b6f7cb4ef52c532c88e55490c4fa92885964757d507adddce/numpy-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
DEBUG Tried 2 versions: biopython 1, numpy 1
Resolved 2 packages in 479ms
DEBUG Identified uncached requirement: biopython==1.77
DEBUG Identified uncached requirement: numpy==2.0.0
[...]
DEBUG Solving with installed Python version: 3.10.12
DEBUG Adding direct dependency: setuptools>=40.8.0
DEBUG No cache entry for: https://pypi.org/simple/setuptools/
DEBUG Searching for a compatible version of setuptools (>=40.8.0)
DEBUG Selecting: setuptools==70.1.0 (setuptools-70.1.0-py3-none-any.whl)
DEBUG No cache entry for: https://files.pythonhosted.org/packages/55/b3/b3a3415d2debd837106ed417f8681d8af63037fed367fa1b85dbfef081f1/setuptools-70.1.0-py3-none-any.whl.metadata
DEBUG Tried 1 versions: setuptools 1
DEBUG Installing in setuptools==70.1.0 in /root/.cache/uv/environments-v0/.tmp4GN5pP
DEBUG Identified uncached requirement: setuptools==70.1.0
DEBUG Downloading and building requirement for build: setuptools==70.1.0
DEBUG No cache entry for: https://files.pythonhosted.org/packages/55/b3/b3a3415d2debd837106ed417f8681d8af63037fed367fa1b85dbfef081f1/setuptools-70.1.0-py3-none-any.whl
DEBUG Installing build requirement: setuptools==70.1.0
DEBUG Calling `setuptools.build_meta:__legacy__.get_requires_for_build_wheel()`
DEBUG Calling `setuptools.build_meta:__legacy__.build_wheel("/root/.cache/uv/built-wheels-v3/pypi/biopython/1.77/s1czoIEwLOoCMojvWuoZF/.tmpCPVxEH", {}, None)`
error: Failed to download distributions
  Caused by: Failed to fetch wheel: biopython==1.77
  Caused by: Failed to build: `biopython==1.77`
  Caused by: Build backend failed to build wheel through `build_wheel()` with exit status: 1
--- stdout:

--- stderr:
We need both setuptools AND wheel packages installed for bdist_wheel to work. Try running: pip install wheel
---

This is in a container based on Debian 11 Bullseye. Please let me know if you're like a Dockerfile+docker-compose.yaml that reproduce the issue.
Trying older releases, this appears to be a regression: 0.1.17 is the newest release that works for me, while 0.1.18 is the oldest release that fails for me. https://github.com/astral-sh/uv/issues/2313 https://github.com/astral-sh/uv/pull/2341 caught my eye in the commit titles https://github.com/astral-sh/uv/compare/0.1.17...0.1.18
@charliermarsh I see at the time, your spider sense tingled https://github.com/astral-sh/uv/issues/2313#issuecomment-1986875178
The --no-build-isolation flag mentioned above and the --legacy-setup-py flag mentioned in #2341 both work around the issue:
root@bb2a87c16f30:/srv# uv pip install --no-build-isolation --no-cache --system --reinstall biopython==1.77
+ uv pip install --no-build-isolation --no-cache --system --reinstall biopython==1.77
Resolved 2 packages in 233ms
   Built biopython==1.77
Downloaded 2 packages in 8.43s
Uninstalled 2 packages in 17ms
Installed 2 packages in 15ms
 - biopython==1.77
 + biopython==1.77
 - numpy==2.0.0
 + numpy==2.0.0

root@bb2a87c16f30:/srv# uv pip install --no-cache --system --reinstall --legacy-setup-py biopython==1.77
+ uv pip install --no-cache --system --reinstall --legacy-setup-py biopython==1.77
Resolved 2 packages in 474ms
   Built biopython==1.77
Downloaded 2 packages in 6.83s
Uninstalled 2 packages in 32ms
Installed 2 packages in 13ms
 - biopython==1.77
 + biopython==1.77
 - numpy==2.0.0
 + numpy==2.0.0

The latest version of biopython git+https://github.com/biopython/biopython.git seems to behave the same as version 1.77.
I can reproduce the issue with pip 24.1 with the major difference that it works by default and only fails if --use-pep517 is explicitly chosen.
root@bb2a87c16f30:/srv# pip install --force-reinstall git+https://github.com/biopython/biopython.git
+ pip install --force-reinstall git+https://github.com/biopython/biopython.git
Collecting git+https://github.com/biopython/biopython.git
  Cloning https://github.com/biopython/biopython.git to /tmp/pip-req-build-str6jtfe
  Running command git clone --filter=blob:none --quiet https://github.com/biopython/biopython.git /tmp/pip-req-build-str6jtfe
  Resolved https://github.com/biopython/biopython.git to commit cca7983d2e054db56123295034f4eefc919cb7fa
  Preparing metadata (setup.py) ... done
Collecting numpy (from biopython==1.84.dev0)
  Using cached numpy-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (60 kB)
Using cached numpy-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (19.3 MB)
Building wheels for collected packages: biopython
  Building wheel for biopython (setup.py) ... done
  Created wheel for biopython: filename=biopython-1.84.dev0-cp310-cp310-linux_x86_64.whl size=3168711 sha256=a2589ad16cecf149ca311cbca2c0da225b06dd9951f736b4d8e75351603237ef
  Stored in directory: /tmp/pip-ephem-wheel-cache-w490jw9c/wheels/c0/e3/29/3b7bf898d7fe6d4de17fcab24cd454bc9e1deec60388309d51
Successfully built biopython
Installing collected packages: numpy, biopython
  Attempting uninstall: numpy
    Found existing installation: numpy 2.0.0
    Uninstalling numpy-2.0.0:
      Successfully uninstalled numpy-2.0.0
  Attempting uninstall: biopython
    Found existing installation: biopython 1.84.dev0
    Uninstalling biopython-1.84.dev0:
      Successfully uninstalled biopython-1.84.dev0
[...]
Successfully installed biopython-1.84.dev0 numpy-2.0.0

root@bb2a87c16f30:/srv# pip install --force-reinstall --use-pep517 --quiet git+https://github.com/biopython/biopython.git
+ pip install --force-reinstall --use-pep517 --quiet git+https://github.com/biopython/biopython.git
  error: subprocess-exited-with-error

  × Building wheel for biopython (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [1 lines of output]
      We need both setuptools AND wheel packages installed for bdist_wheel to work. Try running: pip install wheel
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for biopython
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (biopython)

Thanks for the write-up! I think this is expected since we use PEP 517 by default, and (as you noted) the package doesn't seem to install with --use-pep517. In that case, the package itself would need to be fixed to enable installing without build isolation.


Separate rowHeight for each item? How to fit to screen height

I am trying to figure out how to fit the grid exactly to the screen height (no scrolling).
Lets take a simple example were there are only 2 items, and the screen height is 401.
Well, 401 does not divide well by 2, so I can't use rowHeight=200.
(This is with margin=0)

One approach to solve this might result in having one item's height 200 and the other 201
Another approach is to have both items height=200, and have a margin of 1px.

I found a similar issue in which it was suggested to use rowHeight=1, and use the h argument for controlling the height in pixels. but that works only if the margin is 0.
Is this solvable with react-grid-layout?
Any suggestions?
Thanks,
Erez

I am trying to achieve the same thing.
Instead of trying to be precise, I am just going the "make-believe" route. Set overflow hidden on the container of the gridlayout. And just calculate the rowHeight everytime the screen resizes with a Math.ceil. E.g. if screen height is 400 and I am using a 12 rows grid, my row height is Math.ceil(400.0/12) so 34. Yes, the total height ends up at 408 so 8 pixels are hidden on the bottom, but visually it works pretty well most of the time for me as the difference is less than 5 pixels.
To the extreme, I think there are number CSS positioning tricks that you can use making the final result more "believable" to be precise such as shifting the outer container etc.


Send message doesnot redirecting to mail.

Whne we are clicking on a send message button we are not redirecting to a mail. i think there are some bugs.

Issue Resolved.


Fix: #47; ensure directory with same name does not exists

This PR is in reference to issue #47

This is really nice. Less errors = good 👍 Thanks!


Fix: Specification link fixed

@volovyk-s just FYI specification is just copy-pasta and doesn't correspond to current commands AFAIK


Rails 5 : Syntax fix recursive loop where prepend-ed modules exist

HI
in_controller.rb  line 600 ish
ancestors[1..-1] is the issue.
protected
     def filter_access_permissions # :nodoc:
        unless filter_access_permissions?
         ancestors[1..-1].reverse.each do |mod|
           mod.filter_access_permissions if mod.respond_to?(:filter_access_permissions, true)
         end
       end
       class_variable_set(:@@declarative_authorization_permissions, {}) unless filter_access_permissions?
       class_variable_get(:@@declarative_authorization_permissions)[self.name] ||= []
     end

This is a recursive method and ancestors[1..-1] appears to be designed to remove the current object's class from the iterator, with the assumption that the current object is its own first ancestor.
eg if ancestors returns
[ ActionController::Base, ActiveRecord::Railties::ControllerRuntime, ....... ]
[1..-1] removes the first element which is assumed to be the current object, in this case ActionController::Base.
However this fails if other gem modules (eg wicked_pdf) are designed to prepend to the front of the ancestors chain.
See Nice Module#prepend explanation here
For example: prepend excerpt from the wicked_pdf gem:
class WickedPdf
  if defined?(Rails)
    if Rails::VERSION::MAJOR >= 5
      class WickedRailtie < Rails::Railtie
        initializer 'wicked_pdf.register' do |_app|
          ActionController::Base.send :prepend, PdfHelper
          ActionView::Base.send :include, WickedPdfHelper::Assets
        end
      end

ancestors returns
[WickedPdf::PdfHelper, ActionController::Base, ActiveRecord::Railties::ControllerRuntime, .......
[1..-1] removes the first element, in this case the prepend-ed WickedPdf::PdfHelper, leaving the current object ( ActionController::Base) in the iterator which causes the infinite loop.
FIX: Replace
ancestors[1..-1].reverse.each do |mod|

WITH
(ancestors - [self] ).reverse.each do |mod|

Thanks
Simon

I experienced this as well and fixed it similarly. I didn't get down to what was causing it so good find on prepending being the issue. We use WickedPdf as well. :+1:


[SPARK-40158][SQL] Remove useless configuration & extract common code for parquet read

What changes were proposed in this pull request?
The pr aim to:

Remove useless configuration:  hadoopConf.set(ParquetWriteSupport.SPARK_ROW_SCHEMA, readDataSchemaAsJson)
Extract common code:

ParquetFileFormat.buildReaderWithPartitionValues (https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala#L202-L228)
parquet/ParquetScan.createReaderFactory (https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetScan.scala#L66-L93)

Why are the changes needed?
1.Remove useless configuration: hadoopConf.set(ParquetWriteSupport.SPARK_ROW_SCHEMA, readDataSchemaAsJson), reason:
(A).After set property(ParquetWriteSupport.SPARK_ROW_SCHEMA) in ParquetFileFormat.buildReaderWithPartitionValues or ParquetScan.createReaderFactory, the following logic does not read this property. In spark, it is only an intermediate property.
(B).In the original logic (commitId: f479cf3743e416ee08e62806e1b34aff5998ac22)
Compatible with the legacy parquet version is SPARK_METADATA_KEY - org.apache.spark.sql.parquet.row.metadata(save in parquet footer metadata), but not SPARK_ROW_SCHEMA - org.apache.spark.sql.parquet.row.attributes

2.Code deduplication improves maintainability.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass GA.

Can one of the admins verify this patch?


Fix STARTTLS to use port 587

SMTPType.STARTTLS was set to 465 as its default port  - it is actually 587. Tests and the mock SMTPMock object were also changed because this uncovered a bug aka a code path that wasn't actually ever tested.

Closes #39
Checklist

[x] This pull request references any related issue by including "Closes #<ISSUE_NUMBER>"

If no issue exists and your change is not a small fix, please create an issue first.

[x] This pull request includes tests or only affects documentation.
[x] Summarized PR's changes in CHANGELOG.md

Thanks for fixing this @drfraser!

Thanks for fixing this @drfraser!


Adjustment in send message to create new chat instead of using the first chat

What's the point of it? To look more humane?

Ye removed the needs to have some chat in device to make the "rewrite" on chat id before send message. Tha way we also don't need to refresh the page to update the chat.models object.

Awesome, I always had this problem that it crashed if the number is new without any chats.
Thanks


Create a shared library for creating DSUs inside DSU-Fabric (UI) and DSU-Fabric-Integration

OneDrive_1_5-26-2021.zip

To be clarified:
1 ) Fields present in UI view and missing form provided message examples :
a) for batch:
- decommissionReason
- recallSerials (for all types of serial numbers)
b) for product:
- version
- description
- manufName
- batchSpecificVersion
2) Fields missing from UI and present in messages:
a) for batch:
- packagingSiteName
- flagEnableACFBatchCheck
- acfBatchCheckURL
b) for product:
- markets (array)
- inventedName
- internalMaterialCode
- strength
To discuss:
Expiry date format ( in message now is - "expiryDate" : "221200", "expiryDate" : "220601", "expiryDate" : "231001")
Serial numbers format (in message each serial is with quotation mark "snValid":[ "9049444522221", "QI88v00001", "00360000066000234"])
Photo for product - how manage

@salboaie Patrick and me answered your questions. Please see all starting with "=>"
The version / batch specific leaflet topic will be discussed next week and we will provide feedback to you after.


Move watermark on tree to somewhere less in-your-face

Bottom left?  smaller?

I'd go with top left and smaller.

Yep! Much smaller -- it's tacky at the moment.


Key and micro block serialization

Key block/header serializations , microblock header and body

@Artur64 and @d-velev can you estimate this after the other issue is split https://github.com/aeternity/elixir-node/issues/732? then adjust the label from x to number of days please.

Blocked until #724 and #725 are resolved

Could you discuss an estimate together with the team?

Could you discuss an estimate together with the team?

This issue is blocked until #739 is resolved

could you explain why serialization is blocked by proof of fraud?

In micro block structure we have to have PoF implementation and the serialization for it, in order to be compatible to epoch's node.


Downgrade / VMWare Fusion problems

A recent VMWare Fusion update has broken all my Vagrant boxes (not for the first time!) In order to debug what happened, I'd like to install an older version of vmware-fusion. Is there a built in method of doing this?
I followed this method, of installing the brew recipe directly:
https://www.jverdeyen.be/mac/downgrade-brew-cask-application/
But that left me with a broken VMWare install:
vmrun Error:
vmrun must be set-UID root, "/Applications/VMware Fusion.app/Contents/Library/vmware-vmx" is not. Are you running /Applications/VMware Fusion.app/Contents/Library/vmware-vmx from its distribution directory? That copy of the program is not set-UID root.

Thanks

The method in that blog post is absolutely not officially supported, as is not at all recommended. It’s more than likely it’ll break, as it did for you.
Uninstall your current version and look for another one in caskroom/versions.

I guessed as much by the results!
So what's the plan if the particular version is not in caskroom/versions?

So what's the plan if the particular version is not in caskroom/versions?

You add it, as long as it is accepted by the rules or you make your own tap.


Discuss: Remove cterm colors from themes

The genarated cterm values are pretty good . Most of the themes donn't have cterm values in them.  Those cterm values create unnecessary complexicity in themes . Also since rest of the codebase relies on genarated cterm colors (like section separators) They can look weird if themes provide cterm values thata re different . So I think we should remove cterm values from themes and relay solely on genarted cterm values .

Yeah let's do it. A cleaner codebase is always better.


Update configuration & validation

Please read these guidelines before opening an issue.

Feature Description
We need to update the properties used for configuration to match those available in the new client (cloudant.url and friends).
Further as described in Connect Configuration Validation we should utilize the validation and recommendation hooks. So for each ConfigDef:
* the dependencies should be set (e.g. cloudant.password depends on cloudant.username and they depend on some specific auth types cloudant.auth.type=basic|couchdbsession). An implementation of Recommender should be added to provide this help to the user.
* As described in  we should override the Validator class in the ConfigDef to provide option validation.
With that information present in the ConfigDef I think the default implementation of validate() will likely be sufficient for our needs (i.e. I don't currently see a need for us to override and automatically set recommended values).
Since a lot of that will be duplicated between source and sink for all the common options of client config we should move those options to a common function that can be shared.
1. Steps to reproduce and the simplest code sample possible to demonstrate the issue
Currently config validation happens in a few different places:
https://github.com/cloudant-labs/kafka-connect-cloudant/blob/61e0e691fe6a887c5cdc76f1e557221a71dc43cd/src/main/java/com/ibm/cloudant/kafka/connect/CloudantSourceConnector.java#L84
https://github.com/cloudant-labs/kafka-connect-cloudant/blob/61e0e691fe6a887c5cdc76f1e557221a71dc43cd/src/main/java/com/ibm/cloudant/kafka/connect/CloudantSourceTask.java#L187
and we lack any real validation of properties.
2. What you expected to happen
Properties to be validated using standard Kafka Connect approach.
3. What actually happened
No real validation of properties before connector failure.

NB work for this and follow-on tickets will merge to feature branch feature/61-configuration-validation

PR merged: https://github.com/cloudant-labs/kafka-connect-cloudant/pull/70
Closing


Datasets without information leakage.

Are the three datasets under folder data/USPTO50K/canonicalized_csv  already processed by canonicalize_products.py? Thanks.

Yes, they are processed under branch canonical_product. You can find them from this path:
https://github.com/uta-smile/RetroXpert/tree/canonical_product/data/USPTO50K/canonicalized_csv

Thanks.


Feature Request: Option to treat subsequent points with different SWC tags as fork points

Consider a morphology with an axon consisting of a single branch, but with 3 different SWC tags: 2 for axon, 6 for axon hillock, and 7 for axon initial segment. With the current MorphIO parser this gets reduced into 1 Section with 1 type, depending on the SWC tag of the first point of the branch.
I'd like to request a new option to split this branch into 3 Sections, wherever there is an SWC tag transition (taking into account the no_duplicates option)

I think I noticed something similar just recently; let me create an example file, and we can check that we're on the same page.

Conceptually a series of SWC samples with the following tags:
1-1-1-6-6-6-7-7-7

gets converted in MorphIO to a single Section with type 1:
|--------1-------|

more equivalent to this SWC:
1-1-1-1-1-1-1-1-1

I'd like an option to parse this as 3 Section with their respective types:
|--1--|--6--|--7--|

Yup, we're on the same page:
contents =('''1 1 0 4 0 3.0 -1
              2 6 0 0 2 0.5 1
              3 7 0 0 3 0.5 2
              4 8 0 0 4 0.5 3
              5 9 0 0 5 0.5 4''')
n = Morphology(contents, "swc")

should have len(n.sections) == 4
I will take a look when I have moment.  I have a rewrite of the SWC parser in progress, that covers this case, afaik.  I need to finish it though.

@Helveg I have implemented a change that should fix this.
However, it's in a completely new SWC reader - I'm passing all the tests that I have in NeuroM and nearly all of them in MorphIO (the ones that I'm not, I think we should re-evaluate what is right.
I'd really appreciate it if you could try with your code, because I think:
a) it would fix this issue
b) you may turn up other issues.
The branch w/ my new reader is: https://github.com/BlueBrain/MorphIO/compare/efficient-swc-build
Let me know if you have any luck.

Hmm, my version is MorphIO-3.3.6.dev54+g57e4e98.d20230811 and I get this error:
>       morpho = morphio.Morphology(os.fspath(file_like))
E       morphio._morphio.RawDataError:
E       /mnt/c/Users/pwd06/git/arborize/tests/data/morphologies/multitagged.swc:1:error
E       Unable to parse this line

for this SWC file, which looks good:
1 1 0.0 0.0 0.0 2.9000000953674316 -1
2 1 0.0 5.622320175170898 0.0 2.9000000953674316 1
3 6 0.0 5.622320175170898 0.0 0.75 2
4 6 0.0 6.622320175170898 0.0 0.75 3
5 7 0.0 6.622320175170898 0.0 0.3499999940395355 4
6 7 0.0 16.6223201751709 0.0 0.3499999940395355 5
7 8 0.0 16.6223201751709 0.0 0.15000000596046448 6
8 8 0.0 29.222320556640625 0.0 0.15000000596046448 7
9 8 0.0 41.82231903076172 0.0 0.15000000596046448 8
10 8 0.0 54.42232131958008 0.0 0.15000000596046448 9
11 8 0.0 67.02232360839844 0.0 0.15000000596046448 10
12 8 0.0 79.62232208251953 0.0 0.15000000596046448 11
13 8 0.0 92.22232055664062 0.0 0.15000000596046448 12
14 8 0.0<PHONE_NUMBER>3076172 0.0 0.15000000596046448 13
15 8 0.0<PHONE_NUMBER>0488281 0.0 0.15000000596046448 14
16 8 0.0<PHONE_NUMBER>0839844 0.0 0.15000000596046448 15
17 8 0.0<PHONE_NUMBER>53125 0.0 0.15000000596046448 16

I tried loading the above snippet, and it seemed to work.  I will try some variations on it though.  I don't have a WSL2 machine at my disposal, though.

It's the line endings. After running dos2unix on the file the parser works. I think Windows line endings are \r\n, maybe you detect line endings in a platform specific manner?

It's the line endings, I believe. After running dos2unix on the file the parser works. I think Windows line endings are \r\n, maybe you determine the line ending in a platform specific manner?

Yup, thanks for catching that.

Any status updates? :)

Sorry, I was away, and haven't had a single stitch of time to look since I got back.  I think it's a quick fix, though, so I can maybe find time today.

I have added handling of windows end of line here:
https://github.com/BlueBrain/MorphIO/commit/cad7bdd24b33c306078d9a20fb9467566d086332
Could you try it, and see if it fixes this problem?

Parses into 1 soma and 3 sections!

I hope that's what you expected!

It is, thanks :)

When would this land in release? It's a blocker for me, if you can sneak it in in the next few days I would not have to fix a whole series of pesky bugs with my own SWC parser ;)

I'm super busy with other things at the moment.
There are a couple tests that fail, and I think it might be a semantic decision that needs to be made.  For that, I will need to discuss with others what the right decision is.

@Helveg you're input would be helpful: do you think the following is a valid (but small) SWC file:
1 1 0 0 0 1 -1
1 3 1 0 0 1  1

No, the sample identifiers (first column) need to be unique. Other than that it looks fine?

I'd say it's a design choice of MorphIO to allow for 0 length segments or not, but I don't see the harm. I'd find it needlessly restrictive for valid edge cases, and would leave these kinds of quality checks to dedicated morphology auditing tools that one can build on top of MorphIO

Nitpick: the sample identifiers (first column) need to be unique.

Yeah, that was a typo; sorry.

I'd say it's a design choice of MorphIO to allow for 0 length segments or not, but I don't see the harm in allowing it.

Okay, makes sense.
Currently, we're also transforming:
1 1 0 0 0 1 -1
2 3 1 0 0 1  1
3 3 1 1 0 1  2
4 3 1 0 1 1  2

Into:
1 1  0  0  0  1 -1
2 3  1  0  0  1  1
3 3  1  1  0  1  2
4 3  1  0  0  1  1
5 3  1  0  1  1  4

Which seems wrong: ie now the soma has two children instead of one.  Thoughts?

I don't understand? Why add a point?

I don't understand? Why add a point?

Yeah, that's my confusion too; I don't in the "new" parser, and that's what I have to work out.  Thanks for your input.

The new parser is live, which should solve this issue.
I'm going to close this, let me know if there are any problems.

Ok, I don't work on any projects that use this anymore, but I will ping a teammate that might be interested in knowing that we can try to switch to MorphIO now as our default parser :)


ltree demo adaptation- issues with path

hello, thanks for the repo and the example. I tried to adapt it in this way:
this is the 'employee' table from a django model:
created       | timestamp with time zone |           | not null |
modified      | timestamp with time zone |           | not null |
guid          | uuid                     |           | not null |
comp_user_id  | character varying(150)   |           | not null |
comp_username | character varying(150)   |           | not null |
comp_email    | character varying(254)   |           |          |
id            | integer                  |           | not null |
path          | ltree                    |           |          |
first_name    | character varying(150)   |           |          |
last_name     | character varying(150)   |           |          |
manager_id    | integer                  |           |          |
user_id       | integer                  |           |          |
Indexes:
    "corp_companyeuserprofile_pkey" PRIMARY KEY, btree (id)
    "corp_companyeuserprofile_user_id_key" UNIQUE CONSTRAINT, btree (user_id)
    "corp_companyeuserprofile_comp_email_6f7503d7" btree (comp_email)
    "corp_companyeuserprofile_comp_email_6f7503d7_like" btree (comp_email varchar_pattern_ops)
    "corp_companyeuserprofile_comp_user_id_328cb82e" btree (comp_user_id)
    "corp_companyeuserprofile_comp_user_id_328cb82e_like" btree (comp_user_id varchar_pattern_ops)
    "corp_companyeuserprofile_comp_username_9eec69b2" btree (comp_username)
    "corp_companyeuserprofile_comp_username_9eec69b2_like" btree (comp_username varchar_pattern_ops)
    "corp_companyeuserprofile_guid_e3160b25" btree (guid)
    "corp_companyeuserprofile_manager_id_2491a6e2" btree (manager_id)
    "cup_path_btree_idx" btree (path)
    "cup_path_gist_idx" gist (path)
Check constraints:
    "check_no_recursion" CHECK (index(path, id::text::ltree) = (nlevel(path) - 1))
Foreign-key constraints:
    "corp_comp_manager_id_2491a6e2_fk_cornersto" FOREIGN KEY (manager_id) REFERENCES corp_companyeuserprofile(id) DEFERRABLE INITIALLY DEFERRED
    "corp_comp_user_id_39765502_fk_users_use" FOREIGN KEY (user_id) REFERENCES users_user(id) DEFERRABLE INITIALLY DEFERRED
Referenced by:
    TABLE "corp_companyeuserprofile" CONSTRAINT "corp_comp_manager_id_2491a6e2_fk_cornersto" FOREIGN KEY (manager_id) REFERENCES corp_companyeuserprofile(id) DEFERRABLE INITIALLY DEFERRED
Triggers:
    cup_path_after_trg AFTER UPDATE ON corp_companyeuserprofile FOR EACH ROW WHEN (new.path IS DISTINCT FROM old.path) EXECUTE PROCEDURE _update_descendants_manager_path()
    cup_path_insert_trg BEFORE INSERT ON corp_companyeuserprofile FOR EACH ROW EXECUTE PROCEDURE _update_manager_path()
    cup_path_update_trg_two BEFORE UPDATE ON corp_companyeuserprofile FOR EACH ROW EXECUTE PROCEDURE _update_manager_path()

I also adapted the triggers:
-- function to calculate the path of any given manager
CREATE OR REPLACE FUNCTION _update_manager_path() RETURNS TRIGGER AS
$$
BEGIN
    IF NEW.manager_id IS NULL THEN
        NEW.path = NEW.id::text::ltree;
    ELSE
--           SELECT concat_ws('.', path::text, NEW.id::text)::ltree
          SELECT concat_ws('.', path::text, NEW.id::text)::ltree
          FROM comp_companyuserprofile
         WHERE NEW.manager_id IS NULL or id = NEW.manager_id
          INTO NEW.path;
    END IF;
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

-- function to update the path of the descendants of a c.u.p.
CREATE OR REPLACE FUNCTION _update_descendants_manager_path() RETURNS TRIGGER AS
$$
BEGIN
    UPDATE comp_companyuserprofile
       SET path = concat_ws('.', NEW.path::text, subpath(comp_companyuserprofile.path, nlevel(OLD.path))::text)::ltree
     WHERE comp_companyuserprofile.path <@ OLD.path AND id != NEW.id;
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

-- calculate the path every time we insert a new c.u.p.
DROP TRIGGER IF EXISTS cup_path_insert_trg ON comp_companyuserprofile;
CREATE TRIGGER cup_path_insert_trg
               BEFORE INSERT ON comp_companyuserprofile
               FOR EACH ROW
               EXECUTE PROCEDURE _update_manager_path();

-- calculate the path when updating the manager or the csod_user_id
DROP TRIGGER IF EXISTS cup_path_update_trg ON comp_companyuserprofile;
CREATE TRIGGER cup_path_update_trg
               BEFORE UPDATE ON comp_companyuserprofile
               FOR EACH ROW
               WHEN (OLD.manager_id IS DISTINCT FROM NEW.manager_id
                     OR OLD.csod_user_id IS DISTINCT FROM NEW.csod_user_id)
               EXECUTE PROCEDURE _update_descendants_manager_path();

-- if the path was updated, update the path of the descendants
DROP TRIGGER IF EXISTS cup_path_after_trg ON comp_companyuserprofile;
CREATE TRIGGER cup_path_after_trg
               AFTER UPDATE ON comp_companyuserprofile
               FOR EACH ROW
               WHEN (NEW.path IS DISTINCT FROM OLD.path)
               EXECUTE PROCEDURE _update_descendants_manager_path();

However path remains always empty after inserting or updating the entire row or the 'manager_id' column.
Am I doing something wrong?
I there a way to launch the update functions ath the end of the import process instead of implementing them as triggers?

Even if you insert a new row with no manager_id you don't have a path? That's weird…
And yes, you could use regular SQL functions instead of triggers, and call them when you need to. But only the triggers can ensure data integrity.

I do not know if this is the same issue but I also had empty paths when using get_or_create:
obj, created = Object.objects.get_or_create(slug=slug)  # obj is saved under the hood and path is set by triggers
obj.field = value
obj.save()  # obj is saved but we have an empty path in Django object, triggers are not run as slug is not modified,
#             path is overwritten by Django ORM

I fixed this by adding refresh_from_db:
obj, created = Object.objects.get_or_create(slug=slug)  # obj is saved under the hood and path is set by triggers
obj.refresh_from_db()  # read generated path from db into object
obj.field = value
obj.save()  # obj is saved but we have correct path now

Another (tested) option is to exclude path from saved fields:
fields = [f.name for f in Object._meta.get_fields() if f.name != 'path' and not f.auto_created and not f.primary_key]
obj.save(update_fields=fields)

This can be put in save() method for convenience.


Added ability for ShareFileClient.StartCopyAsync() to copy source file SMB attributes

API changes have been detected in Azure.Storage.Files.Shares. You can review API changes here

API changes have been detected in Azure.Storage.Files.Shares. You can review API changes here

API changes have been detected in Azure.Storage.Files.Shares. You can review API changes here


Header is fixed when scrolling inside a pane

This would tackle the feature discussed in #141 .
If the pane has a very olng content which needs to be scrolled, now the header is not scrolled and stays fixed, while the rest scrolls (both vertically and horizontally).
Updated the example "ol3.html".
https://rawgit.com/umbe1987/sidebar-v2/sidebar_scroll/examples/ol3.html

this is a breaking change, increases the filesize and might not be needed by everyone. it also only changes the ol3 implementation but none of the others. I'd prefer to keep the code simple and not include this. those users that want it should be able to copy it from this PR.
thanks for your contribution though 🙏

ok no problem. Thanks again for this project!

@Turbo87 Just wondering, if the change was made consistently across all implementations, are you open to it? Or do you think the current behaviour is what most people are looking for?
Just that I'm seeing the same problem as @umbe1987 - where you scroll to the end of a long tab only to find that the 'close' button has disappeared. It does seem to me that in most cases you'd want to fix the header, but maybe there's a usecase I'm missing?

I guess it might make sense as a default, but it might be easier to adjust the sidebar-header class to use something like position: absolute/fixed. would have to experiment with it to figure out what the best solution is, but I'd like to avoid the extra wrapper element.


Select an item through IGListCollectionContext

Changes in this pull request
We already had a API to deselect an item through IGListCollectionContext this is the support for selecting an item.
Checklist

[x] All tests pass. Demo project builds and runs.
[x] I added tests, an experiment, or detailed why my change isn't tested.
[x] I added an entry to the CHANGELOG.md for any breaking changes, enhancements, or bug fixes.
[x] I have reviewed the contributing guide

nice! 🎉


Tagged releases

Any plans to tag releases of the OpenCL headers on Github, as it is now done for OpenCL-CLHPP?
Thanks,

I would also very welcome this (as the Debian maintainer of this package)

I've now made the first release and closed https://github.com/KhronosGroup/OpenCL-Headers/issues/56. Please raise an issue if you need a release. We'll try to make regular releases.


Converted the request to an async function.

Hi @egdavid, I'n happy to merge this but can you please reopen this PR with dev as your base branch please? We don't merge directly into master unless we're releasing a new version.


Slash command sub command group missing required positional arguement

Summary
After trying to make a series of commands under a sub command group I ran into an issue regarding the group asking for positional arguments which it shouldn't require
Reproduction Steps
class ConfigCommands(commands.Cog):
def init(self, client: commands.Bot):
self.client = client
@commands.slash_command(description="Holds all of the fun commands inside")
async def config(self, inter):
    pass

@config.sub_command(description="Adds the verification role")
async def add_roles(
    self, interaction: disnake.GuildCommandInteraction, role: disnake.Role
):
    await interaction.response.defer()
    return await add_roles(self.client, interaction, role)

@config.sub_command(description="Removes the verification role")
async def remove_roles(self, interaction: disnake.GuildCommandInteraction):
    await interaction.response.defer()
    return await remove_roles(self.client, interaction)

@config.sub_command_group()
async def panel(self):
    pass

@panel.sub_command(description="Setup the verification panel")
async def set(
    self, interaction: disnake.GuildCommandInteraction, channel: disnake.TextChannel
):
    await interaction.response.defer()
    return await set_panel(self.client, interaction, channel)

@panel.sub_command(description="Remove the verification panel")
async def remove(self, interaction: disnake.GuildCommandInteraction):
    await interaction.response.defer()
    return await remove_panel(self.client, interaction)

def setup(client):
client.add_cog(ConfigCommands(client))
Minimal Reproducible Code
No response
Expected Results
The expected result is for it to not error when I run the commands under the panel sub command group.
Actual Results
Traceback (most recent call last):
  File "F:\Comissions\Monkee\lib\site-packages\disnake\client.py", line 515, in _run_event
    await coro(*args, **kwargs)
  File "F:\Discord Bots\Monkee\cogs\error_handler.py", line 50, in on_slash_command_error
    raise error
  File "F:\Comissions\Monkee\lib\site-packages\disnake\ext\commands\base_core.py", line 285, in invoke
    await self(inter, *args, **kwargs)
  File "F:\Comissions\Monkee\lib\site-packages\disnake\ext\commands\base_core.py", line 188, in __call__
    return await self.callback(self.cog, interaction, *args, **kwargs)
TypeError: ConfigCommands.panel() takes 1 positional argument but 2 were given

Intents
messages, guilds, members, reactions, bans
System Information

Python v3.10.0-final
disnake v2.3.0-beta

disnake pkg_resources: v2.3.0

aiohttp v3.7.4.post0

Checklist

[X] I have searched the open issues for duplicates.
[X] I have shown the entire traceback, if possible.
[X] I have removed my token from display, if visible.

Additional Context
After asking in the support discord I was told to make a bug report here.

async def panel(self) is missing 1 positional argument representing the interaction. async def panel(self, inter) should work. Let me know if this solves your problem.

this fixed it, thank you :)


Add more debug information for ANCM.

Some customer issues that people are commonly running into.

btw --- I'm offline for another hour. I'll get this reviewed as soon as I get back online.

similar to our existing platform common error 👉 https://docs.microsoft.com/aspnet/core/host-and-deploy/azure-iis-errors-reference?view=aspnetcore-2.2#platform-conflicts-with-rid.

I didn't want to add it there because this failure isn't Azure app service specific.
This is a common error people are running into and I would like to have it documented ASAP.

That topic covers both. The link is a little unfortunate, but it's more like "Azure/IIS Common Errors" ... no slashes in links.

If Scott can review it now, it can go in now. I'll be back online in an hour if not and we can merge this quickly.

@jkotalik Ok ... see how that looks.

LGTM

Cool
Rick assigned Scott, so he'll have to merge this.


Only render @p properties if has values

Hi,
I have the following console output when using an ExpressionTemplate
2022-03-25 12:47:18.271 INFO:  {}
2022-03-25 12:47:20.725 INFO: TEST3 {"methodName":"RunCommandLineOptionsAndReturnExitCode","className":"ABC"}
as you can see from the first line that it is printing the empty {} where I have no properties.
What expression do I need to check if the @p is empty/null/zero length so as to only render if there are properties?
My expression
"{@t:yyyy-MM-dd HH:mm:ss.fff} {if @l = 'Verbose' then 'TRACE' else if @l = 'Fatal' then 'Fatal' else if @l = 'Error' then 'ERROR'  else if @l = 'Debug' then 'DEBUG' else if @l = 'Information' then 'INFO' else if @l = 'Warning' then 'WARN' else ''}: {@m} {if @p is not null then {..@p} else ''} {@x}\r\n"
i.e.  {if @p is not null then {..@p} else ''}
Also tried

{#if Length({@p}) > 0} {@p} {#else} {#end}
{#if Length(@p) > 0} {@p} {#else} {#end}

Hi! This is currently a bit trickier than it should be - the following should work:
{#if Length(Keys(@p)) > 0} {@p} {#else} {#end}

{#if IsDefined(@p[?])} {@p} {#else} {#end}

HTH!

Thank you very much @nblumhardt.
The second one worked {#if IsDefined(@p[?])} {@p} {#else} {#end}
However, the first one errored with The function name "Keys" was not recognized.


[Feature]: Use automation to update Docker images.

Suggestion Description
Docker hub does not have latest version of ROCm images, I recommend using some sort of automation to build new images and publish in docker hub and in github.
Update the base image to 24.04.
Operating System
No response
GPU
No response
ROCm Component
No response

Hi @rudiservo, you can find the latest Ubuntu 24.04 ROCm docker image at https://hub.docker.com/r/rocm/dev-ubuntu-24.04. Is there a specific image you are referring to?

Hi @harkgill-amd, I think that at the time 24.04 didn't have 6.2.1 the latest version is 6.2.2, a minor change so I don't know how relevant the update is.

We do now have automation in place to build and publish new images to docker hub in the form of a GitHub action. The automation is triggered by a PR, which was just done for ROCm 6.2.2. You should see the new image up shortly.


[Documentation] - Highlight that you need python3-catkin-tools

rosvenv uses catkin build, so I would highlight in the readme that for it to work you need to have previously installed
`sudo apt-get install python3-catkin-tools'

@vonHartz Can you investiagte?

Addressed in fa08298e5c9092779e3f019dca6b787a38d4b9cc


Upgrade to PyCryptodome?

I recently started using the RNCryptor libraries in a few different languages. Thanks for all the great work and documentation.
I was wondering if there was any plans to upgrade the python library to use PyCryptodome and replace PyCrypto? It seems that PyCrypto is unmaintained (issue & blog discussing) and PyCryptodome is a (mostly) drop-in replacement for of the project.
If not, would the project be open to a PR from me to upgrade the library?
Thanks again.

Hi Robert,
I have no such plans, but I will be glad to review and merge your PR.

bump

Hi @rfadams, are you going to do the upgrade?

Got a deprecation warning:
venv/lib/python3.7/site-packages/Crypto/Random/_UserFriendlyRNG.py:77: DeprecationWarning: time.clock has been deprecated in Python 3.3 and will be removed from Python 3.8: use time.perf_counter or time.process_time instead
t = time.clock()
Whether by switching to PyCryptodome or even to cryptography, please, remove pycrypto dependency before Python 3.8 release.
PyCryptodome is a maintained fork of pycrypto and exposes almost the same API as the old PyCrypto so that most applications could run unmodified. Compatibility with pycrypto is described here:
https://pycryptodome.readthedocs.io/en/latest/src/vs_pycrypto.html

Please review PR #9 and advise... thanks!

Thanks.
It's much needed but too late for me - I've already switched to
cryptography package (https://github.com/pyca/cryptography).
On Tue, Jul 2, 2019 at 9:20 PM Carlos<EMAIL_ADDRESS>wrote:

Please review PR #9 https://github.com/RNCryptor/RNCryptor-python/pull/9
and advise... thanks!
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/RNCryptor/RNCryptor-python/issues/7?email_source=notifications&email_token=AL7SD52LE3JZ366ENFKD7RDP5QSKJA5CNFSM4GXYVZ52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZDG2XQ#issuecomment-507931998,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AL7SD52NFLCWWURIPBBUDN3P5QSKJANCNFSM4GXYVZ5Q
.

Hi,
Could you review and merge the previous PR and make a new release please ?

$ pip install rncryptor==3.3.0
Collecting rncryptor==3.3.0
...
Installing collected packages: pycryptodome, rncryptor
Successfully installed pycryptodome-3.8.2 rncryptor-3.3.0

Thank you, @furritos!


TypeError: 'module' object is not callable

Traceback (most recent call last):
File ".\Evaluation.py", line 32, in
evaluate()
File ".\Evaluation.py", line 10, in evaluate
dataProvider = DataProvider()
TypeError: 'module' object is not callable
I run python Evaluation.py and then got this error. Can someone help?

I'm also getting the same issue. can anyone help us in this?

I'm also getting the same issue. can anyone help us in this?

Just changed the below lines
dataProvider = DataProvider()
images = dataProvider.get_testset(True)
to single line as,
images = DataProvider.get_testset(True)
it works for me....

Just changed the below lines
dataProvider = DataProvider()
images = dataProvider.get_testset(True)
to single line as,
images = DataProvider.get_testset(True)
it works for me....


1min and 5min datalogger files

@roylrich Could you remind me what the deal is the the "1min" and "5min" datalogger files? We are going to process them through the pipeline, right?
E.g. PNNL_31_sapflow_1min_20210816000006.dat
Thanks

@roylrich Could you remind me what the deal is the the "1min" and "5min" datalogger files? We are going to process them through the pipeline, right?
E.g. PNNL_31_sapflow_1min_20210816000006.dat
Thanks

These were setup for events, I don't know if anyone uses them? Could we process for maybe 1 month around tempest evenst?

If they're valid data, all good, let's process! No reason not to.


Roll Engine from 3c1d96f0b86f to 570e51cf47c1 (6 revisions)

https://github.com/flutter/engine/compare/3c1d96f0b86f...570e51cf47c1
2021-05-15<EMAIL_ADDRESS>Roll Fuchsia Mac SDK from 9-tG_VdU_... to PVI875Hyp... (flutter/engine#26174)
2021-05-15<EMAIL_ADDRESS>Roll Fuchsia Linux SDK from RT1RbBM69... to q_v1AFsEq... (flutter/engine#26173)
2021-05-15<EMAIL_ADDRESS>Reland "Fix a11y placeholder on desktop; auto-enable engine semantics" (flutter/engine#26134)
2021-05-15<EMAIL_ADDRESS>Roll Skia from df2dbad1a87d to bc8e0d8ba0d4 (1 revision) (flutter/engine#26171)
2021-05-15<EMAIL_ADDRESS>Roll Fuchsia Mac SDK from a6aB7cNgl... to 9-tG_VdU_... (flutter/engine#26169)
2021-05-15<EMAIL_ADDRESS>Roll Skia from aecf8d517143 to df2dbad1a87d (9 revisions) (flutter/engine#26168)
Also rolling transitive DEPS:
fuchsia/sdk/core/linux-amd64 from RT1RbBM69e3n to q_v1AFsEqEWh
fuchsia/sdk/core/mac-amd64 from a6aB7cNgl94H to PVI875Hypv4Y
If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-engine-flutter-autoroll
Please CC<EMAIL_ADDRESS>on the revert to ensure that a human
is aware of the problem.
To report a problem with the AutoRoller itself, please file a bug:
https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/master/autoroll/README.md

Commit queue failed; closing this roll.


Multi Tenant - ApplicationDbContext on BackgroundJob, Worker or isolated process: Example HangFire

The use of the 'contextdb' within an isolated process presents conflicts since the whole definition depends on the httpContext for some internal processes such as audit or multiple tenant (ITenantService tenantService, ICurrentUser currentUserService)
https://github.com/fullstackhero/dotnet-webapi-boilerplate/blob/ef53f36f419a2817c57381719cf77aaafc133f15/src/Infrastructure/Persistence/ApplicationDbContext.cs#L21
The dependency of the httpContext is generated in these processes
https://github.com/fullstackhero/dotnet-webapi-boilerplate/blob/ef53f36f419a2817c57381719cf77aaafc133f15/src/Infrastructure/Services/General/TenantService.cs#L43
https://github.com/fullstackhero/dotnet-webapi-boilerplate/blob/bc433a04013c1b3580f937d7331b4f6fd779b582/src/Infrastructure/Identity/Services/CurrentUser.cs#L14
This step is a great limitation to be able to create isolated or background processes for secondary tasks or Jobs with Hangfire that require an injection of 'ApplicationDbContext' or any of the existing repositories.
I consider that the process of defining tenantId and CurrentUserId should occur in the flow from the application so the services, dbcontext and other dependencies are isolated from the httpcontext layer
PS: in the case of HangFire, an extra solution must be created since achieving tenure from this point requires additional steps such as those mentioned here:
https://stackoverflow.com/questions/57394712/hangfire-multi-tenant-asp-net-core-resolving-the-correct-tenant
It is to clarify that I do not mean to create a database for each hangfire, but to use the dbcontext within it for background tasks which for obvious reasons cannot carry the HTTP context.
https://docs.hangfire.io/en/latest/background-methods/passing-dependencies.html
Example Basic:
BrandGeneratorTask.cs
`public class BrandGeneratorTask {
    private readonly IRepositoryAsync _repository;
    public BrandGeneratorTask(IRepositoryAsync repository)
    {
        _repository = repository;
    }

    [Queue("notdefault")]
    [DisplayName("Generate Random Brand example job on Queue notDefault")]
    public void Execute()
    {
        foreach (int item in Enumerable.Range(0, 100))
        {
            await _repository.CreateAsync<Brand>(new Brand(name: $"Brand Random - {Guid.NewGuid()}", "Funny description"));
        }
        await _repository.SaveChangesAsync();
    }
}`

Call example(representation of the generic call process from the jobservice):
BackgroundJob.Enqueue<BrandGeneratorTask>(x => x.Execute());
It is finally that the process of sending arguments to Hangfire should not be of type ref or out
https://docs.hangfire.io/en/latest/background-methods/passing-arguments.html

#141

#166 Hangfire: Multi tenant resolving the correct(ApplicationDbContext and Services) for tenant on BackgroundJob
@mohitook
Based on your main need to solve the dbcontext for each tenant through dependency injection, I have created this solution that I would like you to test


Add argument checks and tests for BQ StorageAPI sinks.

Some argument checks are added to ensure the options specified by users are valid.

triggeringFrequency is only relevant when using STORAGE_WRITE_API (or FILE_LOADS) on an unbounded data source.
numFileShards is only relevant when using FILE_LOADS on an unbounded data source. It will be ignored if autoSharding is enabled.
numStorageWriteApiStreams is only relevant when using STORAGE_WRITE_API on an unbounded data source. It will be ignored if autoSharding is enabled.
autoSharding is only relevant on unbounded data source.

In addition, integration tests on a small-scale unbounded data source have been added. In these tests, we do fuzz testing on the schema order by shuffling fields in the BigQuery write schema. We also trigger an update to the table schema while data streaming is ongoing, to simulate a potential data-loss problem due to BigQuery schema change.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

Run Java_GCP_IO_Direct PreCommit

It looks like this is already getting a review, so I'm going to bow out (I've been on vacation for most of June, hence no actions.)
But if you need a 2nd look or a committer for merging, please feel free to ping me again.

R: @ahmedabu98


Added secrets yaml and documentation

For some reason this has been missing in the official microsoft repo. This will help lot of new people trying out without much knowledge of kubernetes

@microsoft-github-policy-service agree [company="default"]

@microsoft-github-policy-service agree

Hello? Is this project being maintained?

@KR-Ravindra It's maintained, but the MS employees aren't the most responsive on this repo.  Mostly community responses.  Would be nice to see your changes accepted though.

Someone please accept/merge changes as I have to create mssql secrets manaually

Any merge possibilities on this?
Multiple issues are open for this feature

Any progress -_-; I think this project is already dead!


View documents marked as dirty on open

Open a view document.
View opens but has the * in the top tab to indicate it is dirty

Also happening on Rules documents


WIP: Application review with Maxim

Notes:
MAXIM call about reviewing usage

[x] Missed the discord link
[x] Easy to set up via docker * provide link
[ ] Add section for : Dev lake leverages, docker, grafana. All third party words as links to their homepages
[x] Requirements title should be changed to “Required packages to install”
[x] “Data source” plugins to “Data sources we currently support”
[x] We need a preview at the top for what the final thing looks like
[ ] We need a video screencast
[ ] Dummy proof installation of docker and node.js
[ ] #212

Easy to set up via docker * provide link

Going through the motions, I found this to interrupt the flow by taking me into the overwhelming world of docker too early.
There are 2 cases:

Person knows docker -> no action needed
Person doesn't know docker -> We hold their hand as much as possible by giving them specific commands so they don't need to know too much about docker to get it going.

That's my proposal. Let's just give them minimal info to get going.

This has been completed and the feedback was really helpful


pull request for e2e test at 01/14/2017 14:15:15

:white_check_mark: Validation status: passed
For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Change Main IP of Postal

Hi,
My VPS has multiple IPs and it was preconfigured into the server. When I installed Postal, to my surprise, it picked up a rotating IP as the main IP. I want to change that to main IP. I do not have any other issue with VPS.
Any idea would he helpful.
Thank you.

You should make use of IP Pools to limit which outgoing IP addresses are used by your Postal installation.
You can see more here https://github.com/postalhq/postal/wiki/Configuring-IP-Pools

Yeah, I figured it out. Thanks for your support.


[common] Turn initContainers into a dict instead of a list

Details
Currently, the initContainers key is a list of dicts.
This could turn in to an issue when there are merges being done from library, add-on, chart and user values since merging lists in Helm is not fun and often leads to things being unintentionally overwritten.
Potentially affects:

[ ] This will break a few of our charts that rely on the initContainers functionality
[ ] Documentation needs to be checked

Fixed with the release of common v4.


Trial Jax Template API and retire custom loader

As a Developer, I would like to maintain the least amount of code and retire deprecated code.
Jax has developed an API for the PDX Finder template. The api should be scrapped and then loaded in the PDX Finder to test for inconsistency and mistakes in the data. When the data is validated and loaded the previous Jax pipeline can be retired.
ACCEPTANCE CRITERIA

[x] metadata and molecular data validated
[x] Jax loader retired

moving to the next sprint

related ticket: https://github.com/PDXFinder/pdxfinder/issues/190

From the most recent communications between Steve and I:
I have updated the APIs.

Renamed endpoints.

Added new lines after column headers.

Deduplicated all data (I hope).

The missing patients are in the current patient endpoint (not sure what went wrong there I didn't change anything).

Checked a smattering of the models w/o samples they are models that don’t have sample data ie.
         http://tumor.informatics.jax.org/mtbwi/pdxDetails.do?modelID=J000106949

         Has nothing in the Molecular Data section.

TODO: Need to investigate the Patients w/o samples. I expect they correspond to models that have been discontinued, if so I will try to remove them from the patient API.
I will need to re-test these again and send feedback back to Steve.

There is a brief bug to the code base I will need to write for the API to work. Steve is in development of the Omic API's

moving to the next sprint

@Afollet to forward email communication to @CsabaHalmagyi

@Afollet to forward email communication to @CsabaHalmagyi

Asked Steve to clean up some of the tumor types in the sample sheet. The data loads fine, the indexer picked up 38 unmapped diagnosis combinations in the dataset.

Asked Steve to clean up some of the tumor types in the sample sheet. The data loads fine, the indexer picked up 38 unmapped diagnosis combinations in the dataset.

WIll need to do 38 mappings once the branch is merged - @CsabaHalmagyi to let @zperova know when (to make a new ticket)
Moving to the next Sprint

WIll need to do 38 mappings once the branch is merged - @CsabaHalmagyi to let @zperova know when (to make a new ticket)
Moving to the next Sprint

Removed custom JAX loader and supporting services

Jax is now loaded with UPDOG, the custom loader was removed.

A call for a celebration! Thanks for all the work.


Push command trashes remote files if they don't exist locally

If you run the upload command and provide a wrong path, for example:
drive upload --no-prompt pathh/ path/
and there actually is a folder called path/ on Google Drive it will be trashed. I understand now that "upload" is really more like "remote sync", meaning that it "uploads or deletes" remote files to create a mirror of the local folder structure. But that's not at all communicated clearly in the (gigantic ;) ) README. In my opinion such destructive behavior should be very well stated, for example with a note in the README:
"NOTE: Upload will trash contents on Google Drive if not found locally."
Ideally, it is in the best interest of new users to have a non-destructive push option by default. But I understand this project is governed in a rather authoritarian way and can't expect such user-friendly changes. Still, thank you for making it trash content and not outright delete.

Hello there @dimitrovs, thank you for the issue and welcome to drive.
So for starters, just for clarity, drive upload doesn't exist.
Good points, usually we assume that the user knows that mirroring will delete content that doesn't exist. For sure, we can make it clearer the way we did for pull https://github.com/odeke-em/drive#pulling

But I understand this project is governed in a rather authoritarian way and can't expect such user-friendly changes.
In my honest opinion, the intention isn't to run the project in a rather authoritarian way. It is just
that am very busy with my work at a startup, school and also career advancement, that I might
not respond on time, and this project is done in my free volunteer time, but otherwise I and the other contributors very much welcome any change that makes the project better.
Please feel free to send a pull request, or let me know here, and anyone can send a fix.
Thank you for advocating for the change.

In my honest opinion, the intention isn't to run the project in a rather authoritarian way.

I really don't think so either, you're always open to new suggestions and we really appreciate what you're doing for drive! (useless regarding this issue but it needed to be said)

Thank you @chevillet, I appreciate it.


Use new input capture functions

Fixes a build with disabled obsolete functions with latest Dear ImGui version (see commit).
Also fixed MSVC performance warning (memcmp line).

Thank you!


How to run complexFilter with dynaudnorm and mergeToFile?

Version information

fluent-ffmpeg version: 2.1.2
ffmpeg version: 3.3.1
OS: Mac OS 10.12.4

Code to reproduce
let mergeVideos = async (videoFilenames, outputPath) => {

    return new Promise((resolve, reject) => {
        let video = ffmpeg();

        video.on('start', function (commandLine) {
            console.log('Spawned Ffmpeg with command: ' + commandLine);
        });

        // format: input format
        // duration: input duration
        // audio: audio codec
        // audio_details: audio encoding details
        // video: video codec
        // video_details: video encoding details
        video.on('codecData', function (data) {
            console.log('Input is ' + data.audio + ' audio ' + 'with ' + data.video + ' video');
        });

        // add all videos as input
        videoFilenames.forEach(filename => {
            video.addInput(filename);
        });

        video.complexFilter([
            {
                filter: 'dynaudnorm',
                inputs: '1'
            }
        ]);

        // *Dynamic Audio Normalizer*
        //
        // This filter applies a certain amount of gain to the input audio in order to bring its peak magnitude to a
        // target level (e.g. 0 dBFS). However, in contrast to more "simple" normalization algorithms, the Dynamic Audio
        // Normalizer *dynamically* re-adjusts the gain factor to the input audio. This allows for applying extra gain
        // to the "quiet" sections of the audio while avoiding distortions or clipping the "loud" sections. In other words:
        // The Dynamic Audio Normalizer will "even out" the volume of quiet and loud sections, in the sense that the volume
        // of each section is brought to the same target level. Note, however, that the Dynamic Audio Normalizer achieves
        // this goal *without* applying "dynamic range compressing". It will retain 100% of the dynamic range *within* each
        // section of the audio file.
        //
        // @see {@link https://ffmpeg.org/ffmpeg-filters.html#dynaudnorm}
        // video.audioFilter('dynaudnorm');

        video.mergeToFile(outputPath, './tmp/')
            .on('progress', function (progress) {
                console.log(`processing ${progress.timemark}, ${progress.percent.toFixed(2)}%`);
            })
            .on('error', function (err) {
                reject(err.message);
            })
            .on('end', function () {
                resolve(outputPath);
            });
    });
};

Expected results
I'd like to merge two videos and run the dynaudnorm filter on the second video. I tried to add the dynaudnorm filter through audioFilters, but it throws an exception that -af and filter_complex cannot be used together. How can I merge the two videos and apply the dynaudnorm filter in one complex filter pipeline using fluent-ffmpeg?

Closing, off-topic


UNION ALL - performance question

Hello guys!
It's seems that I have to combine two data set, both of which are gotten by executing SELECT statement. I've just seen that ClickHouse can do UNION ALL and it's can be appropriate for me. But I'm not sure about its performance in general case. Would you recommend to use UNION ALL statement or it was added for the functional completeness? Yes I know it depends on a specific case but as the database developer you may know all pros and cons of using this clause. May be there are advice when it can be appropriate to use UNION ALL statement.
Fix me if I'm wrong. A query with UNION ALL forces the database node which received this query aggregates data sets in the memory. So it can be cause memory overuse on a database nodes. Also it can add unnecessary computation especially when you union data sets with different columns. Finally your logic of data union can be change in the future. The main advantage UNION ALL is possibility to process data by one round trip to the database.
Personally I decided not to use UNION ALL because my logic of data union can be changed. But anyway I'm very interesting about your recommendation.

I never looked in code how exactly it works, but I'm using UNION ALL a lot and it just works - fast and without any problems. The way how clickhouse store data in ONE table (multiple parts which should be read at the same time) is quite similar to the case when you need to read multiple tables.
The only problem for me with UNION ALL - is that you need to pass WHERE conditions to both SELECTs separately.
Small performance check on my data:
:) select count() from tableA  where date>today() - 30;
┌───count()─┐
│ 302539708 │
└───────────┘

1 rows in set. Elapsed: 0.109 sec. Processed 332.76 million rows, 665.52 MB (3.04 billion rows/s., 6.08 GB/s.)

:) select count() from tableB where date>today() - 30;
┌───count()─┐
│ 930953700 │
└───────────┘

1 rows in set. Elapsed: 0.309 sec. Processed 969.66 million rows, 1.94 GB (3.13 billion rows/s., 6.27 GB/s.)

:) select count() from ( select date FROM tableA where date>today() - 30 UNION ALL select date FROM tableB where date>today() - 30 );
┌────count()─┐
│<PHONE_NUMBER> │
└────────────┘

1 rows in set. Elapsed: 0.450 sec. Processed 1.30 billion rows, 2.60 GB (2.89 billion rows/s., 5.78 GB/s.)

@filimonov Thank you. Nice to hear that it works fine.
P.s. To the developers: If there are nothing to add, this issue can be closed.

As far as I know, most people are satisfied with UNION ALL performance of ClickHouse.


Using TIME in shader decreases editor FPS

Godot v3.1.beta.custom_build.9f57939
Linux no GPU (Intel i7 with integrated gpu)
Using TIME in shader decrease performance! See video
I make a shader, just add TIME to fragment expression

As discussed on IRC with @mrcdk and @JFonS, using TIME does not actually significantly impact the performance of the game.
What you see here is the editor FPS, and using TIME causes the editor to redraw at each frame to take this into account while previewing the shader.
I did a small test locally, and I have the same running performance on a simple project whether I use TIME or not (surprisingly even better when using TIME...).
Shader used on a simple CubeMesh:
shader_type spatial;

void fragment() {
        ALBEDO = vec3(abs(sin(TIME)), abs(cos(TIME)), 1.0);
}

I'll add a tooltip to the Label to make this clear.

I don't get it honestly... The time shown should be the time it takes for a frame to be drawn (that is, the value doesn't care how frequently the editor draws a frame).
If it shows something else, even if editor-only, it's probably broken currently??

@starry-abyss Well, how do you measure the time it takes to draw a frame when the whole editor is a single process, a single window, and thus a single frame?

@akien-mga OK. But if TIME really forces more frequent updates, I think then the frame amount without TIME is calculated incorrectly.
I mean, the editor knows on which of the frames updates happen (because it triggers those), so it should be possible to use this actual number for FPS meter instead of the big number.
Well, that would probably mean something like FPS: 1 (1000 ms), but at least less confusing and should hint that low-processor mode should be switched off (I assume) or something?

I can confirm this. my laptop ramp-up the fans when I use TIME in shader, and I think this is related to #24705 (Playing AnimationTree causes performance drop in the editor)
Ps: I want to re-open this issue


Ability to pass database password via docker secret file

Docker secrets allow passing sensitive information via in memory secret files in /run/secrets/my_secret. Current kong expects db passwords via env variables like KONG_PG_PASSWORD which can't use docker secrets for  secure usage.
It would be good if kong docker image supports passing db passwords via file. eg: KONG_PG_PASSWORD_FILE=/run/secrets/kong_pg_password

Any news about this?
meanwhile I forked and added a workaround:
https://github.com/jalberto/docker-kong/tree/env-secrets

Only just noticed this issue, because it is on the docker repo, should probably have been on kong itself.
@endeepak @jalberto as a workaround you could use
export KONG_PG_PASSWORD=$(</run/secrets/kong_pg_password)
kong start

other than that, the change to Kong would be fairly trivial, just modify the config parser https://github.com/Mashape/kong/blob/master/kong/conf_loader.lua
So if someone can supply a patch that would be nice ...

my LUA skill is quite limited, here is what would be needed:

Add KONG_PG_PASSWORD_FILE (probably not the only one needed?)
if KONG_PG_PASSWORD_FILE exist read it and set pg_password to it's content

Doubts:

config precedence ?
bad implications of reading from a file?
how handle special chars?

Yes, also needed for other secrets like Cassandra password, etc.

config precedence ?

is handled by the config loader. Only needs some logic to read either the regular password, or the file based on, but never both (should be an error)

bad implications of reading from a file?

This is being dealt with in the init phase, so not such a pressing issue

how handle special chars?

Not sure I understand, read the full contents of the file, and that would be the password. No interpretation necessary, right?

how handle special chars?
Not sure I understand, read the full contents of the file, and that would be the password. No interpretation necessary, right?

agree, I was thinking more about encoding problems

It is already handled, realize the docker-compose file share here was using docker secret, https://github.com/Kong/docker-kong/tree/master/compose

This has existed in our unofficial docker images as of https://github.com/Kong/docker-kong/pull/348. The unofficial images are available at https://hub.docker.com/repository/docker/kong/kong/tags?page=1 but it's advisable to not use those outside of test / development environments.
The official docker images will have this as of 2.0.4 and I'll probably backport the functionality a few version as well.


Hint that uglify-js can do multiple passes

Quick Summary: Talk about the ,passes={n} addition to uglify-js.
SSCCE

Elm: ???
Browser: ???
Operating System: ???

Additional Details
???

I think it would be good if people knew to choose a higher passes if they have the time, but I would not want people to think that passes=1 is something special and be afraid to change it. Perhaps people would see that there and think that we did some testing to say that this is a good number to use in particular.
So I think the less discoverable approach has the benefit of only emphasizing details that are extremely important to getting good minification. I'm leaning towards that being most important.


Update api branch with master changes

who added vet!!! ;)
agh

@LK4D4 added it :)

I updated all the sample configs

cool, build failures?

I'm working on fixing the vet errors

Fixed it by removing vet

LGTM


NotificationLayout.MediaPlayer about actionButtons

NotificationLayout.MediaPlayer
if actionButton without  icon even if   it has label ,the actionButton won't show.
How can i use actionButton without icon?

Sorry, but i gonna ask you to repost your issue, not erasing the form content. Also, try to use the form for Ask a Question, instead of bug reports.
I gonna close this issue and mark it as invalid.


Recency guarantees through signed pings

SSB is an eventually-consistent system on an asynchronous network. Asynchronous means there's no time-bound on delivery: the message can arrive on the other node at any time. Since the wall clock isn't consistent globally, there's no way to know how old the information is. That missing guarantee is called recency.
To provide a recency guarantee, we need a synchronous channel. This is trivial on a local network. On public net, a sync channel can be arranged through proxies (eg pub relays). Once it's established, users can send requests and measure the latency, giving them an upper-bound on how old the response is. This is the recency.
A "signed ping" is a request for the current seq number of a user's feed over the synchronous channel. With the resulting response time, the caller knows roughly how old the feed is. The ping response is signed to prove authorship, and response seq numbers prevent replay attacks. Signed pings can be cached, but they'd have relatively low TTLs (depending on the recency-guarantee needed).
Signed pings only work if both users are reachable. They can be used as heartbeats to provide fault detection. Because the heartbeat includes latest feed seqs, the recency guarantee transfers to the feed's messages, and they become effectively synchronous. Groups of nodes can then implement leader election (eg Raft) using SSB.

a nonce would be better than a ping seq

Why not have a "ping feed"?

@jtremback as in post ping messages to your ssb feed?

Have a separate feed just for pings?

ah, because you need the synchronous request/response flow to determine recency

Ah, it might be simpler to implement this as ephemeral messages (which have been sitting on the drawing board for a while) you'd just have an ephemeral message every day/hour whatever and if you have the message within that time, then you know you must be up to date.
Even if you do not post for weeks, as long as you are online, you'd post ephemeral heartbeats automatically, and receiving those would prove that the reason you havn't seen a post is because it wasn't created, not because of a network partition. At the same time, because of the way that ephemeral messages are discarded after a ttl these would not clutter the feed.
Question: have we had a discussion about applications that need strong recency support to be viable?

how will ttl work for ephemeral messages without a global clock? the premise for signed ping requests is that they use the latency of the sync request/response flow to measure recency without requiring a synced clock. would ephemeral messages have something similar?
the largest strong-recency conversation revolved around proto-money/caps. the idea was that peers may want to distribute information that has time-sensitivity. to deal with that, time-sensitive data could have recency constraints attached to them (which is different than a ttl)
an example with perms:
9pm dominic says paul can use the bike (recency constraint: 5 minutes)
10pm dominic says paul cant use the bike anymore

so juan has the bike right now, and he sees the 9pm message. at 10:08pm i show up and ask for the bike, and juan's copy of the dominic feed is from 9:58pm (10 minutes old) so he has to ask for latest. either he...

gets the latest and sees that, no, i cant have the bike
cant get latest and says no, because the recency constraint cant be violated

so that's one example. i believe it could be possible to implement leader-based distributed transactions on feeds using the Raft protocol, but i'd need to research that

Oh right, now I remember - hmm - oh right so we add acks to the heatbeats and then we have signed pings.