[1.16.5] Server hang occultism storage

Describe the bug
The server freezes and does not respond to commands
To Reproduce
I cannot reproduce, it is chaotic
System (please complete the following information):

Occultism Version: 1.16.5-1.20.0
OS: ubuntu
Minecraft Version: 1.16.5

Additional context
Thread dump https://pastebin.com/X2rWsusP

@bigenergy thanks for the report - do you have any idea what might be connected to it?
Based on the crash log it happens when a storage remote tries to access the storage system, but that alone should not cause an issue

@bigenergy thanks for the report - do you have any idea what might be connected to it? Based on the crash log it happens when a storage remote tries to access the storage system, but that alone should not cause an issue

perhaps this happens if the chunk in which the occultism storage is located is not loaded, but this is just an assumption

or maybe you should add a check for existence tile in the world

The crash happens (as far as I can see from the log) when it tries to find the dimension the storage controller is in within the dimension registry - that is before the tile is accessed.
The only causes I can think of are either a storage remote item's nbt being modified / created via commands, or a dimension no longer existing after a modpack update, but both I suspect there must be another edge case that is more random.
I will add sanity checks for this crash, but I suspect whatever causes it will cause other issues as well

Looking into this further, I don't think this can be fixed on my end.
The thread gets stuck on Forge's "ResourceKey"/"RegistryKey" class, on String#intern():
   private static <T> ResourceKey<T> create(ResourceLocation pRegistryName, ResourceLocation pLocation) {
      String s = (pRegistryName + ":" + pLocation).intern();
      return (ResourceKey<T>)VALUES.computeIfAbsent(s, (p_135796_) -> {
         return new ResourceKey(pRegistryName, pLocation);
      });
   }

As long as a storage remote is linked by the mod (and not manually), this should work. Even if someone manually places random NBT that should not influence Forge's registry handling, it should just not found anything.
I'm closing this for now, I can further investigate if we find a way to reproduce it reliably, then I can debug it. At that point please ping me so I can reopen


improve static assets perf & memory usage, refactor

Can you branch off the master? The Merge remote-tracking branch 'brunch/master' warnings are unhelpful.

This is still relevant.

See the comment from 16 days ago.


Create KotoolsNumber type

Description
Create the Number type in the kotools.types.number package.
The type should be available for all platforms.
Dependencies

[x] #98

Checklist

[x] Create the type with its constructor.
[x] Implement the builders of this type.
[x] Update Work in progress section in changelog.
[ ] Rename the Number type to KotoolsNumber for convenience.

Resources

Documentation of kotlin.Number.

Done.

Rename the Number type to KotoolsNumber for convenience.

Don't include this issue in the version 3.0.0. We will focus on the Int hierarchy instead.


monetdb: 11.43.9 -> 11.43.15

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://repology.org/metapackage/monetdb/versions.
meta.description for monetdb is: "An open source database system"
meta.homepage for monetdb is: "https://www.monetdb.org/"
meta.changelog for monetdb is: ""
Updates performed

Version update

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found 11.43.15 with grep in /nix/store/rcv4v8ayh62lbskc9zkb8x75byy39x7n-monetdb-11.43.15
found 11.43.15 in filename of file in /nix/store/rcv4v8ayh62lbskc9zkb8x75byy39x7n-monetdb-11.43.15

Rebuild report (if merged into master) (click to expand)

1 total rebuild path(s)

1 package rebuild(s)

First fifty rebuilds by attrpath
monetdb

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/rcv4v8ayh62lbskc9zkb8x75byy39x7n-monetdb-11.43.15 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A monetdb https://github.com/r-ryantm/nixpkgs/archive/92521d9c1f3005b27a7802427822be4ef7e1e1e3.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/rcv4v8ayh62lbskc9zkb8x75byy39x7n-monetdb-11.43.15
ls -la /nix/store/rcv4v8ayh62lbskc9zkb8x75byy39x7n-monetdb-11.43.15/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

monetdb

Maintainer pings
cc @StillerHarpo for testing.

Some binaries (for example mktest.py) are broken


large error_log file inside api folder

Hi,
I've noticed the error_log file inside the "api" folder is becoming larger and larger. After opening it it's pointing to this line of code and it's repeated thousands of times inside the file:
PHP Notice:  Undefined index: stealthfound in /home/xxxxx/public_html/xxxxxxxxxxxx.com/UnamWebPanel/api/endpoint.php on line 18
Line 18 inside endpoint.php is:
return htmlspecialchars($data[$key]) ?? '';
I'm not skilled enough to know what's causing this, can someone help please?
Thanks

You can update your web panel here: https://github.com/UnamSanctam/UnamWebPanel/releases, that's an older version of the XSS protection panel.


Feature/wd3734

Pull Request Considerations
Although some of these things can only be done after merged, please ensure all items have been considered when doing pull requests.

Updated README.md(s) for new functionality or changes.
Have you added an env file?

Did you update Openshift secrets/config maps in repository?
Did you added this env variable explanation in the appropriate README.md file.

Have you changed any Openshift configurations/templates?

Have you applied changes.

Did changes trigger Openshift img re builds.

Check Openshift instance of application to ensure your changes are working.

CONTEXT:
Adding tests onto wd-3734. 86.3% test coverage for the backend. Still need to add a test to check "verifyToken" but leaving it for now as it requires more research and dependencies.

approving this due to time, and we can add openshift configs after other changes have been merged.


Update and Enhance DESCRIBE.md Documentation

PR Type:
Documentation

PR Description:
This PR updates the DESCRIBE.md documentation file. It restructures the 'Markers template' section, provides an example of how to use description markers, and adds visual aids to illustrate the before and after effects of using markers. The 'Configuration params' subsection is also updated to include the 'use_description_markers' parameter.

PR Main Files Walkthrough:
 files:
docs/DESCRIBE.md: The 'Markers template' section has been restructured and enhanced with an example and visual aids. The 'Configuration params' subsection now includes the 'use_description_markers' parameter.

/describe

Preparing review...

Preparing review...


Can't download flexible criteria

You get a missing route error. Looks like it's trying to do a POST, but MarkUs is only set to respond to a GET.

Dibs!


Fix circular import crash, Remove VTK js again.

Critical: Refactor constants to dive_utils to prevent circular import crash.  main is currently broken, workers fail to start
Adds very stupid test to confirm that imports all work.  This test would have broken on the initial problem, but I forgot to write the test first to show it failing.
Removes VTK.js again because apparently it got added back in.
Refactor prod TIMEZONE for duplicati and watchtower.  Watchtower is currently updating at the wrong time (2AM UTC).  Apparently /etc/timezone isn't working as advertised.  w/e.

Great catch, fixed in ebdc7bb


Restore neuron's .type property to reliably track input, hidden, and output nodes

Description
Because we are not reliably tracking the type of nodes, basically whether they are an: input, hidden, or output we're experiencing bugs related to the management of network structure: #163 and the long-standing #51
Sometime before commit 5b741fd3c91625e55b1dc1a7a112fdf4fddb7081 we removed the .type attribute of nodes but it's an elegant way to keep track of what the status of a node is.
I think the long-standing bugs we've been having #51 and maybe even #34 are related to not setting and keeping the status of nodes current.
Files
src/architecture/Node
src/architecture/Group
src/architecture/Layer
src/architecture/Network
methods/mutation
To Reproduce
Run .evolve or .mutate with Network or Neat
Additional Notes
Throughout this it will be important to preserve Carrot's ability to make arbitrary shaped networks which may include connecting outputs to existing input and hidden nodes without becoming an output node itself
Tasks
Include specific tasks in the order they need to be done in. Include links to specific lines of code where the task should happen at.

[x] Reintroduce types to neurons
[ ] Set the .type attribute on all constructors
[ ] Add code to update types when adding nodes, removing nodes, swapping, connecting, and disconnecting -- basically, updating the mutation code
[ ] Remove any assumptions about nodes order or ensure that any mutations respect ordered assumptions

Fixed in the typescript version


rel: Release v2.0.2

Ⅰ. Describe what this PR did
Ⅱ. Does this pull request fix one issue?

Ⅲ. Why don't you add test cases (unit test/integration test)?
Ⅳ. Describe how to verify it
Ⅴ. Special notes for reviews

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 43.52%. Comparing base (ef31e09) to head (284c46b).
Report is 172 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1438      +/-   ##
==========================================
+ Coverage   35.91%   43.52%   +7.61%
==========================================
  Files          69       76       +7
  Lines       11576    12320     +744
==========================================
+ Hits         4157     5362    +1205
+ Misses       7104     6622     -482
- Partials      315      336      +21

see 69 files with indirect coverage changes


Numerical addressing

Moving towards a next generation internal addressing scheme which will hopefully help us getting rid of the ugly and error prone container + container annotation index implementation.

@michael could you give me your feedback on the new address based API?
Some examples:

Getting an address for a path:
https://github.com/substance/substance/blob/b11916f50a0d0e3b1e90accea1654ad197335c5c/test/unit/document/container.test.js#L12
Traversing addresses:
https://github.com/substance/substance/blob/b11916f50a0d0e3b1e90accea1654ad197335c5c/test/unit/document/container.test.js#L49
https://github.com/substance/substance/blob/b11916f50a0d0e3b1e90accea1654ad197335c5c/test/unit/document/container.test.js#L119
Getting a range of paths:
https://github.com/substance/substance/blob/b11916f50a0d0e3b1e90accea1654ad197335c5c/test/unit/document/container.test.js#L194
Comparing addresses:
https://github.com/substance/substance/blob/b11916f50a0d0e3b1e90accea1654ad197335c5c/document/container_selection.js#L234
https://github.com/substance/substance/blob/b11916f50a0d0e3b1e90accea1654ad197335c5c/document/container_selection.js#L243

Some places where this is used already:
https://github.com/substance/substance/blob/b11916f50a0d0e3b1e90accea1654ad197335c5c/document/transformations/copy_selection.js#L77
The good thing about addresses is:

they are sortable/comparable
address is an array of indexes which represent childIndexes

From an address you can get to the top-level node using:
container.getChildAt(address[0]);

Thanks, I will have a look later! Need to go now.

This is good stuff! I think it's a powerful concept to allow hierarchical addressing (for lists etc.) while still allowing a flat view of components. I read through the tests, and I have not much to add. I'd like to test this inside the writer app, but there are still too many things breaking.

Hmmm. Maybe I was too fast with removing the container annotation index regarding smooth transition.
Should I recover that, so that we can try this out iteratively in the writer?

Not sure, probably it's more efficient to keep it a radical change. :) But maybe we should bring it together when I'm back. Also I think this should be tested first with the editor interface, then moving on to the writer. I'd rather continue with my Writer Refactor efforts, while it's getting better, there are still some smelly things I'd like to get rid of.
Maybe you have time to give me some thoughts regarding #63.

@michael plz review and merge

Looks very good. I didn't see anything break in the Lens Writer integration. Merging.


Replication for MS-MARCO passage & doc

Enviroment:
OS: Ubuntu 18.04.3 LTS
Java: openjdk version "11.0.8" 2020-07-14
Maven: Apache Maven 3.6.0
Replication results
Passage: Identical
$ python tools/scripts/msmarco/msmarco_eval.py \
 collections/msmarco-passage/qrels.dev.small.tsv runs/run.msmarco-passage.dev.small.tsv
#####################
MRR @10: 0.18741227770955546
QueriesRanked: 6980
#####################

$ tools/eval/trec_eval.9.0.4/trec_eval -c -mrecall.1000 -mmap \
 collections/msmarco-passage/qrels.dev.small.trec runs/run.msmarco-passage.dev.small.trec
map                   	all	0.1957
recall_1000           	all	0.8573

Document: Identical
(inference took 3.5 hours w/o SSD for this task)
2020-09-08 22:31:23,156 INFO  [main] search.SearchCollection (SearchCollection.java:715) - Total run time: 03:34:01

$ tools/eval/trec_eval.9.0.4/trec_eval -c -mmap -mrecall.1000 \
 src/main/resources/topics-and-qrels/qrels.msmarco-doc.dev.txt runs/run.msmarco-doc.dev.bm25.txt
map                   	all	0.2310
recall_1000           	all	0.8856

Small issues:
This notebook has an outdated command: cd eval -> cd tools/eval

Small issues:
This notebook has an outdated command: cd eval -> cd tools/eval

Can you please file an issue against this repo for above? I'll address next change I have.


Allow lookup operator [] with Java Map and List, and in operator with Java Collection

We should let you write:

list[i] and map[key] with java.util.Map and java.util.List, and
x in collection with any java.util.Collection.

+1000

Note: this is a trivial change to the typechecker, so when @tombentley and or @FroMage is ready to implement it, just let me know...

That's a feature though, so 1.3, right?

It's a trivial feature.

Alright, so in is done :-)

the current transform uses stuff like getSupertype(Correspondence)

@tombentley where precisely is this code?

@gavinking the code generator should now handle list[index] and map[key].

yay! thanks :-)

Actually what should we do if the List index is out of bounds? Should we just call get() and let the IOOBE propagate, or should the expression evaluate to null instead? Same goes for #6064.

@tombentley that is a great question, and I have no particular intuition on the matter.
I suppose null would be nice...

Well null in the case of arrays of primitive type would mean we'd have to box the result. That would often run counter to the user's choice to use Java arrays (performance), so for primitive arrays at least I think propagating the exception is the right thing to do. Then for consistency I suppose we should propagate for ObjectArray and probably for List too, though that's not totally clear.
Of course, if we propagate then the type of primarray[index] would be non-optional.

Well I don't think it's necessary that primitive arrays and List/Map do precisely the same thing, since, as you've observed, they're different...

OK, I'll fix the List one to return null then.

OK, I'll fix the List one to return null then.

Waiting.... ;-)

@gavinking can you prohibit stuff like this:
interface JavaListIndex<T> satisfies JList<T>  {
    shared void m() {
        value t = super[0];
    }
}

where get is not implemented by any super.

@tombentley it's not already prohibited?!

Doesn't seem to be

Ouch. Can you open a separate bug for that. 'Cos I want to close this one.

Done: #6067

This is done. Yay!


Attempting to copy the coverage report out

It's inside the container!
Changes
Add a summary of the changes in this pull request
Checklist

[ ] Attach this pull request to the appropriate card in Trello.
[ ] Update the CHANGELOG.md if needed.

Still cannot find a way to know what the running container is called so we can copy the file out - closing for now.


ValueError: setting an array element with a sequence.

Writing some quick n dirty code just to get the framework going. Yes, I know it could be much more elegant. Having a maddening error I can't seem to figure out.
import gym
from gym import spaces
import numpy as np
from stable_baselines.common.vec_env.dummy_vec_env import DummyVecEnv
from stable_baselines.a2c.a2c import A2C
from stable_baselines.common.policies import CnnPolicy
class CustomEnv(gym.Env):
    metadata = {'render.modes': ['human']}

    def __init__(self, ob_sp,N_DISCRETE_ACTIONS):
        super(CustomEnv, self).__init__()
        self.action_space = spaces.Discrete(N_DISCRETE_ACTIONS)

        self.l=ob_sp["l"]
        self.h=ob_sp["h"]
        self.x=ob_sp["x"]
        self.y=ob_sp["y"]
        self.z=ob_sp["z"]
        self.dtype=ob_sp["dtype"]

        self.observation_space = spaces.Box(low=self.l, high=self.h, shape=(self.x,self.y,self.z), dtype=self.dtype)

    def step(self, action):
        self.obs,self.reward,self.done,self.comment=self.execute(action)
        self.steps+=1
        return self.obs, self.reward, self.done, {}

    def reset(self):
        self.steps=0
        self.done=False
        self.obs=self.random()
        return self.obs

    def render(self, mode='human', close=False):
        if mode == 'rgb_array':
            return self.obs
        elif mode == 'human':
            from gym.envs.classic_control import rendering
            if self.viewer is None:
                self.viewer = rendering.SimpleImageViewer()

            if self.obs.ndim<=3:obs=np.array(self.obs,dtype=self.dtype)

            else:obs=np.array(self.obs[:,:,:,0],dtype=self.dtype)
            self.viewer.imshow(obs)
            return self.viewer.isopen

    def random(self):
        self.obs=np.random.randint(self.l,self.h,size=(self.x,self.y,self.z),dtype=self.dtype)
        return self.obs

    def execute(self,action):
        self.random()
        self.reward=self.random()
        if self.steps>100: self.done=True
        return self.obs, self.reward, self.done, {}

print("program start")
#if __name__=="main":
print("Dummy run")
shape={}
shape["l"]=0
shape["h"]=255
shape["x"]=128
shape["y"]=64
shape["z"]=3
shape["dtype"]=np.uint8
N_DISCRETE_ACTIONS=2

env=DummyVecEnv([lambda: CustomEnv(shape,N_DISCRETE_ACTIONS)])
model= A2C(CnnPolicy,env).learn(total_timesteps=1000)

Getting this back:

Traceback (most recent call last):
File "/Users/kodiak/5th/custom_env.py", line 71, in
model= A2C(CnnPolicy,env).learn(total_timesteps=1000)
File "/Users/kodiak/anaconda3/envs/tflow/lib/python3.6/site-packages/stable_baselines/a2c/a2c.py", line 236, in learn
obs, states, rewards, masks, actions, values, ep_infos, true_reward = runner.run()
File "/Users/kodiak/anaconda3/envs/tflow/lib/python3.6/site-packages/stable_baselines/a2c/a2c.py", line 330, in run
obs, rewards, dones, infos = self.env.step(clipped_actions)
File "/Users/kodiak/anaconda3/envs/tflow/lib/python3.6/site-packages/stable_baselines/common/vec_env/base_vec_env.py", line 94, in step
return self.step_wait()
File "/Users/kodiak/anaconda3/envs/tflow/lib/python3.6/site-packages/stable_baselines/common/vec_env/dummy_vec_env.py", line 36, in step_wait
self.envs[env_idx].step(self.actions[env_idx])
ValueError: setting an array element with a sequence.

I know the code hiccups because somewhere a non-scalar is being put into an array cell. But where?
Help would be most appreciated
:-)

Hello,
the problem comes from this line:
self.reward=self.random()

the reward must be a scalar, not a ndarray that looks like an image ;)

Ah, yes, of course. Thank you so much. Kudos!


Inconsistent event status

When an event is received, the status is sent as TRIGGERED. However, when using the find endpoint and retrieving this event, the status is PROCESSED. Is this the expected status only after successful delivery of an event?

That is correct, when BigBen calls the tenant, there's no knowing in advance that the call will succeed or not (the http API may be down, or kafka producer may time out, etc). Hence it calls with status TRIGGERED. Only if the call succeeds (after any number of configured retries), it marks the event as PROCESSED.
The status TRIGGERED (due to failure of the tenant to respond) is not preserved in db because it can be sort of inferred from the following condition: status == UN_PROCESSED && error != null
Let me know what you think.


Is it possible to specify which module to import from with Transforms?

Couldn't find it in the documentation or by digging a bit in the source code; is it possible to specify which file to import when setting up our transforms?
Say I have a Contact file for a controller in app/controller/ and a Contact object for my routing in app/routes. It'd like to be able to tell grunt-ts to import not the controller but the route. Something like ts:ref=routes.Contact.
Is this at all possible?

Is this at all possible?

Not at the moment. File names need to be be unique.

Couldn't find it in the documentation

If someone can doc this it would be great :rose:

Alright, thanks for the answer, I'll try and work around it for now.
Knowing that it's not possible though, isn't it something that would be useful? It's far from far-fetched to imagine a scenario where there are multiple files with the same name in different modules.

@gCardinal Hi - sorry this floated out there so long.  I am willing to consider a PR once v5 comes out.  If you're interested to create this functionality, please let me know.  For now, I'll mark this as jump in.


error in create_data.py create_kitti_info_file

/home/fupeng/anaconda3/lib/python3.7/site-packages/numba/cuda/envvars.py:16: NumbaDeprecationWarning:
Environment variables with the 'NUMBAPRO' prefix are deprecated, found use of NUMBAPRO_NVVM=/usr/local/cuda/nvvm/lib64/libnvvm.so.
For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-numbapro-environment-variables
warnings.warn(errors.NumbaDeprecationWarning(msg))
/home/fupeng/anaconda3/lib/python3.7/site-packages/numba/cuda/envvars.py:16: NumbaDeprecationWarning:
Environment variables with the 'NUMBAPRO' prefix are deprecated, found use of NUMBAPRO_LIBDEVICE=/usr/local/cuda/nvvm/libdevice.
For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-numbapro-environment-variables
warnings.warn(errors.NumbaDeprecationWarning(msg))
/home/fupeng/anaconda3/lib/python3.7/site-packages/numba/cuda/envvars.py:16: NumbaDeprecationWarning:
Environment variables with the 'NUMBAPRO' prefix are deprecated, found use of NUMBAPRO_CUDA_DRIVER=/usr/lib/x86_64-linux-gnu/libcuda.so.
For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-numbapro-environment-variables
warnings.warn(errors.NumbaDeprecationWarning(msg))
middle_class_name PointPillarsScatter
num_trainable parameters: 66
{'Car': 5}
[-1]
load 14357 Car database infos
load 2207 Pedestrian database infos
load 734 Cyclist database infos
load 1297 Van database infos
load 56 Person_sitting database infos
load 488 Truck database infos
load 224 Tram database infos
load 337 Misc database infos
After filter database:
load 10520 Car database infos
load 2104 Pedestrian database infos
load 594 Cyclist database infos
load 826 Van database infos
load 53 Person_sitting database infos
load 321 Truck database infos
load 199 Tram database infos
load 259 Misc database infos
remain number of infos: 3712
remain number of infos: 3769
WORKER 0 seed:<PHONE_NUMBER>
WORKER 1 seed:<PHONE_NUMBER>
/home/fupeng/fupeng/second.pytorch/second/core/geometry.py:97: NumbaWarning:
Compilation is falling back to object mode WITH looplifting enabled because Function "points_in_convex_polygon_3d_jit" failed type inference due to: Invalid use of type(CPUDispatcher(<function surface_equ_3d_jit at 0x7ff72b24eae8>)) with parameters (array(float64, 4d, A))

parameterized
[1] During: resolving callee type: type(CPUDispatcher(<function surface_equ_3d_jit at 0x7ff72b24eae8>))
[2] During: typing of call at /home/fupeng/fupeng/second.pytorch/second/core/geometry.py (118)

File "core/geometry.py", line 118:
def points_in_convex_polygon_3d_jit(points,

num_surfaces = np.full((num_polygons,), 9999999, dtype=np.int64)
normal_vec, d = surface_equ_3d_jit(polygon_surfaces[:, :, :3, :])
^
@numba.jit(nopython=False)
/home/fupeng/fupeng/second.pytorch/second/core/geometry.py:97: NumbaWarning:
Compilation is falling back to object mode WITH looplifting enabled because Function "points_in_convex_polygon_3d_jit" failed type inference due to: Invalid use of type(CPUDispatcher(<function surface_equ_3d_jit at 0x7ff72b24eae8>)) with parameters (array(float64, 4d, A))

parameterized
[1] During: resolving callee type: type(CPUDispatcher(<function surface_equ_3d_jit at 0x7ff72b24eae8>))
[2] During: typing of call at /home/fupeng/fupeng/second.pytorch/second/core/geometry.py (118)

File "core/geometry.py", line 118:
def points_in_convex_polygon_3d_jit(points,

num_surfaces = np.full((num_polygons,), 9999999, dtype=np.int64)
normal_vec, d = surface_equ_3d_jit(polygon_surfaces[:, :, :3, :])
^
@numba.jit(nopython=False)
/home/fupeng/fupeng/second.pytorch/second/core/geometry.py:97: NumbaWarning:
Compilation is falling back to object mode WITHOUT looplifting enabled because Function "points_in_convex_polygon_3d_jit" failed type inference due to: cannot determine Numba type of <class 'numba.dispatcher.LiftedLoop'>
File "core/geometry.py", line 123:
def points_in_convex_polygon_3d_jit(points,

sign = 0.0
for i in range(num_points):
^
@numba.jit(nopython=False)
/home/fupeng/fupeng/second.pytorch/second/core/geometry.py:97: NumbaWarning:
Compilation is falling back to object mode WITHOUT looplifting enabled because Function "points_in_convex_polygon_3d_jit" failed type inference due to: cannot determine Numba type of <class 'numba.dispatcher.LiftedLoop'>
File "core/geometry.py", line 123:
def points_in_convex_polygon_3d_jit(points,

sign = 0.0
for i in range(num_points):
^
@numba.jit(nopython=False)
Traceback (most recent call last):
File "pytorch/train.py", line 660, in
fire.Fire()
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/fire/core.py", line 138, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/fire/core.py", line 468, in _Fire
target=component.name)
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/fire/core.py", line 672, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "pytorch/train.py", line 415, in train
raise e
File "pytorch/train.py", line 237, in train
example = next(data_iter)
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 819, in next
return self._process_data(data)
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 846, in _process_data
data.reraise()
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/torch/_utils.py", line 369, in reraise
/home/fupeng/fupeng/second.pytorch/second/core/geometry.py:97: NumbaWarning:
Compilation is falling back to object mode WITH looplifting enabled because Function "points_in_convex_polygon_3d_jit" failed type inference due to: Invalid use of type(CPUDispatcher(<function surface_equ_3d_jit at 0x7ff72b24eae8>)) with parameters (array(float64, 4d, A))

parameterized
[1] During: resolving callee type: type(CPUDispatcher(<function surface_equ_3d_jit at 0x7ff72b24eae8>))
[2] During: typing of call at /home/fupeng/fupeng/second.pytorch/second/core/geometry.py (118)

File "core/geometry.py", line 118:
def points_in_convex_polygon_3d_jit(points,

num_surfaces = np.full((num_polygons,), 9999999, dtype=np.int64)
normal_vec, d = surface_equ_3d_jit(polygon_surfaces[:, :, :3, :])
^
@numba.jit(nopython=False)
raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
data = fetcher.fetch(index)
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/fupeng/fupeng/second.pytorch/second/pytorch/builder/input_reader_builder.py", line 42, in getitem
return self._dataset[idx]
File "/home/fupeng/fupeng/second.pytorch/second/data/dataset.py", line 68, in getitem
prep_func=self._prep_func)
File "/home/fupeng/fupeng/second.pytorch/second/data/preprocess.py", line 354, in read_and_prep_v9
example = prep_func(input_dict=input_dict)
File "/home/fupeng/fupeng/second.pytorch/second/data/preprocess.py", line 197, in prep_pointcloud
num_try=100)
File "/home/fupeng/fupeng/second.pytorch/second/core/preprocess.py", line 681, in noise_per_object_v3
point_masks = points_in_convex_polygon_3d_jit(points[:, :3], surfaces)
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/numba/dispatcher.py", line 370, in _compile_for_args
raise e
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/numba/dispatcher.py", line 327, in _compile_for_args
return self.compile(tuple(argtypes))
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/numba/compiler_lock.py", line 32, in _acquire_compile_lock
return func(*args, **kwargs)
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/numba/dispatcher.py", line 659, in compile
cres = self._compiler.compile(args, return_type)
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/numba/dispatcher.py", line 83, in compile
pipeline_class=self.pipeline_class)
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/numba/compiler.py", line 955, in compile_extra
return pipeline.compile_extra(func)
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/numba/compiler.py", line 377, in compile_extra
return self._compile_bytecode()
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/numba/compiler.py", line 886, in _compile_bytecode
return self._compile_core()
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/numba/compiler.py", line 873, in _compile_core
res = pm.run(self.status)
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/numba/compiler_lock.py", line 32, in _acquire_compile_lock
return func(*args, **kwargs)
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/numba/compiler.py", line 254, in run
raise patched_exception
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/numba/compiler.py", line 245, in run
stage()
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/numba/compiler.py", line 463, in stage_objectmode_frontend
cres = self.frontend_looplift()
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/numba/compiler.py", line 435, in frontend_looplift
lifted=tuple(loops), lifted_from=None)
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/numba/compiler.py", line 1017, in compile_ir
lifted_from=lifted_from)
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/numba/compiler.py", line 385, in compile_ir
return self._compile_ir()
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/numba/compiler.py", line 893, in _compile_ir
return self._compile_core()
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/numba/compiler.py", line 873, in _compile_core
res = pm.run(self.status)
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/numba/compiler_lock.py", line 32, in _acquire_compile_lock
return func(*args, *kwargs)
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/numba/compiler.py", line 254, in run
raise patched_exception
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/numba/compiler.py", line 245, in run
stage()
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/numba/compiler.py", line 711, in stage_objectmode_backend
self._backend(lowerfn, objectmode=True)
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/numba/compiler.py", line 687, in _backend
lowered = lowerfn()
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/numba/compiler.py", line 659, in backend_object_mode
self.flags)
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/numba/compiler.py", line 1148, in py_lowering_stage
lower.lower()
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/numba/lowering.py", line 202, in lower
self.library.add_ir_module(self.module)
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/numba/targets/codegen.py", line 185, in add_ir_module
ll_module = ll.parse_assembly(ir)
File "/home/fupeng/anaconda3/lib/python3.7/site-packages/llvmlite/binding/module.py", line 26, in parse_assembly
raise RuntimeError("LLVM IR parsing error\n{0}".format(errmsg))
RuntimeError: Failed in object mode pipeline (step: object mode frontend)
Failed in object mode pipeline (step: object mode backend)
LLVM IR parsing error
:578:48: error: expected value token
%".353" = call i8 @"PyLong_FromSsize_t"(i64 slice(1, 3, None))
^
/home/fupeng/fupeng/second.pytorch/second/core/geometry.py:97: NumbaWarning:
Compilation is falling back to object mode WITH looplifting enabled because Function "points_in_convex_polygon_3d_jit" failed type inference due to: Invalid use of type(CPUDispatcher(<function surface_equ_3d_jit at 0x7ff72b24eae8>)) with parameters (array(float64, 4d, A))

parameterized
[1] During: resolving callee type: type(CPUDispatcher(<function surface_equ_3d_jit at 0x7ff72b24eae8>))
[2] During: typing of call at /home/fupeng/fupeng/second.pytorch/second/core/geometry.py (118)

File "core/geometry.py", line 118:
def points_in_convex_polygon_3d_jit(points,

num_surfaces = np.full((num_polygons,), 9999999, dtype=np.int64)
normal_vec, d = surface_equ_3d_jit(polygon_surfaces[:, :, :3, :])

I don't know the reason, if someone know , could you tell me ? it the numba version is higer?
I solution is to comment the code !
https://github.com/traveller59/second.pytorch/issues/67


Make version history and Report issue buttons on docs side-by-side ?

The buttons on the docs page aren't actually side-by-side -- any chance we could make them appear horizontally stacked?
Screenshot of http://googlecloudplatform.github.io/gcloud-ruby/docs/v0.1.0/Gcloud/Datastore/KeyfileError.html:

The buttons are now side-by-side.


[Completion] Don’t generate USRs for local decls

Doing so can cause issues since we cannot guarantee that e.g any parent closures have been type-checked.
rdar://128294522

@swift-ci please test

@swift-ci please smoke test


Searchium.ai partner information

Organization name
Searchium.ai
Brief description of your business and what it does (50 words or less)
Searchium.ai is a SaaS platform that optimizes neural search applications.
Based on in-memory computing built specifically for search, we give you a significant advantage, easy-to-integrate neural search that scales to billions without sacrificing speed or accuracy.
Name of your relevant product(s) or service for the OpenSearch audience
The Searchium OpenSearch plugin enables you to connect remotely to a hardware accelerator for low latency searching and infrastructure cost reduction.
What OpenSearch technologies do you specialize in
Search
Type of business
Product/technology
Home/main office location or address
18 Raul Wallenberg, Tel Aviv, Israel
How can people contact you (email and phone)
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Regions you serve
Searchium.ai's products and services are available globally, covering all regions. (North America, South America, Europe, Middle East, Africa, Asia Pacific, Australia)
Industry specialization
Software and technology
Logo

Links
https://searchium.ai
https://betterprogramming.pub/tired-of-troubleshooting-idle-search-resources-use-opensearch-benchmark-for-performance-tuning-d4277c9f724
https://towardsdatascience.com/bolster-opensearch-performance-with-5-simple-steps-ca7d21234f6b
https://towardsdatascience.com/text-search-vs-vector-search-better-together-3bd48eb6132a
https://medium.com/gsi-technology/the-apu-novel-hardware-for-accelerated-similarity-search-beaef9b19b97
https://medium.com/gsi-technology/semantic-vector-search-taking-the-leap-from-keyword-search-18855e96854a
https://towardsdatascience.com/speeding-up-bert-search-in-elasticsearch-750f1f34f455
https://github.com/Searchium-ai

@noamschwartz - Thanks, Noam, for proving the information.


Cannot run as service

Any ideas??
$ sudo systemctl start wire-pod
$ sudo systemctl status wire-pod
● wire-pod.service - Wire Escape Pod (coqui)
     Loaded: loaded (/lib/systemd/system/wire-pod.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2022-10-27 19:56:22 CEST; 4s ago
    Process: 731 ExecStart=/home/wirepod/wire-pod/chipper/start.sh (code=exited, status=1/FAILURE)
   Main PID: 731 (code=exited, status=1/FAILURE)
        CPU: 35ms

Oct 27 19:56:22 escapepod systemd[1]: Started Wire Escape Pod (coqui).
Oct 27 19:56:22 escapepod start.sh[734]: go: module cache not found: neither GOMODCACHE nor GOPATH is set
Oct 27 19:56:22 escapepod systemd[1]: wire-pod.service: Main process exited, code=exited, status=1/FAILURE
Oct 27 19:56:22 escapepod systemd[1]: wire-pod.service: Failed with result 'exit-code'.

Wire-pod is giving me (more) grey hair.
It ought to be SO easy, but haven't had much luck yet :(
# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 11 (bullseye)
Release:        11
Codename:       bullseye

Running as Raspbian on a RPI 4 4GB

May you send the output of ls /home/wirepod/wire-pod/chipper? Could you also run ./setup.sh daemon-enable in the wire-pod directory again and paste the full output here?

$ ls /home/wirepod/wire-pod/chipper
build.sh  cmd-leopard  docker-compose.yaml  epod    go.sum   pkg        source.sh  stttest.pcm  weather-map.json
cmd       cmd-vosk     Dockerfile           go.mod  LICENSE  README.md  start.sh   useepod      webroot

$ ./setup.sh daemon-enable

Debian-based Linux confirmed.
aarch64 architecture confirmed.
This script must be run as root. sudo ./setup.sh

$ sudo ./setup.sh daemon-enable

Debian-based Linux confirmed.
aarch64 architecture confirmed.
Checks have passed!

cat: ./epod/ep.crt: No such file or directory
cat: ./epod/ep.key: No such file or directory
[Unit]
Description=Wire Escape Pod (coqui)

[Service]
Type=simple
WorkingDirectory=/home/wirepod/wire-pod/chipper
ExecStart=/home/wirepod/wire-pod/chipper/start.sh

[Install]
WantedBy=multi-user.target

wire-pod.service created, building chipper with VOSK STT service...

./chipper/chipper has been built!
Created symlink /etc/systemd/system/multi-user.target.wants/wire-pod.service → /lib/systemd/system/wire-pod.service.

systemd service has been installed and enabled! The service is called wire-pod.service

To start the service, run: 'systemctl start wire-pod'
Then, to see logs, run 'journalctl -fe | grep start.sh'

Now I can't even do a ´sudo systemctl start wire-pod´ doesn't exist.

Okay - for some reason systemctl daemon-reload wasn't run.
After running that manually it seems like it's staring this time. Perhaps this was related to the failed VOSK build in another issue


Implement a keybinding to restart/reload a tab

Description of the new feature/enhancement
ConEmu has a default binding of Win+tilde to reload/restart the current tab.  This would be a useful feature to add to Windows Terminal, specifically to resolve issues whereby a fresh start of a tab is useful (eg: crashes, troubleshooting).

Yea, I guess I could see that being useful. Thanks for the suggestion!

Agreed! Just want to add a concrete example:
You make a change on ~/.bashrc (or similar things) and don't want to do a source ... but a simple tab realod.

I mean, you could just exec bash -li. Terminal would destroy the process tree and relaunch it; is that really what you want instead of just sourcing your .bashrc again?

I think there is no definitive answer to your question but imho a relaunch as you described it would often be good enough and with a keybinding the comfort wins :)

Hey I'm gonna cross-link a pair of issues:

This would be a good idea to include in the context menu, #1912
This could be built directly into the connection somehow? With #4379 perhaps?

FWIW; I use Terminal tabs to ssh to other hosts a lot. When those ssh connections get dropped - e.g. I reboot the remote host, or there's some sort of network interruption - I keep wishing I could right-click on the tab and select a "restart" option of some sort, rather than having to close the dead tab, open a new one, then drag it to the position the old one was occupying.

+1 for this feature - being able to ctrl+r an SSH session would be super useful!

@bbeckford If you'd like to "+1" this feature, the best way to do that is by hitting the 👍 button on this issue

That way, you avoid unnecessarily pinging everyone following this thread. Thanks!

I would like this for both tabs and panes.  I encounter this issue with PowerShell.  It will stop outputting data to the screen (unknown cause), or the remote sessions will get hosed because the temp files are deleted.   Reconnecting the remote sessions don't work so restarting the shell seems to be the only fix.  I also encounter this if PS or some other process inside starts consuming too much memory, or the shell becomes unresponsive due to whatever abuse I'm hurling at it.  I tend to have consoles open for days, even weeks at a time, so this happens "regularly" for me.

If anyone is interested in a quick solution, I posted in #4379 a quick hack to get this functionality working in Windows Terminal here:
https://github.com/microsoft/terminal/issues/4379#issuecomment-905112832

I would love to see this particular feature prioritized. It's an incredible quality of life update that would drive a huge amount of impact from the users perspective.

Its a bit of a stretch and might not be perfect, but with https://github.com/microsoft/terminal/issues/960 just merged, you could add the following action to "restart" a pane
{ "command": {
    "action": "multipleActions",
    "actions" = [
      { "action": "closePane" },
      { "action": "restoreLastClosed" } ]
    }
}

This does what it says and closes the pane and then creates a new pane with the same settings as the last closed pane.

My usecase from #12232:
When ssh hangs in WSLv1 (usually because of network issues or after laptop sleep / hibernate), somehow it doesn't react to any Ctrl+C, Ctrl+Z. Then it's impossible to kill the ssh process, and it can take a few minutes for it to bail out by itself.

Alternative feature request: somehow detecting the current process in the bash session and providing a command to kill it

My usecase:
open an ssh session and start screen on that, start a job in screen and let it run.
I want to reload the ssh session so that screen can continue in background, and I come back to new ssh session in same tab
When ever I want I can go back to my screen from new session.
I know screen has option to exit without terminating, but being iterm user on mac, I find it more convenient to just reload the tab instead.

Its a bit of a stretch and might not be perfect, but with #960 just merged, you could add the following action to "restart" a tab
        {
          "command": {
            "action": "multipleActions",
            "actions": [
                "closeTab",
                "restoreLastClosed"
            ]
            },
          "name": "Restart tab"
        },

This does what it says and closes the tab and then creates a new tab with the same settings as the last closed tab. Replace closeTab with closePane if you want to restart a single pane (at the time of writing the closePane version is buggy and doesn't work as part of multipleActions as expected).

The problem with this is the tab position is not restored, if tab closed is first one, the restored tab is the last one.
the position should also be restored

Its a bit of a stretch and might not be perfect, but with #960 just merged, you could add the following action to "restart" a tab
        {
          "command": {
            "action": "multipleActions",
            "actions": [
                "closeTab",
                "restoreLastClosed"
            ]
            },
          "name": "Restart tab"
        },

This does what it says and closes the tab and then creates a new tab with the same settings as the last closed tab. Replace closeTab with closePane if you want to restart a single pane (at the time of writing the closePane version is buggy and doesn't work as part of multipleActions as expected).

In case there is only one tab left, the whole window will be closed directly, and I found that this problem can be solved by split pane command.
        {
            "command":
            {
                "action": "multipleActions",
                "actions":
                [
                    {
                        "action": "splitPane",
                        "size":0.1,
                        "split": "up",
                        "splitMode":"duplicate"
                    },
                    {
                        "action": "moveFocus",
                        "direction": "previous"
                    },
                    {
                        "action": "closePane"
                    }

                ]
            },
            "keys": "ctrl+r",
            "name": "restart tab"
        }

I think this method is close to perfect, except when closing the pane, it shows the content behind the terminal, which may be a rendering engine issue. I made the newly added pane as large as possible (by setting size to 0.1) to minimize its impact.
In addition, the most important thing is that this method does not require an update to the preview version.

I think

它有点牵强，可能并不完美，但#960刚刚合并，您可以添加以下操作来“重新启动”选项卡
        {
          "command": {
            "action": "multipleActions",
            "actions": [
                "closeTab",
                "restoreLastClosed"
            ]
            },
          "name": "Restart tab"
        },

这会执行它所说的并关闭选项卡，然后创建一个与上次关闭的选项卡具有相同设置的新选项卡。如果要重新启动单个窗格，请替换closeTab为（在编写该版本时存在错误，并且不能按预期作为 multipleActions 的一部分工作）。closePane``closePane

如果只剩下一个tab，直接关闭整个窗口，我发现这个问题可以通过split pane命令解决。
        {
            "command":
            {
                "action": "multipleActions",
                "actions":
                [
                    {
                        "action": "splitPane",
                        "size":0.1,
                        "split": "up",
                        "splitMode":"duplicate"
                    },
                    {
                        "action": "moveFocus",
                        "direction": "previous"
                    },
                    {
                        "action": "closePane"
                    }

                ]
            },
            "keys": "ctrl+r",
            "name": "restart tab"
        }

我认为这种方法接近完美，除了关闭窗格时，它显示了终端后面的内容，这可能是渲染引擎的问题。我使新添加的窗格尽可能大（通过设置size为 0.1）以最大程度地减少其影响。
另外，最重要的是这种方法不需要更新到预览版。

Thanks for your answer! but I think it is not perfect, I just want to reopen this SSH quickly, but the "ctrl + r" will do 3 actions include spliting and closing  the original pane  ,and there is a dividing line will flash quickly.

For anyone wanting to bind a key to the new restartConnection feature, this worked for me in my settings.json
{
    "$help": "https://aka.ms/terminal-documentation",
    "$schema": "https://aka.ms/terminal-profiles-schema",
    "actions": [
        {
            "command": "restartConnection",
            "keys": "alt+shift+r"
        }
    ]
}

The JSON schema seems to think it's invalid, but it works for me.
Looks like "$schema": "https://aka.ms/terminal-profiles-schema" needs to be updated to lint for it.


(feat) support for downgraded github timeouts

can you confirm that if no duration is defined, the attribute is omitted?  Just want to make sure it doesn't output "0s" which could be interpreted as an immediate timeout


Add pkgs.overrideWithScope

Add pkgs.overrideWithScope. This function is similar to pkgs.override, except that it filters the arguments like callPackage does.
While this might seems like a small addition, this is actually a really really big change for the future of Nixpkgs. This function can help us simplify a lot the manipulation of emacsPackages, pythonPackages, haskellPackages, linuxPackages, ... This can also help us rewrite the bootstrapping done before all-packages and move it within all-packages.nix. and on a simpler note, this will help solving the #43755 as explained in #43828.
The reason which makes this function is a revolution for the implementation of package sets is that it make it simple to ""recursively"" (dependencies of dependencies of dependencies) and conherently override the input of a set of packages with a simple function, which does not appear in future overlay.
To make it clearer, here is a set of overlay:
import <nixpkgs> { overlays = [
  # Add an interpreter and its package set.
  (self: super:
  let interpreterPackages = withSet: {
      foo = super.lib.callPackageWith withSet ./pkgs/interpreter/foo { /* interpreter libxml */ };
      bar = super.lib.callPackageWith withSet ./pkgs/interpreter/bar { /* interpreter foo */ };
      baz = super.lib.callPackageWith withSet ./pkgs/interpreter/baz { /* interpreter foo bar qt */ };
      qux = super.lib.callPackageWith withSet ./pkgs/interpreter/qux { /* interpreter bar baz */ };
    };
  in
  rec {
    interpreter27 = super.callPackage ./pkgs/interpreter_27.nix {};
    interpreter35 = super.callPackage ./pkgs/interpreter_35.nix {};
    interpreter27Packages = let withSet = self // { interpreter = interpreter27; } // self.interpreter27Packages; in
      super.lib.mapAttrs (n: v: v.overrideWithScope withSet) (interpreterPackages withSet);
    interpreter35Packages = let withSet = self // { interpreter = interpreter35; } // self.interpreter35Packages; in
      super.lib.mapAttrs (n: v: v.overrideWithScope withSet) (interpreterPackages withSet);
  })
  # Change a package dependency. Like any ordinary package, as opposed as today.
  (self: super: {
    interpreter27Packages = super.interpreter27Packages // {
      # Will change foo within bar, baz, qux (as expected).
      foo = super.interpreter27Packages.foo.override { libxml = null; };
    };
  })
  # Duplicate a package set with a custom interpreter.
  (self: super: {
    interpreter27ProfilePackages = let withSet = self // self.interpreter27ProfilePackages; in
      super.lib.mapAttrs (n: v: v.overrideWithScope withSet) (super.interpreter27Packages // {
      interpreter = super.interpreter27.override { withProfiler = true; };
    });
  })
]; }

What is important to note, is that:

Changing a package within an existing attribute set, except for the attribute set manipulation, is like any ordinary package.
Creating a forked configuration (with a profiler for example), is as simple as changing the interpreter in a new set which is isolated with overrideWithScope.

To make it simple, pkgs.overrideWithScope make it possible to leverage the Nixpkgs fix-point without the burden of writing too many pkgs.override by hand (which need to have no-more than the expected set of arguments).
As it leverage the Nixpkgs fix-point, there is no need to unwrap the function to override packages within a given set, like is done with package sets today (Haskell being infamous due to the stacking of multiple fix-points).
Ideally, we would want to have the pkgs.overrideWithScope evaluated before the resolution of the arguments of a given package, which would not be possible until we add some post-processing of all packages at the end of Nixpkgs fix-point. This enforces that we have to provide the withSet argument to the interpreter set, and implies that we cannot have a raw set of packages which lives independently of a given interpreter version. (Note, one could cheat by providing a fake version of missing packages interpreter = null; libxml = null; qt = null; )
Another issue, which already exists but is made explicit here, is that self // self.interpreterPackages is inefficient in terms of memory because attribute sets are not lazy.  This does not worry me much as this is a fixable issue in Nixpkgs, as long as the expression is strictly used as argument of callPackagesWith and overrideWithScope
Other suggestions:

Renaming overrideWithScope to overrideWith.
Replacing override by overrideWithScope.

I'm not quite sure about this idiom. Say we had this function:
let
  # injectOverride
  # :  (name: String)
  # -> ({ a } -> { a } -> { a })
  # -> ({ name: {a}; ..b } -> { name: {a}; ..b } -> { name: {a}; ..b })
  injectOverride = name: f: self: super: {
    "${name}" = super.${name} // f self.${name} super.${self} // {
      __overlay__ = lib.composeExtensions (super.__inner__ or (_: _: {})) f;
    };
  };

Then we can just do:
import <nixpkgs> { overlays = [
  # Add an interpreter and its package set.
  (self: super:
  let
    pkgs = self;
    interpreterPackages = self: {
      callPackage = pkgs.newScope self;
      foo = self.callPackage ./pkgs/interpreter/foo { /* interpreter libxml */ };
      bar = self.callPackage ./pkgs/interpreter/bar { /* interpreter foo */ };
      baz = self.callPackage ./pkgs/interpreter/baz { /* interpreter foo bar qt */ };
      qux = self.callPackage ./pkgs/interpreter/qux { /* interpreter bar baz */ };
    };
  in
  rec {
    interpreter27 = self.callPackage ./pkgs/interpreter_27.nix {};
    interpreter35 = self.callPackage ./pkgs/interpreter_35.nix {};
  }
  // injectOverride "interpreter27Packages"
       (lib.extend (_: _: { interpreter = interpreter27; }) interpreterPackages)
  // injectOverride "interpreter35Packages"
       (lib.extend (_: _: { interpreter = interpreter35; }) interpreterPackages))
  # Change a package dependency. Like any ordinary package, as opposed as today.
  (injectOverride "interpreter27Packages" {
      # Will change foo within bar, baz, qux (as expected).
      foo = super.interpreter27Packages.foo.override { libxml = null; };
  })
  # Duplicate a package set with a custom interpreter.
  (self: super: injectOverride "interpreter27ProfilePackages"
    super.interpreter27Packages.__overlay__ self super)
]; }

@Ericson2314
The problem with the idiom you are suggesting is that users of a given sub-set of packages will have to know about the injectOverride function, and know when to use it.  As opposed to overrideWithScope which would only be required at the creation/cloning of the sub-set.
(Also this should always be super.callPackage)
Honestly, if post-processing of Nixpkgs was an option, and every package would use callPackage (which is far from being the case), then we could make the callPackage function return a functor like object which is derived into a derivation only after going through the Nixpkgs fix-point, and thus we could go with just the following:
import <nixpkgs> { overlays = [
  # Add an interpreter and its package set.
  (self: super:
  let interpreterPackages = {
      foo = super.callPackage ./pkgs/interpreter/foo { /* interpreter libxml */ };
      bar = super.callPackage ./pkgs/interpreter/bar { /* interpreter foo */ };
      baz = super.callPackage ./pkgs/interpreter/baz { /* interpreter foo bar qt */ };
      qux = super.callPackage ./pkgs/interpreter/qux { /* interpreter bar baz */ };
    };
  in
  rec {
    interpreter27 = super.callPackage ./pkgs/interpreter_27.nix {};
    interpreter35 = super.callPackage ./pkgs/interpreter_35.nix {};
    interpreter27Packages = { interpreter = interpreter27; } // interpreterPackages;
    interpreter35Packages = { interpreter = interpreter35; } // interpreterPackages;
  })
  # Change a package dependency. Like any ordinary package, as opposed as today.
  (self: super: {
    interpreter27Packages = super.interpreter27Packages // {
      # Will change foo within bar, baz, qux (as expected).
      foo = super.interpreter27Packages.foo.override { libxml = null; };
    };
  })
  # Duplicate a package set with a custom interpreter.
  (self: super: {
    interpreter27ProfilePackages = super.interpreter27Packages // {
      interpreter = super.interpreter27.override { withProfiler = true; };
    });
  })
]; }

As the scope would be determine by the location of the package within Nixpkgs.  But this future is not yet there, because not every package uses callPackage, and not every dependency comes from self.

Bear with me for a minute.

Q: What is nixpkgs?
A: Roughly, a fixed point over an attrset of packages (functions of type deps -> derivation applied to nixpkgs itself).

Q: What is pkg.override?
A: A way to override deps with an attrset. I would call it overrideArgs, but whatever.

Q: What are pkg.overrideAttrs and pkg.overrideDerivation?
A: Ways to override derivation in the above. I have no idea why we need both, I think overrideAttrs covers all the cases.

Q: What is config.packageOverrides?
A: A way to override nixpkgs by changing the base of the fixed point by merging nixpkgs with another attrset.

Q: What are overlays?
A: A Way to override nixpkgs by introducing arbitrary number of additional steps into the fixed point.

Q: What overrideWithScope of this PR adds?
A: A way to override deps with an attrset, duplicating callPackage filtering.

Q: Why?
A: https://github.com/NixOS/nixpkgs/pull/43828#pullrequestreview-138894720

I'm :+1: on adding this.
Btw, override* functions have two dimensions now: they override either deps or the derivation, and either apply raw attrs or filter them out.

override: deps, raw
overrideWithScope: deps, filter
overrideDerivation: should not exist
overrideAttrs: derivation, raw

I have no idea if overrideAttrs that filters has any use, though.
Btw btw, if we treat nixpkgs as a fixed point over pkgs -> pkgs applied to all-packages.nix (think metanixpkgs with nixpkgs as a single package called nixpkgs that is made by callPackage over itself) with overlays introducing arbitrary steps into the fixed point there, then overlays are just repeated overrides of metanixpkgs without filtering. Which bears two questions:

Wouldn't it be nice to expose the whole nixpkgs as a package that can be overriden?
For instance, this way use-flags will be just such overrides.

What is the equivalent of overlays for packages?

For a bit now, I've been thinking it would be nice if the callPackage mechanism could add an override function that would do the equivalent of adding another overlay and then re-evaluate the package.
The idea would be then be that you could also effect changes in all the dependencies of a package as well for cases where the dependent packages must also have the change or else you will windup with the final executable sucking in two different versions of the same shared library (a typical example that you see in some existing codes is the version of boost or cuda libraries used in its dependencies).
I originally thought this might be what this pull request was, but, upon closer looking, I don't believe so.
I thought I would mention it here though as it does seem there is some significant overlap.
Cheers!  -Tyson
PS:  Technically, I expect, it could make sense to only apply the overlay to runtime dependencies (i.e., not to the build time ones).

Can we move forward with this?

Does this need review? testers? redesign?

I know think https://github.com/NixOS/nixpkgs/pull/51213 is a better approach than this and my first comment's suggestion.

@tomberek as I understand it, there are 2 problems here:

Increased memory usage during evaluation (hm, how much?)
People will start using it, so in case Nixpkgs later decides that @Ericson2314 approach is better, it will be harder to deprecate this

But I'd like to merge this as is. It helps a bit with overrides in overlays.

This pull request has been mentioned on Nix community. There might be relevant details there:
https://discourse.nixos.org/t/wip-rfc-a-new-nixpkgs-frontend-for-language-infrastructures/3447/9

A year later and I had another look at this, because I need a way to override the Nix packages set from within itself.
@nbp, @Ericson2314 and I had a brief discussion last year on IRC as well about this PR
https://logs.nix.samueldr.com/nixos-dev/2018-07-31#1433188;
The main issue I still have is not having a good way to refer to packages in the main set pkgs. We need a solution there. @nbp proposed renaming e.g. these packages by prepending _ or suffixing '.

Is there a target when the pending review will take place?

Still relevant to me.

Relevant to no one else.  I will close this.
Feel free to restart.

Stalled for two years...

This is relevant for #43755.

A five-year-old PR stalled for two years...


Improve tooling support

Added a few tools for support and made the tzdata not depend on gentoo. Updating timezones requires deleting the tzdata-latest.tar.gz, hope that is fine.

 Maybe we can do something in the Makefile that looks for the tarball in the Gentoo dir (and possibly other dirs if something similar exists elsewhere) and if not, download it?
As for the HTTP server, it might be nicer to use python -m SimpleHTTPServer or something like it.

I started with python -m SimpleHTTPServer, but that doesn't redirect all requests back to index.html as you would with apache.  The serve.py code is basically the same thing, but with that additional handler that always uses index.html.
We can certainly turn the thing around to check for gentoo directories first.


iox-#2052 Implement move/copy constructor and assignment for FixedPositionContainer

Pre-Review Checklist for the PR Author

[x] Code follows the coding style of CONTRIBUTING.md
[x] Tests follow the best practice for testing
[x] Changelog updated in the unreleased section including API breaking changes
[x] Branch follows the naming format (iox-123-this-is-a-branch)
[x] Commits messages are according to this guideline

[x] Commit messages have the issue ID (iox-#123 commit text)
[x] Commit author matches Eclipse Contributor Agreement (and ECA is signed)

[x] Update the PR title

Follow the same conventions as for commit messages
Link to the relevant issue

[x] Relevant issues are linked
[x] Add sensible notes for the reviewer
[x] All checks have passed (except task-list-completed)
[ ] All touched (C/C++) source code files from iceoryx_hoofs are added to ./clang-tidy-diff-scans.txt
[ ] Assign PR to reviewer

Notes for Reviewer

Move/copy constructors and assignment for FixedPositionContainer are implemented.
At present, when invoking the move/copy constructor or assignment for FixedPositionContainer, only the move/copy assignment of elements within FixedPositionContainer::m_data gets triggered (without calling the constructor). This is
because the FixedPositionContainer constructor is realized using assignment, mirroring the behavior of vector.
Unit tests are added. To test the newly added code under the following scenarios, you can execute it via build/dust/test/dust_moduletests:

From an empty container
From a single-element container
From a multi-element container
From a full-capacity container

For move cases, the criteria for determining whether FixedPositionContainer::clear() has been executed is based on detecting if the element (ComplexType) has invoked the DTor.

Checklist for the PR Reviewer

[ ] Commits are properly organized and messages are according to the guideline
[ ] Code according to our coding style and naming conventions
[ ] Unit tests have been written for new behavior
[ ] Public API changes are documented via doxygen
[ ] Copyright owner are updated in the changed files
[ ] All touched (C/C++) source code files from iceoryx_hoofs have been added to ./clang-tidy-diff-scans.txt
[ ] PR title describes the changes

Post-review Checklist for the PR Author

[ ] All open points are addressed and tracked via issues

References

Closes #2052

I found a weird behavior when using git. I noticed that using git add -p followed by git commit -s doesn't behave as I expect. It changes the content that I've added.  Am I doing anything wrong?

The test ran on the Linux platform (Ubuntu 22.04 LTS via WSL) for both C++14 and C++17.

@Dennis40816 I don't use git add -p myself so I cannot tell if you did something wrong. Usually I always commit the whole file.
Could you please rebase to master and fix the conflict in the files. I try to start reviewing tomorrow.

With the latest merge it seems there is another merge conflict. Please update again. It also seems that the gcc builds are broken. I guess you need to move the specialization to the .inl file

Ok, I'll switch to g++ 8.4.0 and take a took what's going on.

Only a few minor things left and from my point of view it could be merged. Thanks for being so patient :)
Oh, could you add one more test, e.g. IteratorsAfterMoveWorkAsExpected

source container begin() == end()
destination container iterates until end() <- if I did not overlook something then then it was never checked that ++it results in end()

Sure. Thank you so much for your patience and kindness. Your willingness to accommodate and review my mistakes has been incredibly beneficial.

@Dennis40816 I reviewed your PR as well, and it looks like a great job!
As soon as the requested additional test cases are in, I would approve it.

@elfenpiff  Thank you so much! I am happy to contribute to the project.
Btw, occasionally the tests fail (as with the recent builds and tests), and the reasons are still unclear.

@Dennis40816 can you create an issue with the sporadically failing test names. When we have time we can look into it, except when you are eager to dig deeper. 😀

Btw, occasionally the tests fail (as with the recent builds and tests), and the reasons are still unclear.

I saw only two failures during one CI run. It was the bazel and thread sanitizer and looked like the test file was too large but after triggering the CI again it passed. Did you see more of these? If yes, maybe the file with the tests needs to be split into two.

I saw only two failures during one CI run. It was the bazel and thread sanitizer and looked like the test file was too large but after triggering the CI again it passed.

You're right. But something weird happened.
In build-test-ubuntu-with-thread-sanitizer-clang-latest, all constructors were supposed to be invoked, as the destination container was empty, indicating that constructors should be called initially.
[ RUN      ] FixedPositionContainer_test.UsingCopyCtorFullCapacityContainerPreservesAllElements
/home/runner/work/iceoryx/iceoryx/iceoryx_dust/test/moduletests/test_container_fixed_position_container.cpp:214: Failure
Value of: stats.copyCTor
Expected: is equal to '\n' (10, 0xA)
  Actual: 9 (of type unsigned long)
/home/runner/work/iceoryx/iceoryx/iceoryx_dust/test/moduletests/test_container_fixed_position_container.cpp:216: Failure
Value of: stats.copyAssignment
Expected: is equal to 0
  Actual: 1 (of type unsigned long)
[  FAILED  ] FixedPositionContainer_test.UsingCopyCtorFullCapacityContainerPreservesAllElements (1 ms)

And something similar in build-test-ubuntu-bazel

Since this PR is already quite large and copy_and_move_impl is already a bit complex I think it would be best to add the initialization of m_status in this PR and if you like, you could create a follow up PR to refactor copy_and_move_impl.
I like what you did with the Helper structs and the copy_and_move_impl. Maybe you can come up with a solution which does not need the presence of the assignment operators when the copy/move ctor is used and still share the code.
I haven't fully thought it through but having something like this
enum class SpecialMemberOperation {
    CopyConstructor,
    CopyAssignment,
    MoveConstructor,
    MoveAssignment,
};

Could be used as template parameter for the Helper instead of the boolean and this could then also be used for copy_and_move_impl. If this works out, the Helper structs could be moved to the design module and a small design document could document how they can be used with a copy_and_move_impl.
What do you think?

Oh, and I'm a bit disappointed that the undefined behavior sanitizer did not find the issue with the uninitialized access. I was hoping that it could find issues like this.

Since this PR is already quite large and copy_and_move_impl is already a bit complex I think it would be best to add the initialization of m_status in this PR and if you like, you could create a follow up PR to refactor copy_and_move_impl.
I like what you did with the Helper structs and the copy_and_move_impl. Maybe you can come up with a solution which does not need the presence of the assignment operators when the copy/move ctor is used and still share the code.
I haven't fully thought it through but having something like this
enum class SpecialMemberOperation {
    CopyConstructor,
    CopyAssignment,
    MoveConstructor,
    MoveAssignment,
};

Could be used as template parameter for the Helper instead of the boolean and this could then also be used for copy_and_move_impl. If this works out, the Helper structs could be moved to the design module and a small design document could document how they can be used with a copy_and_move_impl.
What do you think?

Seems Interesting! I'll try to create a PR(should the issue be created first or not?) about refactoring copy_and_move_impl. We can discuss about the details in that issue or PR.

Do you mean something like below?
// transfer src data to destination
    for (; rhs_it.to_index() != Index::INVALID; ++i, ++rhs_it)
    {
        if (is_move && m_status[i] == SlotStatus::USED)
        // ...

       for (; i < CAPACITY; ++i)
    {
        if (is_move && m_status[i] == SlotStatus::USED)
        {
            m_data[i].~T();
        }

Sounds Interesting! I'll try to create a PR(should the issue be created first or not?) about refactoring copy_and_move_impl. We can discuss about the details in that issue or PR.

It's up to you. You can either as additional PR of #2052 and if you are successful create another issue to move the helper classes to the design module and write a small design document. Alternatively, you could create the issue for the helper classes and try to make it work (including a small design document) and afterward adjust the copy_and_move_impl for this class. Whatever fits your workflow best

Do you mean something like below?
// transfer src data to destination
    for (; rhs_it.to_index() != Index::INVALID; ++i, ++rhs_it)
    {
        if (is_move && m_status[i] == SlotStatus::USED)
        // ...

       for (; i < CAPACITY; ++i)
    {
        if (is_move && m_status[i] == SlotStatus::USED)
        {
            m_data[i].~T();
        }

I'm a bit lost :sweat_smile:
This is code for copy_and_move_impl, right? It won't really work since is_move wil also be true for the move assignment and then the move assignment operator should be used. The issue exists only for the copy and move ctor.
The more I think about it, the more I come to the conclusion that the code of the default ctor needs to be moved to something like init and this should be called at the top of the copy and move ctors. With this change we can merge the PR. Then we have to think if it is possible have some elegant solution for copy_and_move_impl without calling init beforehand

Oh my god. You're right. I don't even understand what I was thinking just now. I think I might need to make these modifications tomorrow. I'm sorry for not thinking it through. I think I will follow your suggestion and use an init function (which might do the same thing as the constructor function). Thank you for the reminder.

Oh my god. You're right. I don't even understand what I was thinking just now. I think I might need to make these modifications tomorrow. I'm sorry for not thinking it through. I think I will follow your suggestion and use an init function (which might do the same thing as the constructor function). Thank you for the reminder.

No problem :)
I guess it is already late at your location. Take your time.

@Dennis40816 congratulations to your first merged PR in iceoryx. Thanks for the patience with me and I'm looking forward to more contributions from you :)


Scikit learn estimator optimization

Is there a possibility to fine tune Scikit estimator let's say RandomForest using swarmops?

@Hvass-Labs  so there's no possibility to even use particle swarm optimization in for hyperparameters tunning in python?


Pressing Ctrl P twice opens two pop ups

Description of your problem
What steps will reproduce the problem?

Press Ctrl-P
Press Ctrl-P again

What is the expected output? What do you see instead?
The first Ctrl-P opens the selection pop up. The second Ctrl-P should (preferably) close it while open, or at least do nothing. Instead, it seems a second pop up is opened over the first one (you then have to press Esc twice to close both of them).
Versions and main components

Spyder Version: Spyder 3.0.0.dev0 (from spyder-ide conda channel)
Python Version: 3.5.2
Operating system: Linux

Dependencies
Please go to the menu entry Help > Optional Dependencies (or
Help > Dependencies), press the button Copy to clipboard
and paste the contents below:
jedi >=0.8.1     :  0.9.0 (OK)
matplotlib >=1.0 :  None (NOK)
nbconvert >=4.0  :  4.2.0 (OK)
numpy >=1.7      :  None (NOK)
pandas >=0.13.1  :  None (NOK)
pep8 >=0.6       :  1.7.0 (OK)
psutil >=0.3     :  4.3.0 (OK)
pyflakes >=0.6.0 :  1.2.3 (OK)
pygments >=2.0   :  2.1.3 (OK)
qtconsole >=4.2.0:  4.2.1 (OK)
rope >=0.9.4     :  0.9.4-1 (OK)
sphinx >=0.6.6   :  1.4.1 (OK)
sympy >=0.7.3    :  None (NOK)

Thanks for reporting @pitrou! I verified that the error is present in master.

You need to focus dockwidgets more often than closing them. The popup on the other hand in constantly being shown and hidden, so I feel it's a more natural choice to toggle it with the shortcut.

I feel it's a more natural choice to toggle it with the shortcut

I agree with @Nodd on this one. Pressing Ctrl+P should toggle the switcher between visible and hidden.


Special workspaces occasionally appear on screen after wakeup/dpms wakeup

Hyprland Version
 System/Version info
Hyprland, built from branch main at commit d7d333d162da2d3fc852b2c7a3faa2709440cefa dirty (opengl: apply box rot to projections).
Date: Sun Jan 7 18:51:08 2024
Tag: v0.34.0-31-gd7d333d1

flags: (if any)

Bug or Regression?
Bug
Description
As I said in the title, special workspaces will occasionally appear on my desktop, after the mentioned events. They are stacked on top of each other, but this seems to be only visual, because I can't click on them, and they don't go away until I toggle them twice (when I toggle one, it disappears, and comes back again, becoming interactive).
I have 2 monitors, and they appear on my primary monitor, if that matters.
How to reproduce
Not consistent

hyprctl dispatch dpms off or systemctl suspend (unless you don't use systemd)
Turn the screen on again, or wake the computer up
Should see the special workspaces

Crash reports, logs, images, videos
No response

Just reproduced this on latest
 System/Version info
Hyprland, built from branch main at commit 252aaaecfa3b57eb07822177a8bf3b609aff7115 dirty (input: add special_fallthrough).
Date: Tue Jan 9 13:17:55 2024
Tag: v0.34.0-35-g252aaaec

flags: (if any)

I am having the same issue.
I have 2 monitors, and if I turn off one of the monitor and turn it back on, then the special workspace is activated, some times all of the speical workspaces are activated and I have to toggle all them to take them down. And for me I'm not able to focus on the special workspaces that are toggled after the monitor is turned on. I have to toggle them off and on to make them foucs again.
System/Version info:
Hyprland, built from branch  at commit c4365f20ed8ff0dd480b7ed7cf1bfff1a0b6911a  ().
Date: 2024-01-17
Tag:

flags: (if any)

System Information:
System name: Linux
Node name: gza
Release: 6.6.9-xanmod1
Version: #1-NixOS SMP PREEMPT_DYNAMIC Tue Jan  1 00:00:00 UTC 1980

GPU information:
24:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU104 [GeForce RTX 2070 SUPER] [10de:1e84] (rev a1) (prog-if 00 [VGA controller])
2f:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 23 [Radeon RX 6600/6600 XT/6600M] [1002:73ff] (rev c1) (prog-if 00 [VGA controller])

os-release: BUG_REPORT_URL="https://github.com/NixOS/nixpkgs/issues"
BUILD_ID="24.05.20240115.c3e128f"
DOCUMENTATION_URL="https://nixos.org/learn.html"
HOME_URL="https://nixos.org/"
ID=nixos
LOGO="nix-snowflake"
NAME=NixOS
PRETTY_NAME="NixOS 24.05 (Uakari)"
SUPPORT_URL="https://nixos.org/community.html"
VERSION="24.05 (Uakari)"
VERSION_CODENAME=uakari
VERSION_ID="24.05"

plugins:

I'm also seeing this running hyprland v.0.35.0, without any plugins.

I haven't had this happen in ages, closing. I can reopen it if someone else has.


add status label to pipelinerun/taskrun schedule metrics

While tracking our scheduling alert appstudio grafana panel in RHTAP prod, saw some evidence that failed runs overtly affected the ration calculation, so adding the ability to filter out failed runs.

merging it so that we can load it to staging in time for Gabe to use those labels.


*.props files not avialable during the build

A check at the build logs reveals that the *.props are not included in the build:
job/28598824711
Warning: /home/runner/work/Qowaiv/Qowaiv/src/Qowaiv.Diagnostics.Contracts/Qowaiv.Diagnostics.Contracts.csproj(1,2): warning Proj0200: Define the <IsPackable> node explicitly. (https://dotnet-project-file-analyzers.github.io/rules/Proj0200.html) [TargetFramework=net8.0]
Warning: /home/runner/work/Qowaiv/Qowaiv/src/Qowaiv.Diagnostics.Contracts/Qowaiv.Diagnostics.Contracts.csproj(1,2): warning Proj0400: Define the <IsPublishable> node explicitly. (https://dotnet-project-file-analyzers.github.io/rules/Proj0400.html) [TargetFramework=net8.0]
Warning: /home/runner/work/Qowaiv/Qowaiv/src/Qowaiv.Diagnostics.Contracts/Qowaiv.Diagnostics.Contracts.csproj(1,2): warning Proj0010: Define the <OutputType> node explicitly. (https://dotnet-project-file-analyzers.github.io/rules/Proj0010.html) [TargetFramework=net8.0]
Warning: /home/runner/work/Qowaiv/Qowaiv/src/Qowaiv.Diagnostics.Contracts/Qowaiv.Diagnostics.Contracts.csproj(1,2): warning Proj0202: Define the <Description> or <PackageDescription> node explicitly or define the <IsPackable> node with value 'false'. (https://dotnet-project-file-analyzers.github.io/rules/Proj0202.html) [TargetFramework=net8.0]
Warning: /home/runner/work/Qowaiv/Qowaiv/src/Qowaiv.Diagnostics.Contracts/Qowaiv.Diagnostics.Contracts.csproj(1,2): warning Proj0203: Define the <Authors> node explicitly or define the <IsPackable> node with value 'false'. (https://dotnet-project-file-analyzers.github.io/rules/Proj0203.html) [TargetFramework=net8.0]
Warning: /home/runner/work/Qowaiv/Qowaiv/src/Qowaiv.Diagnostics.Contracts/Qowaiv.Diagnostics.Contracts.csproj(1,2): warning Proj0204: Define the <PackageTags> node explicitly or define the <IsPackable> node with value 'false'. (https://dotnet-project-file-analyzers.github.io/rules/Proj0204.html) [TargetFramework=net8.0]
Warning: /home/runner/work/Qowaiv/Qowaiv/src/Qowaiv.Diagnostics.Contracts/Qowaiv.Diagnostics.Contracts.csproj(1,2): warning Proj0205: Define the <RepositoryUrl> node explicitly or define the <IsPackable> node with value 'false'. (https://dotnet-project-file-analyzers.github.io/rules/Proj0205.html) [TargetFramework=net8.0]
Warning: /home/runner/work/Qowaiv/Qowaiv/src/Qowaiv.Diagnostics.Contracts/Qowaiv.Diagnostics.Contracts.csproj(1,2): warning Proj0206: Define the <PackageProjectUrl> node explicitly or define the <IsPackable> node with value 'false'. (https://dotnet-project-file-analyzers.github.io/rules/Proj0206.html) [TargetFramework=net8.0]
Warning: /home/runner/work/Qowaiv/Qowaiv/src/Qowaiv.Diagnostics.Contracts/Qowaiv.Diagnostics.Contracts.csproj(1,2): warning Proj0207: Define the <Copyright> node explicitly or define the <IsPackable> node with value 'false'. (https://dotnet-project-file-analyzers.github.io/rules/Proj0207.html) [TargetFramework=net8.0]
Warning: /home/runner/work/Qowaiv/Qowaiv/src/Qowaiv.Diagnostics.Contracts/Qowaiv.Diagnostics.Contracts.csproj(1,2): warning Proj0209: Define the <PackageReadmeFile> node explicitly or define the <IsPackable> node with value 'false'. (https://dotnet-project-file-analyzers.github.io/rules/Proj0209.html) [TargetFramework=net8.0]
Warning: /home/runner/work/Qowaiv/Qowaiv/src/Qowaiv.Diagnostics.Contracts/Qowaiv.Diagnostics.Contracts.csproj(1,2): warning Proj0212: Define the <PackageIcon> node explicitly or define the <IsPackable> node with value 'false'. (https://dotnet-project-file-analyzers.github.io/rules/Proj0212.html) [TargetFramework=net8.0]
Warning: /home/runner/work/Qowaiv/Qowaiv/src/Qowaiv.Diagnostics.Contracts/Qowaiv.Diagnostics.Contracts.csproj(1,2): warning Proj0213: Define the <PackageIconUrl> node explicitly or define the <IsPackable> node with value 'false'. (https://dotnet-project-file-analyzers.github.io/rules/Proj0213.html) [TargetFramework=net8.0]
Warning: /home/runner/work/Qowaiv/Qowaiv/src/Qowaiv.Diagnostics.Contracts/Qowaiv.Diagnostics.Contracts.csproj(1,2): warning Proj0210: Define the <PackageLicenseExpression> or <PackageLicenseFile> node explicitly or define the <IsPackable> node with value 'false'. (https://dotnet-project-file-analyzers.github.io/rules/Proj0210.html) [TargetFramework=net8.0]
Warning: /home/runner/work/Qowaiv/Qowaiv/src/Qowaiv.Diagnostics.Contracts/Qowaiv.Diagnostics.Contracts.csproj(1,2): warning Proj1000: Use the .NET project file analyzers. (https://dotnet-project-file-analyzers.github.io/rules/Proj1000.html) [TargetFramework=net8.0]
Warning: /home/runner/work/Qowaiv/Qowaiv/src/Qowaiv.Diagnostics.Contracts/Qowaiv.Diagnostics.Contracts.csproj(1,2): warning Proj1003: Add SonarAnalyzer.CSharp. (https://dotnet-project-file-analyzers.github.io/rules/Proj1003.html) [TargetFramework=net8.0]

This leads to a lot of noise. On top of that, these artifacts can not be used when we switch to auto deployments.

I've looked into this, and it seems that dotnet-project-file-analyzers has issues with backslashes \ in the path while on Linux. Can reproduce these warnings on WSL with Ubuntu 22.04, and replacing every \ with a / in the every .csproj file resolves the warnings. But ideally, this should be resolved in dotnet-project-file-analyzers.

@renzo-baasdam I was not aware that this was the problem here. That being said: there is already a suggestion to implement a rule that checks for that.

Issue has been resolved.


Phone Authentication Error

Description
I am currently experiencing a phone authentication error that I was not before. Please see error below:
Optional(Error Domain=FIRAuthErrorDomain Code=17999 "An internal error has occurred, print and inspect the error details for more information." UserInfo={NSLocalizedDescription=An internal error has occurred, print and inspect the error details for more information., FIRAuthErrorUserInfoNameKey=ERROR_INTERNAL_ERROR, NSUnderlyingError=0x1250c6d40 {Error Domain=FIRAuthInternalErrorDomain Code=3 "(null)" UserInfo={NSUnderlyingError=0x10767bbd0 {Error Domain=com.google.HTTPStatus Code=503 "(null)" UserInfo={data={length = 207, bytes = 0x7b0a2020 22657272 6f72223a 207b0a20 ... 5d0a2020 7d0a7d0a }, data_content_type=application/json; charset=UTF-8}}, FIRAuthErrorUserInfoDeserializedResponseKey={
code = 503;
errors =     (
{
domain = global;
message = "Error code: 39";
reason = backendError;
}
);
message = "Error code: 39";
}}}})
Code:
let configuration = Configuration(requestCode: { phone, completion in
PhoneAuthProvider.provider().verifyPhoneNumber(phone, uiDelegate: nil, completion: completion)
}, signIn: { verificationID, verificationCode, completion in
let credential = PhoneAuthProvider.provider().credential(withVerificationID: verificationID, verificationCode: verificationCode)
Auth.auth().signIn(with: credential) { _, error in completion(error) }
})
Billing level:

Reproducing the issue
No response
Firebase SDK Version
11.4
Xcode Version
15.1
Installation Method
Swift Package Manager
Firebase Product(s)
Analytics, Authentication, Crashlytics, Firestore, Functions, Messaging, Storage
Targeted Platforms
iOS
Relevant Log Output
No response
If using Swift Package Manager, the project's Package.resolved

Expand Package.resolved snippet

Replace this line with the contents of your Package.resolved.

If using CocoaPods, the project's Podfile.lock

Expand Podfile.lock snippet

Replace this line with the contents of your Podfile.lock!

Thanks for filing, @ElrB. Per checking, this is possibly related to a blocked region or carrier. Since I don't have a visibility to project specific, please file a ticket through our support channel for investigation. That being said, I'll go ahead and close the issue.


syncShutdownGracefully() does not wait for all connections?

I'm probably missing something obvious: Just wondering if it's possible to run an async operation off the main event loop in such a way that syncShutdownGracefully() or some other mechanism will safely wait for it to complete/finish before the server process ends? Right now it only seems to wait for the things are are actually running on the event loop. It would be nice if it waited for all open connections to return a response before the server process ends?  I would like to shutdown the server and ensure that it does not drop any client connections. Is there a way to accomplish that currently? Thanks!
class Responder: HTTPServerDelegate {
    func respond(to req: HTTPRequest, on channel: Channel) -> EventLoopFuture<HTTPResponse> {
        let p = channel.eventLoop.makePromise(of: HTTPResponse.self)

	// if the server is closed/shutdown while getDataFrom3rdPartyService() is running on separate queue, so as not to block the event loop thread, the client connection will be dropped !!??
        DispatchQueue.global().async {
            let res = getDataFrom3rdPartyService()
            p.succeed(res)
        }

        return p.futureResult
    }
}

this NIO example implements what's asked here: https://github.com/apple/swift-nio-extras/blob/master/Sources/HTTPServerWithQuiescingDemo/main.swift

Thank you!

@jcl-dev does Vapor's http already use this technique? The above link is only an example from the NIO examples, not sure if Vapor uses it. Just wanted to point out that this is not too hard with NIO.

I appreciate it.
The vapor/http HTTPServer class appears to use ServerQuiescingHelper, but it does not appear to work in my testing if the request is processing off the channel.eventLoop. When I send a SIGTERM to my server process, it intercepts it and calls shutdown() on the HTTPServer instance. I'm probably just doing it wrong.
let group = MultiThreadedEventLoopGroup(numberOfThreads: System.coreCount)
let server = HTTPServer(
    configuration: .init(
        hostname: "<IP_ADDRESS>",
        port: port,
        maxBodySize: 10_000_000,
        supportVersions: .init([.one])),
    on: group)
_ = try! server.start(delegate: Responder()).wait()

// Allow the process to intercept the default kill signal which is SIGTERM in order to exit gracefully
signal(SIGTERM, SIG_IGN)
let sigterm = DispatchSource.makeSignalSource(signal: SIGTERM)
sigterm.setEventHandler {
    log.info("received SIGTERM")
    _ = server.shutdown()
}
sigterm.resume()

// Wait for the server to close (indefinitely).
try server.onClose.wait()

// Make sure to shutdown the group gracefully when the process exits.
try? group.syncShutdownGracefully()
log.info("graceful shutdown complete")

oh yes, it does indeed. If it doesn't work for you, then there must be some bug in there.
Can you confirm the NIO example does work for you?

Yes, I can confirm the NIO example you gave works as-is, and also works when modified to use DispatchQueue.global().asyncAfter to fulfill a promise:
let p = context.channel.eventLoop.makePromise(of: String.self)
DispatchQueue.global().asyncAfter(deadline: .now() + 10) {
    p.succeed("Fulfill!")
}
p.futureResult.whenSuccess { succeedString in
    buffer.clear()
    buffer.writeStaticString("done with the request now\n")
    buffer.writeString("\(succeedString)\n")
    context.write(self.wrapOutboundOut(.body(.byteBuffer(buffer))), promise: nil)
    context.writeAndFlush(self.wrapOutboundOut(.end(nil)), promise: nil)
}

thanks for confirming this!

The latest HTTP server stuff is here now: https://github.com/vapor/vapor/blob/master/Sources/Vapor/Server/HTTPServer.swift
Last I tested this (using telnet), server quiesce + sigterm was working as expected. There should really be a unit test for this, though.

Ah, thanks!


Iynere add geckodriver

Checklist

[X] I've run make from the root directory to see all Dockerfiles are created successfully.
[n/a] I've updated the documentation if necessary.

Motivation and Context
This PR relates to issue #125.

adds geckodriver to all -browsers images

tested jq, curl, geckodriver syntax locally
ran make to ensure Dockerfiles can be created successfully

Description

curl GitHub API to get latest linux64 geckodriver release
download latest linux64 geckodriver release
move to /usr/local/bin & test version #

LGTM!

trying to load the firefox browser via node selenium sdk just hangs. any ideas?

@jonathanong not sure, if you open a support ticket with more info & a link to a build where you're running into this, someone on support can definitely help you out though


Add Platform Build Number to Output

In the critical/daily runs, it would be nice to see a platform build number.
Example here for OWA: https://blogs.technet.microsoft.com/sammykailini/2013/09/20/how-to-find-the-version-or-build-number-for-an-office-web-apps-2013-farm/

Done, in next release


[v2.6] Refactor RKE1 into separate test functions

Issue:  Refactor RKE1 provisioning test to use separate test functions for additional tests

Problem

Recently, we refactored the hosted provisioning tests to do two of the following:

Rework redundant provisioning code into a private method
Separate provisioning test and additional test cases into their own, unique test functions

This greatly helps moving forward for internal CI/CD pipeline and overall helps to better organize our ever-growing test framework. As such, the same approach has been made to the RKE1 provisioning tests.
Once this is done, the same effort will be done for RKE2/K3s, to ensure full parity across the board.
Solution

Like with the hosted provisioning tests, the redundant provisioning code for node provider and custom clusters has been put into a single private method. The now separated unique test functions reference this private method and only keep unique code in their respective functions.
Testing

Engineering Testing
Manual Testing

Locally tested static and dynamic input functions for both node provider and custom cluster.
Automated Testing

Jenkins runs will be provided offline to reviewers.

Since you're already working RKE1 Markus, can you fix this issue with with node pools. In azure node pools need to have unique names, and the way you designed causes failures. It's simple fix just need to add this line. In case your PR gets Merged before my code coverage PR. https://github.com/rancher/rancher/blob/dd234230eacc910e37bd5df62e9005b324951663/tests/framework/extensions/rke1/nodepools/nodepools.go#L90

@markusewalker Since you're already working on RKE1, can you fix this issue with with node pools. In azure node pools need to have unique names, and the way you designed it causes failures. It's a simple fix just need to add this line. In case your PR gets Merged before my code coverage PR.
https://github.com/rancher/rancher/blob/dd234230eacc910e37bd5df62e9005b324951663/tests/framework/extensions/rke1/nodepools/nodepools.go#L90

@igomez06 sure thing, I'll fix that as well. Seems to be some conflicting files issues with my branch not pulling latest Corral changes Caleb did recently...I'll work through that while fixing the auto-rke1 naming issue.

Caleb's PR broke some our tests. Ugur has a PR going today to fix it. You can wait until then to make it easier for you. I just added you to review as well. https://github.com/rancher/rancher/pull/40107


[Improvement-11678][API] Improvement the error message when batch delete workflow

this close #11678
The effect is as follows：

Codecov Report

Merging #11682 (dfab4f4) into dev (45ebd52) will decrease coverage by 0.03%.
The diff coverage is 9.37%.

@@             Coverage Diff              @@
##                dev   #11682      +/-   ##
============================================
- Coverage     39.60%   39.56%   -0.04%
+ Complexity     4696     4694       -2
============================================
  Files          1014     1014
  Lines         37959    37971      +12
  Branches       4243     4243
============================================
- Hits          15032    15024       -8
- Misses        21318    21341      +23
+ Partials       1609     1606       -3

Impacted Files
Coverage Δ

...er/api/controller/ProcessDefinitionController.java
58.33% <0.00%> (+9.33%)
:arrow_up:

...api/service/impl/ProcessDefinitionServiceImpl.java
31.64% <0.00%> (-0.67%)
:arrow_down:

.../org/apache/dolphinscheduler/api/enums/Status.java
100.00% <100.00%> (ø)

...olphinscheduler/rpc/remote/NettyServerHandler.java
69.04% <0.00%> (-11.91%)
:arrow_down:

...eduler/server/worker/task/WorkerHeartBeatTask.java
68.08% <0.00%> (-8.52%)
:arrow_down:

...che/dolphinscheduler/api/python/PythonGateway.java
17.22% <0.00%> (ø)

...er/master/dispatch/host/assign/RandomSelector.java
83.33% <0.00%> (+5.55%)
:arrow_up:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

related pr: #12177