Memory profiling

@wigging could give this a try https://github.com/bloomberg/memray

Already working on it. I ran memory-profiler and need to collect the results. I'll try memray too. Any particular pack configurations or parameters you want me to run?

Not really, just see how it scales with number of processors

Below are some memory profiling results. I used the Memray and memory_profiler tools to log the memory usage during various pack simulations. Information about the system I ran the tests on is here:
OS: Ubuntu 20.04.4 LTS x86_64
CPU: Intel Xeon E5-2680 v3 (20) @ 2.499GHz
GPU: NVIDIA Tesla K40m
Memory: 582MiB / 58269MiB

Small pack tests, 16p2s, 1 cycle
Memory tests for the pack_16p2s.py example using the Casadi and Ray solvers with 2, 5, 10, and 20 CPU cores and 58 GB of RAM. Experiment is a single charge/discharge cycle with a period of 10 seconds. Results are from the Memray profiler.

Run
Cores
Solver
Total memory

1
2
casadi
309 MB

2
5
casadi
278 MB

3
10
casadi
247 MB

4
20
casadi
248 MB

5
2
ray
16.9 GB

6
5
ray
17.0 GB

7
10
ray
16.9 GB

8
20
ray
16.9 GB

Memory tests for the pack_16p2s.py example using the Casadi and Ray solvers with 20 CPU cores and 58 GB of RAM. Experiment is a single charge/discharge cycle with a period of 10 seconds. Results are from the Memray profiler.

Run
Cores
Description
Total memory

1
20
casadi
280 MB

2
20
casadi
312 MB

3
20
casadi
280 MB

4
20
ray
16.9 GB

5
20
ray
17.1 GB

6
20
ray
17.0 GB

Small pack tests, 16p2s, 10 cycles
Memory tests for the pack_16p2s.py example using the Casadi and Ray solvers with 20 CPU cores and 58 GB of RAM. Experiment is a single charge/discharge cycle with a period of 10 seconds. The experiment is repeated ten times (10 cycles). Results are from the Memray profiler.

Run
Cores
Description
Total memory

1
20
casadi
316 MB

2
20
casadi
284 MB

3
20
casadi
316 MB

4
20
ray
16.9 GB

5
20
ray
17.0 GB

6
20
ray
16.9 GB

Small pack tests, 16p2s, 10 cycles, SEI degradation
Memory tests for the pack_16p2s_sei.py example using the Casadi and Ray solvers with 20 CPU cores and 58 GB of RAM. Experiment is a single charge/discharge cycle with a period of 10 seconds. The experiment is repeated ten times (10 cycles). SEI degradation is implemented for capacity loss. Results are from the Memray profiler.

Run
Cores
Description
Total memory

1
20
casadi
329 MB

2
20
casadi
329 MB

3
20
casadi
330 MB

4
20
ray
17.0 GB

5
20
ray
16.9 GB

6
20
ray
17.0 GB

Medium pack tests, 32p20s, 1 cycle
Memory tests for the pack_32p20s.py example using the Casadi and Ray solvers with 2, 5, 10, 20 CPU cores and 58 GB of RAM. Experiment is a single charge/discharge cycle with a period of 10 seconds. Results from the Memray profiler.

Run
Cores
Solver
Total memory

2
5
casadi
353 MB

3
10
casadi
352 MB

4
20
casadi
320 MB

6
5
ray
16.9 GB

7
10
ray
16.9 GB

8
20
ray
17.0 GB

Results from the memory_profiler tool.

Run
Cores
Solver
Total memory

2
5
casadi
372 MiB

3
10
casadi
371 MiB

4
20
casadi
371 MiB

6
5
ray
258 MiB

7
10
ray
258 MiB

8
20
ray
259 MiB

Megapack tests, 1 cycle
Memory tests for the megapack.py example using the Casadi and Ray solvers with 20 CPU cores and 58 GB of RAM. Experiment is a single charge/discharge cycle and period is set to 30 seconds. Results are from the Memray profiler. Memray crashed on the Casadi runs.

Run
Cores
Description
Total memory

1
20
casadi
x

2
20
casadi
x

3
20
casadi
x

4
20
ray
16.9 GB

5
20
ray
16.8 GB

6
20
ray
705 MB

Well probably we would see a trend with fewer cycles. Going from 1 to 10 increased by 20mb it would seem for the small pack. Would be interesting to see the increase on the medium pack

Did more tests with the memory_profiler tool for the small pack. I also have some medium pack results which I'll share in the next comment.
Small pack tests, 16p2s, 1 cycle
Memory_profiler results for pack_16p2s.py. Experiment is a single charge/discharge cycle. Period is 10 seconds. Top plot is casadi and bottom plot is ray.

Run
Cores
Description
Total memory

1
20
casadi
285 MiB

2
20
ray
245 MiB

Small pack tests, 16p2s, 10 cycles
Memory_profiler results for pack_16p2s.py. Experiment is a single charge/discharge for 10 cycles. Period is 10 seconds. Top plot is casadi and bottom plot is ray.

Run
Cores
Description
Total memory

1
20
casadi
325 MiB

2
20
ray
253 MiB

Here are the medium pack results.
Medium pack test, 32p20s, 100 cycles
Memory_profiler results for pack_32p20s.py at 20 second intervals. Experiment is a single charge/discharge for 100 cycles. Period is 30 seconds. Top plot is casadi and bottom plot is ray.

Run
Cores
Solver
Total memory

1
20
casadi
872 MiB

2
20
ray
758 MiB

Medium pack test, 32p20s, 1000 cycles
Memory_profiler results for pack_32p20s.py at 20 second intervals. Experiment is a single charge/discharge for 1000 cycles. Period is 30 seconds. Top plot is casadi and bottom plot is ray.

Run
Cores
Solver
Total memory

1
20
casadi
5.78 GB

2
20
ray
5.66 GB

Thanks @wigging does it break down what variables are contributing at all?

The memory_profiler tool just gives total memory over time and it works with the Casadi and Ray liionpack solvers. Memray shows how the memory is allocated but I can't get it to work properly with the Ray solver. So my next test is to use Memray with the Casadi solver and see if I can tell what is increasing the memory.

Here are more results using the memory_profiler tool. Compare these to the Memray results in the next comment.
Medium pack test, 32p20s, 10 cycles
Memory_profiler results for pack_32p20s.py at 10 second intervals. Experiment is a single charge/discharge for 10 cycles. Period is 30 seconds. Plot is casadi results.

Run
Cores
Solver
Total memory

1
20
casadi
405 MiB

Here are results from the Memray profiler.
Medium pack test, 32p20s, 10 cycles
Memray results for pack_32p20s.py. Experiment is a single charge/discharge for 10 cycles. Period is 30 seconds. Memray ran out of storage space for experiments with more than 10 cycles.

Run
Cores
Solver
Total memory

1
20
casadi
357 MB

Memray stats results.
📏 Total allocations:
        146343

📦 Total memory allocated:
        356.778MB

📊 Histogram of allocation size:
        min: 1.000B
        -------------------------------------------
        < 5.000B   :   49 ▇
        < 33.000B  : 1864 ▇▇▇▇▇▇
        < 191.000B : 2395 ▇▇▇▇▇▇▇
        < 1.076KB  : 8954 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇
        < 6.206KB  : 6295 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇
        < 35.766KB :  751 ▇▇▇
        < 206.112KB:   88 ▇
        < 1.160MB  :  398 ▇▇
        < 6.684MB  :    7 ▇
        <=38.521MB :    6 ▇
        -------------------------------------------
        max: 38.521MB

📂 Allocator type distribution:
         MALLOC: 19830
         REALLOC: 731
         MMAP: 177
         CALLOC: 70

🥇 Top 5 largest allocating locations (by size):
        - get:/home/cades/miniconda3/envs/lipack/lib/python3.9/site-packages/casadi/casadi.py:6354 -> 38.521MB
        - _horzcat:/home/cades/miniconda3/envs/lipack/lib/python3.9/site-packages/casadi/casadi.py:20190 -> 33.018MB
        - spsolve:/home/cades/miniconda3/envs/lipack/lib/python3.9/site-packages/scipy/sparse/linalg/_dsolve/linsolve.py:203 -> 32.000MB
        - dot:<__array_function__ internals>:180 -> 32.000MB
        - solve:/home/cades/liionpack/liionpack/solvers.py:185 -> 30.777MB

🥇 Top 5 largest allocating locations (by number of allocations):
        - __init__:/home/cades/miniconda3/envs/lipack/lib/python3.9/site-packages/pybamm/solvers/solution.py:93 -> 17822
        - integrator:/home/cades/miniconda3/envs/lipack/lib/python3.9/site-packages/casadi/casadi.py:14436 -> 12788
        - load_default_certs:/home/cades/miniconda3/envs/lipack/lib/python3.9/ssl.py:575 -> 10094
        - get:/home/cades/miniconda3/envs/lipack/lib/python3.9/site-packages/casadi/casadi.py:6354 -> 8962
        - _horzcat:/home/cades/miniconda3/envs/lipack/lib/python3.9/site-packages/casadi/casadi.py:20190 -> 7680

Memray tree results.
Allocation metadata
-------------------
Command line arguments: 'pack_32p20s.py'
Peak memory size: 356.778MB
Number of allocations: 528176676

Biggest 10 allocations:
-----------------------
📂 203.743MB (100.00 %) <ROOT>
└── [[3 frames hidden in 2 file(s)]]
    └── 📂 203.743MB (100.00 %) _run_code  /home/cades/miniconda3/envs/lipack/lib/python3.9/runpy.py:87
        ├── [[1 frames hidden in 1 file(s)]]
        │   └── 📂 198.737MB (97.54 %) solve  /home/cades/liionpack/liionpack/solver_utils.py:425
        │       ├── [[2 frames hidden in 1 file(s)]]
        │       │   └── 📂 77.018MB (37.80 %) step  /home/cades/liionpack/liionpack/solvers.py:81
        │       │       ├── [[2 frames hidden in 2 file(s)]]
        │       │       │   └── 📄 38.521MB (18.91 %) get  /home/cades/miniconda3/envs/lipack/lib/python3.9/site-packages/casadi/casadi.py:6354
        │       │       ├── [[5 frames hidden in 3 file(s)]]
        │       │       │   └── 📄 33.018MB (16.21 %) _horzcat  /home/cades/miniconda3/envs/lipack/lib/python3.9/site-packages/casadi/casadi.py:20190
        │       │       └── [[2 frames hidden in 2 file(s)]]
        │       │           └── 📄 5.480MB (2.69 %) call  /home/cades/miniconda3/envs/lipack/lib/python3.9/site-packages/casadi/casadi.py:12324
        │       ├── [[2 frames hidden in 2 file(s)]]
        │       │   └── 📄 32.000MB (15.71 %) spsolve
        │       │       /home/cades/miniconda3/envs/lipack/lib/python3.9/site-packages/scipy/sparse/linalg/_dsolve/linsolve.py:203
        │       ├── [[1 frames hidden in 1 file(s)]]
        │       │   └── 📂 38.424MB (18.86 %) setup_actors  /home/cades/liionpack/liionpack/solvers.py:453
        │       │       ├── [[9 frames hidden in 7 file(s)]]
        │       │       │   └── 📄 32.000MB (15.71 %) dot  <__array_function__ internals>:180
        │       │       └── [[2 frames hidden in 2 file(s)]]
        │       │           └── 📄 6.424MB (3.15 %) map  /home/cades/miniconda3/envs/lipack/lib/python3.9/site-packages/casadi/casadi.py:12565
        │       ├── 📄 30.777MB (15.11 %) solve  /home/cades/liionpack/liionpack/solvers.py:185
        │       ├── 📄 10.259MB (5.04 %) solve  /home/cades/liionpack/liionpack/solvers.py:183
        │       └── 📄 10.259MB (5.04 %) solve  /home/cades/liionpack/liionpack/solvers.py:184
        └── [[57 frames hidden in 11 file(s)]]
            └── 📄 5.006MB (2.46 %) _compile_bytecode  <frozen importlib._bootstrap_external>:647

Here is the HTML file for the flame graph generated from Memray.
flamegraph-32p20s-10cyc-casadi.html.zip

The flame graph represents allocations contributing to peak memory usage. According to the flame graph for the 32p20s 10 cycles test using the Casadi solver, most of the memory usage is associated with the following liionpack functions, methods, and properties:

self.setup_actors()
solve_circuit_vectorized()
self.step_actors()
self.output

Would it be easy to run the same script for twice as long and compare the output to see which objects grew? We could also just try throwing in a bunch of del statements to objects that could be the issue to see what works. I've never really understood how garbage collection works - especially for parallel processes but potentially as the processes are getting continuously used rather than destroyed that might be doing something gc is not expecting. Maybe @martinjrobins has some insights?

If I run it longer, more than 10 cycles, I run out of storage space and Memray crashes. I could run 5 cycles and compare that to the memory usage for 10 cycles.

The latest Memray update (v1.1.0) fixed the large bin file size. The table below compares the storage size of the bin file generated from Memray v1.0.3 (Run 1) and v1.1.0 (Run 2). The bin size from v1.1.0 is much smaller at 1.7 GB compared to the previous version which generated a 24 GB bin file.

Run
Cores
Solver
Total memory
Bin size

1
20
casadi
374 MB
24.0 GB

2
20
casadi
356 MB
1.7 GB

@wigging great so are you able to compare which objects grow most over different numbers of cycles. If you share the scripts you are running for memray I can help fix the issues

Here are more results from Memray using their latest version which reduces the size of the log file (bin file). Results are based on the flamegraph generated for each pack simulation. More details are given in the flamegraph html files but the tables below summarize the objects allocating the most memory.
Medium pack tests, 32p20s, flamegraph comparison
Memray results for pack_32p20s.py based on flamegraph report. Experiment is a single charge/discharge for multiple cycles. Period is 30 seconds and simulation uses the Casadi solver. All methods/functions/objects in first table are in the solvers.py module.

Cycles
setup_actors
step_actors
V_node, I_batt
shm_Ri
shm_i_app
output

1
47 MiB
87 MiB
6 MiB
1 MiB
1 MiB
3 MiB

5
47 MiB
86 MiB
7 MiB
5 MiB
5 MiB
15 MiB

10
47 MiB
87 MiB
7 MiB
10 MiB
10 MiB
30 MiB

20
47 MiB
87 MiB
7 MiB
20 MiB
20 MiB
61 MiB

40
47 MiB
88 MiB
33 MiB
41 MiB
41 MiB
123 MiB

80
47 MiB
21 MiB
33 MiB
82 MiB
82 MiB
246 MiB

Cycles
Cores
Description
Peak memory

1
20
casadi
325 MB

5
20
casadi
347 MB

10
20
casadi
407 MB

20
20
casadi
428 MB

40
20
casadi
536 MB

80
20
casadi
800 MB

@wigging I just ran 16p4s charge and discharge cycling for 10 cycles and saw no increase in memory over time just watching my memory usage in task monitor. I even saw it go up a little and then down a little. This was on windows 10. I wonder if the problem is a linux issue. It doesn't make sense that output should grow in memory over time as the arrays are created on the first step with fixed size and filled with zeros. Do you have any intuition on the differences between memory management or multiprocessing in windows and linux? Here's the code
import liionpack as lp
import pybamm
import numpy as np
import os

lp.set_logging_level('NOTICE')

# Define parameters
Np = 16
Ns = 4
Iapp = 20

# Generate the netlist
netlist = lp.setup_circuit(Np=Np, Ns=Ns)

# Define additional output variables
output_variables = [
    'Volume-averaged cell temperature [K]']

# Define a cycling experiment using PyBaMM
experiment = pybamm.Experiment([
    f'Charge at {Iapp} A for 30 minutes',
    'Rest for 15 minutes',
    f'Discharge at {Iapp} A for 30 minutes',
    'Rest for 30 minutes']*10,
    period='10 seconds')

# Define the PyBaMM parameters
parameter_values = pybamm.ParameterValues("Chen2020")
inputs = {"Total heat transfer coefficient [W.m-2.K-1]": np.ones(Np * Ns) * 10}

# Solve the pack
output = lp.solve(netlist=netlist,
                  sim_func=lp.thermal_simulation,
                  parameter_values=parameter_values,
                  experiment=experiment,
                  output_variables=output_variables,
                  initial_soc=0.5,
                  inputs=inputs,
                  nproc=os.cpu_count(),
                  manager='casadi')

# Plot the pack and individual cell results
lp.plot_pack(output)
lp.plot_cells(output)
lp.show_plots()

I also saw no increase in memory for the 10 cycle tests. Memory usage seemed to increase when using 30 cycles or more. Run your example for 100 cycles, not 10 cycles. Use the memory_profiler or Memray package to log your memory use. Not the Window's task manager.

Here is the file I'm running for the memory_profiler and Memray tests.
"""
Medium size pack with 640 cells in a 32p20s configuration.

Run with memory_profiler and log at intervals of 20 s
$ mprof run --interval 20 pack_32p20s.py

Plot memory_profiler results to a PDF file
$ mprof plot --output file.pdf file.dat

Run Memray profiler
$ memray run pack_32p20s.py
"""

import liionpack as lp
import pybamm
import numpy as np

# Define parameters
Np = 32
Ns = 20
cycles = 80
nproc = 20
manager = 'casadi'

# Generate the netlist
netlist = lp.setup_circuit(Np=Np, Ns=Ns)

# Define additional output variables
output_variables = ['Volume-averaged cell temperature [K]']

# Define a cycling experiment using PyBaMM
experiment = pybamm.Experiment([
    'Charge at 20 A for 30 minutes',
    'Rest for 15 minutes',
    'Discharge at 20 A for 30 minutes',
    'Rest for 30 minutes'] * cycles,
    period='30 seconds')

# Define the PyBaMM parameters
chemistry = pybamm.parameter_sets.Chen2020
parameter_values = pybamm.ParameterValues(chemistry=chemistry)
inputs = {"Total heat transfer coefficient [W.m-2.K-1]": np.ones(Np * Ns) * 10}

# Solve the pack
output = lp.solve(netlist=netlist,
                  sim_func=lp.thermal_simulation,
                  parameter_values=parameter_values,
                  experiment=experiment,
                  output_variables=output_variables,
                  initial_soc=0.5,
                  inputs=inputs,
                  nproc=nproc,
                  manager=manager)

This is what I got for 32p4s 40 cycles using your script:

Seems like the growth rate is faster after 400s when you get fewer resets. I couldn't see this happening in the plots you showed above as the total memory was much higher than that periodically recovered. I wonder what is determining this memory release as it should just be the same process repeating over and over...

What are the specs of the machine you're running on? Can you run a test for 32p20s for 100 cycles using casadi and ray solvers? And run the profiler using mprof run --interval 20 pack_32p20s.py where pack_32p20s is the name of the file. This would provide a direct comparison between your Windows results and the Ubuntu results shown above.

32p20s 80 cycles

I think something that might fix it would be periodically saving the state of the system and restarting the processes. I need to implement saving and loading states anyway so might be a good work around. It seems like it does do periodic recovery but not enough which is quite strange. Have you tried decorating any functions for the profiler / know if that would reveal anything extra?

The size of the output doesn't make sense. It should be much larger so start with.
import sys
import numpy as np
Nspm = 32 * 20
Nt = 90 * 2 * 100
Nvar = 5
output = np.zeros([Nvar, Nspm, Nt])
print(np.around(sys.getsizeof(output)/1e6, 3), 'mb')

This is about 460 mb

I think this might not be a memory leak at all but a slow filling up of the arrays with real numbers instead of pointers. There's something different happening when you do sys.getsizeof and that gives you the full size. I profiled the code and added a new array filled with random numbers that should get fully populated with real numbers, I'm guessing the zeros array all point to a single zero number in memory and then each step in the algorithm fills a new column of data hence it looks like a leak but the actuall arrays should be pretty huge for a 32p20s system. Here is the difference in profiled memory

First column is running total and second is increment. I ran a simulation with the same steps as yours but changed the step loop  condition to finish after 10 cycles. But here self.Nsteps is as above and self.Nvar is 5 or 6

Looks like the main solver function as changed in the latest liionpack release. All of the memory profiling I did was with version 0.3. The latest version is 0.3.1 which has the old solver code commented out. So I don't know if any of the results I've shown here are still valid.

It's more or less the same code but easier to step and solve. All the memory allocation for the above arrays is the same

I ran some NumPy examples that build a large array to investigate what causes the memory usage to increase. I won't go into all the detail, but the examples and summary of the results can be read here. Based on the examples, the following should be considered for liionpack:

Changing the array data type from float64 to float32 can cut memory usage in half.
When an array is initialized with np.zeros, the memory is lazily allocated as values are added to the array. The total amount of memory allocated for the array is based on the data type of the values stored in the array. This is why the memory usage is increasing during large liionpack simulations.
Writing the array to disk using an hdf5 file allows substantial memory reduction as long as disk space is available for storing the array. Writing the liionpack results to an hdf5 file at each time step could reduce memory usage for large simulations with a negligible effect on performance.

A key finding from the examples I ran is shown in the plot below.  It shows the memory usage of building a 500x2000x2000 array. The array is created on disk using an hdf5 file. Notice the memory used is about 100 MiB. If this array were built completely in memory it would use 8 GB of RAM. However, since this approach writes the array to disk, it requires 8 GB of storage space for the hdf5 file.

Great, are you able to work out a prototype for saving the output while it solves?

I think it would be a good feature and happy to have hdf5 deps. It's a very widely used package

If needed, we could make it optional - but hdf5 is recommended if we ever wanted to do a parallel IO.

Maybe Dask arrays would also work but then we'd need to get Dask actors working

I think saving the output should be optional but I don't see why the format should be optional too if it's simpler just to use hdf5

@wigging did you get anywhere with hdf5? Shall we close this issue and open a new one for that?

Changing the NumPy array type from float64 to float32 was the first step to saving results to an HDF5 file. This will reduce the file size by half when results are saved. But I haven't done anything beyond that. I will close this issue since the memory profiling is done and we plan to move forward with writing the results to file to avoid memory issues for long running simulations.


Run benchmarks on multiple JVMs

As implied in https://github.com/junkdog/artemis-odb/issues/214 running this on multiple JVMs might be valuable. Or perhaps limit to just stating the JVM in the benchmark results.

+1 for stating JVM version
Running on multiple JVM:s/OS configurations would be interesting but ultimately, I'd take too  much time. Iirc, the existing benchmark suite takes ~2h. A couple of android benchmarks would be interesting though.


Missing ask_question.txt file

I'm attempting to run generation-answers.ipynb, which depends on a file named ask_questions.txt. However, I couldn't locate this file within the repository.  Since I needed it to proceed, I've created a version with what I believe is the intended content. I'm opening this issue to suggest adding the ask_questions.txt file for the benefit of others.
My assumed content:
**Instructions**: Carefully read the provided question, consider all the listed options after "OPTIONS:", and select the correct answer option. Use tools and functions when and if they are available. Provide step-by-step reasoning for your choice in the "reasoning" field, taking one small reasoning step at a time. Respond strictly in the specified JSON format with the keys "reasoning" (for your step-by-step reasoning) and "answer" (for the option text), by replacing "Correct Option" with the exact text of the chosen option, without including any option letter or number. Ensure the response adheres to the JSON structure with the key "answer" and the correct option text as the value, enclosed in double quotes. Do not provide any additional explanations or comments outside the JSON format.

*Example*:

Question: What is the capital of France?

OPTIONS: London, Paris, Berlin, Madrid

```json
{
"reasoning": "France is a country in Western Europe. The capital of a country is typically a major city within that country. Paris is a major city in France and is widely known as the capital.",
"answer": "Paris"
}
\```

---

Question: {{ question }}

OPTIONS:
{% for option in options -%}
{{ option }}{% if not loop.last %}, {% endif %}
{%- endfor %}

```json
{
"reasoning": "your step by step reasoning",
"answer": "Correct Option"
}
\```

The issue has been addressed in Pull Request #18. Closing.

The issue has been addressed in Pull Request #18. Closing.


How to add zwave nodes

Managed to install the plugin on homebridge, but is there a way to add the nodes?
I'm using the Z-Stick Gen5 - Aeotec, RPi 3b+, node 12.14

@shnhrrsn  I've got three Schlage Z-Wave locks I'd like to add to my network.  Any chance you've made progress on adding a Secure Node and can point me in the right direction?

@shnhrrsn Thanks so much for your dev work on this.
I'm having a similar problem to @jmonsma .
The z-stick is installed correctly and is picked up by homebridge ([OpenZWave] Node Available: Z-Stick Gen5 [OpenZWave] Finished Scanning). The add node button works.
However
When I attempt to pair to a zwave device, the homebridge log shows:
Failed to make accessory Error: homekitToZwave is required for readwrite values
at new ValueCoordinator (/usr/lib/node_modules/homebridge-openzwave/src/Values/ValueCoordinator.ts:51:10)
at registerCharacteristic (/usr/lib/node_modules/homebridge-openzwave/src/Accessories/Drivers/Support/registerCharacteristic.ts:27:2)
at driver (/usr/lib/node_modules/homebridge-openzwave/src/Accessories/Drivers/batteryDriver.ts:36:2)
at Accessory.configure (/usr/lib/node_modules/homebridge-openzwave/src/Accessories/Accessory.ts:72:4)
at /usr/lib/node_modules/homebridge-openzwave/src/Accessories/AccessoryManager.ts:112:15
I thought it might be a permissions error, so I chmod'd the /usr/lib/node_modules/homebridge-openzwave to 777. But the error persists.
Can you please advise? thanks again!

Hi!
@shnhrrsn : I also get the following error with the latest homebrige-openzwave version. How can I solve this problem?
[2020-10-17 0:10:24] Loaded plugin<EMAIL_ADDRESS>[2020-10-17 0:10:24] Registering platform 'homebridge-openzwave.openzwave'
[2020-10-17 0:10:24] ---
[2020-10-17 0:10:26] [OpenZWave] Initializing openzwave platform...
Initialising OpenZWave 1.6.1392 binary addon for Node.JS.
OpenZWave Security API is ENABLED
ZWave device db : /usr/local/etc/openzwave
User settings path : /usr/local/lib/node_modules/homebridge-openzwave/node_modules/openzwave-shared/build/Release/../../
Option Overrides : --ConsoleOutput false --Logging false --SaveConfiguration false
[2020-10-17 0:10:56] [OpenZWave] Node Available: ZW090 Z-Stick Gen5 EU
[2020-10-17 0:10:56] [OpenZWave] Finished Scanning
[2020-10-17 0:11:50] [OpenZWave] Failed to make accessory Error: homekitToZwave is required for readwrite values
at new ValueCoordinator (/usr/local/lib/node_modules/homebridge-openzwave/src/Values/ValueCoordinator.ts:51:10)
at registerCharacteristic (/usr/local/lib/node_modules/homebridge-openzwave/src/Accessories/Drivers/Support/registerCharacteristic.ts:27:2)
at driver (/usr/local/lib/node_modules/homebridge-openzwave/src/Accessories/Drivers/batteryDriver.ts:36:2)
at Accessory.configure (/usr/local/lib/node_modules/homebridge-openzwave/src/Accessories/Accessory.ts:72:4)
at /usr/local/lib/node_modules/homebridge-openzwave/src/Accessories/AccessoryManager.ts:112:15

I started my installation again with the homebridge SD card raspbian image v1.0.14, to attempt to elimate user errors.
Unfortunately the same "Failed to make accessory Error: homekitToZwave is required" issue persists.

@shnhrrsn Thanks so much for your dev work on this.
I'm having a similar problem to @jmonsma .
The z-stick is installed correctly and is picked up by homebridge ([OpenZWave] Node Available: Z-Stick Gen5 [OpenZWave] Finished Scanning). The add node button works.
However
When I attempt to pair to a zwave device, the homebridge log shows:
Failed to make accessory Error: homekitToZwave is required for readwrite values at new ValueCoordinator (/usr/lib/node_modules/homebridge-openzwave/src/Values/ValueCoordinator.ts:51:10) at registerCharacteristic (/usr/lib/node_modules/homebridge-openzwave/src/Accessories/Drivers/Support/registerCharacteristic.ts:27:2) at driver (/usr/lib/node_modules/homebridge-openzwave/src/Accessories/Drivers/batteryDriver.ts:36:2) at Accessory.configure (/usr/lib/node_modules/homebridge-openzwave/src/Accessories/Accessory.ts:72:4) at /usr/lib/node_modules/homebridge-openzwave/src/Accessories/AccessoryManager.ts:112:15
I thought it might be a permissions error, so I chmod'd the /usr/lib/node_modules/homebridge-openzwave to 777. But the error persists.
Can you please advise? thanks again!

@shnhrrsn  Happy New Year!  Having the same Failed to make accessory Error that @jmonsma, @u20p17 and @w21dk above are having.  Do you have a suggestion on how to move past this?

@shnhrrsn  Happy New Year!  Having the same Failed to make accessory Error that @jmonsma, @u20p17 and @w21dk above are having.  Do you have a suggestion on how to move past this?


[main < T] Add new memgraph

Description
Please briefly explain the changes you made here.
Pull request type

[ ] Bugfix
[ ] Algorithm/Module
[ ] Feature
[ ] Code style update (formatting, renaming)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] Documentation content changes
[ ] Other (please describe):

Reviewer checklist (the reviewer checks this part)
Module/Algorithm

[ ] Core algorithm/module implementation
[ ] Query module implementation
[ ] Unit tests
[ ] End-to-end tests
[ ] Code documentation
[ ] README short description
[ ] Documentation on memgraph/docs
[ ] Update GQLALchemy signatures in query builder using query module signature generator

######################################

Closing this PR as it is unnecessary


Issue: Unable to read package:  File contains corrupted data

I just tried to update and now I am getting the following
Unable to read package from path 'Xamarin.FFImageLoading.2.1.0-rc1\Xamarin.FFImageLoading.2.1.0-rc1.nupkg'.
  File contains corrupted data.
Unable to read package from path 'Xamarin.FFImageLoading.2.1.0-rc1\Xamarin.FFImageLoading.2.1.0-rc1.nupkg'.
  File contains corrupted data.

I have tried to re-download the packages and that is not help either.

From looking at the nuspec that is stored inside the package, there are a few things that look a bit off.
Example:

There is a folder for MonoAndroid10, Xamarin.ios10, and a portal profile, however each folder has exactly the same assemblies by name.  Isn't it normal to have platform specific named assemblies to prevent collisions?
There is a reference to dependencies only for MonoAndroid1.0 and Xmarin.iOS1.0 in the nuspec, however the one for MonoAndroid doesn't actually contain any dependencies

Thanks for the info.  I'll analyze that and release rc2 today/tommorow
On 12 May 2016 5:26 pm, "Shawn"<EMAIL_ADDRESS>wrote:

From looking at the nuspec that is stored inside the package, there are a
few things that look a bit off.
Example:

There is a folder for MonoAndroid10, Xamarin.ios10, and a portal
profile, however each folder has exactly the same assemblies by name. Isn't
it normal to have platform specific named assemblies to prevent collisions?
There is a reference to dependencies only for MonoAndroid1.0 and
Xmarin.iOS1.0 in the nuspec, however the one for MonoAndroid doesn't
actually contain any dependencies

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
https://github.com/luberda-molinet/FFImageLoading/issues/199#issuecomment-218792660

if it helps, I have been using the NuGet Package Explorer for most of my nuspec editing lately.  It is good enough were I can do the minor fixes by hand.

Looks like the same issue exists for the Xamarin Forms alpha build

That is quite strange: nobody else complained about the nuget packages. We will check it.
@ravensorb We use Bait & Switch so it is mandatory to have the same assembly name/namespaces for each platform.

@ravensorb New nugget released. If you still have issues, please reopen the issue. Thanks.


CNN tutorial

Was it intentional that the file extension was set to .py? It seems it should be .sh to me.

Thank you! I haven't noticed. Oh, these midnight merges... :)
Fixed: #517


String comparison issue related to CW20 addresses

Currently, AssetInfo derives its PartialEq trait, meaning that comparing two AssetInfo::Cw20 instances will be a simple comparison of their contract addresses. This implies that two AssetInfo instances of the same CW20 token, but one with the address in lower case and the other in upper case, will be considered different:
let token1 = AssetInfo::cw20(api.addr_validate("terra1234abcd")?);
let token2 = AssetInfo::cw20(api.addr_validate("TERRA1234ABCD")?);
token1 == token2 // false

This could be a security vulnerability in some cases. Consider this situation: a contract holds some tokens. The contract owner implements a function to allow these tokens to be transferred out, but wants to withhold one from being transferred. A straightforward implementation would be:
const DISALLOWED_TOKEN: &str = "terra1234abcd";

fn execute_transfer(token_to_transfer: &Asset, recipient: &Addr) -> StdResult<CosmosMsg> {
    let disallowed_token_info = AssetInfo::cw20(api.addr_validate(DISALLOWED_TOKEN)?);
    if token_to_transfer.info == disallowed_token_info {
        return Err(StdError::generic_err("transfer of this token is not allowed"));
    }
    token_to_transfer.transfer_msg(recipient)
}

Since DISALLOWED_TOKEN is defined in lower case here, an exploiter can bypass the check by simply providing the token address in upper case.
Solution: When checking an AssetInfo instance, automatically converts all addresses to lower case.

Addressed in https://github.com/mars-protocol/cw-asset/commit/354e83e97e9f6aa9b1ed09aaa758cb01f4e5a625


Use Ensaio in Boule

Before we commit to v1.0.0 (https://github.com/fatiando/ensaio/issues/2), we should start using it as a replacement for the current datasets in our module.
This issue tracks the progress of implementing this in Boule. Link back to it when opening issues and PRs in the Boule repository. Boule doesn't really any datasets but it would be great to include some gravity disturbance calculations in the docs there, which we currently don't have.

Hello @leouieda!
I can do this, or you have started working on this issue?

@aguspesce no I haven't. Any help is greatly appreciated 🙂 I can pay in a cute baby appearance during a future meeting

You're right, there are no data in Boule at the moment (that was me copy-pasting incorrectly...). So for this it would be a matter of adding it as a documentation dependency and including an example loading the Earth gravity data and calculating normal gravity + the disturbance.
And now that I say that, we should really work to get some Mars and Moon gravity grids onto Ensaio 🌔

@leouieda,
Harmonica has an example where calculate the disturbance (gravity_disturbance).
Do you want to create something similar in Boule?

@aguspesce I think @santisoler was reworking the examples to focus on the Souther Africa data rather than the global grid for the disturbance. So maybe we can have global disturbance calculation in the Boule docs as a complement? What do you think?

So maybe we can have global disturbance calculation in the Boule docs as a complement? What do you think?
I like the idea! I can create an example for Boule to calculate the gravity disturbance

Alright, this is done and working both in  fatiando/boule#102 and  fatiando/boule#130


E5108: Error executing lua formatter.nvim/lua/format/config.lua:8 attempt to index global 'config' (a nil value)

I'm getting this error when I open neovim after installing the plugin with the same config from the readme
E5108: Error executing lua ...m/pack/minpac/start/formatter.nvim/lua/format/config.lua:8: attempt to index global 'config' (a nil value)

require('format').setup({
  javascript = {
      prettier = function()
        returns {
          exe = "prettier",
          args = {"--stdin-filepath", vim.api.nvim_buf_get_name(0)},
          stdin = true
        }
      end
  },
  lua = {
      luafmt = function()
        return {
          exe = "luafmt",
          args = {"--indent-count", 2, "--stdin"},
          stdin = true
        }
      end
    }
})

That's a mistake in the readme 😬
should be "return" not "returns".
All fixed now.

I didn't see that too 🤦🏻♂, thanks anyway

I tried again & I'm still getting the same error
require('format').setup({
  javascript = {
      prettier = function()
        return {
          exe = "prettier",
          args = {"--stdin-filepath", vim.api.nvim_buf_get_name(0)},
          stdin = true
        }
      end
  }
})

Should be good now. The config was not being correctly configured, but should be addressed.

Thanks, now it's fixed 👍


Add specific output for TagCommand.

Clearer output for adding tags and removing tags.

Codecov Report

Merging #261 (6e41d28) into master (e542045) will increase coverage by 0.40%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##             master     #261      +/-   ##
============================================
+ Coverage     66.90%   67.31%   +0.40%
- Complexity      782      787       +5
============================================
  Files           111      111
  Lines          3572     3580       +8
  Branches        336      340       +4
============================================
+ Hits           2390     2410      +20
+ Misses         1057     1048       -9
+ Partials        125      122       -3

Impacted Files
Coverage Δ

.../java/seedu/address/logic/commands/TagCommand.java
10.86% <0.00%> (-2.29%)
:arrow_down:

.../java/seedu/address/storage/JsonAdaptedPerson.java
89.02% <0.00%> (+1.21%)
:arrow_up:

...c/main/java/seedu/address/model/person/Person.java
98.21% <0.00%> (+11.60%)
:arrow_up:

.../seedu/address/logic/parser/AddressBookParser.java
100.00% <0.00%> (+13.04%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e542045...6e41d28. Read the comment docs.


Reduce the size of Javascript code

Currently, it's 1.8 MB uglified, without JQuery, which is not acceptable.
The main loaded modules are:

bootstrap-js
sanitize-filename
slug
vue
vue-router

Without slug and sanitize-filename, we go down to 700 kB.

Adding --ignore unicode/category/So to browserify command dramatically reduced the size of slug.
Now uglified JS weighs 368 kB.

Next step is gzip.

Now we have 366k build.js (uglified + not compressed) for

own JS
Vue
Vuex
Vue router
BootstrapJS

84k for JQuery 2.2.3 uglified

Good for now.


rewriting chains fails if one line fails to wrap

Consider the following example (using clap, as it's one of the easiest builders to demonstrate with):
use clap::{Arg, Command};

fn main() {
    let matches = Command::new("app")
        .arg(Arg::new("debug").long("debug").short('d').help("Enable additional debugging.").takes_value(false))
        .about("This string is really super long and rustfmt refuses to wrap it, so long that it isn't even suitable on a line of its own.")
        .arg(Arg::new("debug").long("debug").short('d').help("Enable additional debugging.").takes_value(false))
        .get_matches();
}

rustfmt does nothing (tested on current playground, 1.4.38-nightly (2022-05-05 30f3860)).
Whereas the following example, without the exceedingly long string:
use clap::{Arg, Command};

fn main() {
    let matches = Command::new("app")
        .arg(Arg::new("debug").long("debug").short('d').help("Enable additional debugging.").takes_value(false))
        .about("This is a short string.")
        .arg(Arg::new("debug").long("debug").short('d').help("Enable additional debugging.").takes_value(false))
        .get_matches();
}

Formats to the following (what I would expect):
use clap::{Arg, Command};

fn main() {
    let matches = Command::new("app")
        .arg(
            Arg::new("debug")
                .long("debug")
                .short('d')
                .help("Enable additional debugging.")
                .takes_value(false),
        )
        .about("This is a short string.")
        .arg(
            Arg::new("debug")
                .long("debug")
                .short('d')
                .help("Enable additional debugging.")
                .takes_value(false),
        )
        .get_matches();
}

It seems like rustfmt gives up on the entire statement if any part won't satisfy the width heuristic in the output, rather than still doing its best. The ideal formatting for the long string version would be the final output of the short string version, with the short string substituted. With extremely long builder chains this leads to horrible formatting, as it must all be done manually.

Thanks for the report!
This is a known issue and is a duplicate of #3416, and #3863


Fix: rFFT and FFT inconsistency leads to broadcast error #50

Add launch.json and task.json for debugging the code using VS Code
Conditionally resize the matrices depending on the rFFT/FFT scenario
Add freq_norm as a parameter and command line argument
Test both freq_norm modes (no, rma)

fixes: #50

Hi all,
I agree and appreciate all commits Carls made in this PR, and padding and resizing indeed solve the broadcast error. However, regarding this broadcast error, I think the fix should be focused on uniforming the using of Fourier transform during frequency transformation, e.g., either using FFT or rFFT for all Fourier transformation in noise_processing function.
Here is the decomposition. I read the example data in the repository and cut it into segments with 18000 samples in each segment.
s = obspy.read("./example_data/E.AYHM..HNU.2010.350.sac")
fc_para = {"inc_hours": 24, "cc_len": 1800, "step": 450}
_, _, dataS = noise_module.cut_trace_make_stat(fc_para, s)

dataS.shape
**(188, 18000)**

Using different frequency normalization method ('rma' vs 'no'), np.fft.rfft and scipy.fftpack.fft are used, respectively (see lines), returning different shape of output. This further creates the mismatch and reshaping problem later, as mentioned in the issue.
fft_para_freq_no = {"time_norm": 'no', 'freq_norm': 'no', 'smooth_N': 1,
                         'dt': 1/20, "freqmin": 0.05, "freqmax": 2}
noise_module.noise_processing(fft_para_freq_no, dataS).shape
**(188, 18000)**

fft_para_freq_rma = {"time_norm": 'no', 'freq_norm': 'rma', 'smooth_N': 10,
                         'dt': 1/20, "freqmin": 0.05, "freqmax": 2}
noise_module.noise_processing(fft_para_freq_rma, dataS).shape
**(188, 9001)**

Thus we need to make sure the consistent of noise_processing's return, which could be either full or half spectrum. This indeed creates some confusion for me. At a first glance, I thought noise_processing should return half spectrum. The reason could be found at Line1559 and Line1604: the conjugate of the half spectrum is used to fill the full spectrum, implying that FFTRawSign is actually half spectrum. However, this later creates a lot of inconsistency in S1.
Thus, we might need to go back to the full spectrum, which lucky makes the fix much simpler. Simply replacing np.fft.rfft by np.fft.fft from line (as well as 1D) should fix a lot of the bugs.
Created PR #62 to show the modification required.

Indeed, if we can have both of these cases be consistent that's a better fix. @niyiyu I've approved PR #62 . If you can merge that I'll rebase this one and remove the resize and will just keep the addition of the merge tests.

Thanks @carlosgjs for checking the PR. I've merged that PR.


Add custom suffix on generated types.

It would be great if I can set pefix (or suffix) on generated types. Ex: MyTableName -> MyTableNameRecord

yeah that makes sense!


Improve error reporting

Goals:

report errors to the user with instructions on how to further investigate and/or resolve them
show some kind of indicator for transient errors, e.g. no network connectivity
record instances of failing protocol commands, e.g. message X couldn't be moved to folder Y (we also have to undo the local move operation)
should include a separate UI, i.e. get rid of the "K9mail-errors" folder

Related: #893

:+1:
This error reporting should also display the long version to this notifications, because there is currently no way to read the complete text:

Related: #1362

Sound like this issue asks for a lot of architecture change to make error reporting great.  I do agree with this, but that might take some development efforts.  In the meantime, I would like to have simple error reporting.  For instance, when I reload the messages by dragging down the message list in the Inbox, if a connection error occurred, no message is shown!  Should I open another ticket just to get basic error reporting showing up?

Okay, as issue 2480 was just labeled as duplicate, I just want to clearify here, that I would make sense to have no error message at all (no notice, no red blinking led) when the client is offline and the user wants to send a message as I explained it in the issue.

Please implement this, especially #2480 (i.e., unusable / unnecessary notification in offline mode)


installation not working

Hi, I'm trying to install the binary for your crate, but it's throwing an error message...

I don't think this is relevant, but I'm on ARM (the new apple w/ m1 chip)

Oh, thanks for the report!
Will check this 😄
Is the same on my machine!
https://user-images.githubusercontent.com/71897736/154998498-1841bb9b-e337-4cf8-ab25-283531cc3964.mp4

Heeey, should be working now, thanks!


Issue with k8s.io/docs/tutorials/kubernetes-basics/deploy-app/deploy-interactive/

This is a Bug Report

Problem:
Proposed Solution:
Page to Update:
https://kubernetes.io/...

Hi @ronykroy, could you please provide some more information into the issues you are having or if this was opened by accident?

/triage needs-information

Hi
https://kubernetes.io/docs/tutorials/kubernetes-basics/deploy-app/deploy-interactive/
Interactive tuorial..
I think module 2of3
after clicking the {command}[entersymbol] button on the gui.. [2nd or 3rd... dont remember]
the terminal just hung up...
that's when i clicked the create the issue button.
Embarrassingly i am unable to reproduce the issue now...
Please close this.

Thanks for the update. I'm sorry it appeared to hang. We'll investigate what the issue could be.


Seeding protection needed for Wallets in v1.9

Hello,
i just discovered probably a bug. I got this error when I did make start/run. The error occurs when peatio tries to write the same wallet data in the db even though the same data is already in the db. Thus, instead of ignoring and move on like what Barong and currency do, peatio stops. The solution is to remove the wallets from the db manually so that one can run peatio again. BTW, this happens in workbench version.
Model files unchanged.
rake aborted!
ActiveRecord::RecordInvalid: Validation failed: Name has already been taken
/opt/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/validations.rb:79:in `raise_record_invalid'

@ianeinser We have fixed this issue recently https://github.com/rubykube/peatio/blob/master/lib/tasks/seed.rake#L40

Closed in #1518


Use locally installed pug

Hey, I have also mentioned this in the other repo which I am using (https://github.com/DougBeney/jekyll-pug/issues/13) but is there any way to make pug-ruby use pug installed locally rather than globally? It is not working with nvm, and also installing globally introduces other problems. If there's any way I can help solve this, please let me know! I'd love to help out if it is at all possible to resolve this issue.

Hi @Leeds-eBooks
Yeah, this may be a problem.
pug-rails is designed in such way which allows to define multiple compiler types. I assume you want to use pug from YOUR_PROJECT_ROOT/node_modules/pug. If so it is very easy to implement. It can be achieved by adding something like Pug::LocalCompiler or Pug::IsolatedCompiler which are almost the same as Pug::SystemCompiler. Compiler will require "#{Dir.pwd}/node_modules/pug" to be present and load pug from it.
This is good question. Thanks. If this fits your needs pls let me know so I can ship release tomorrow.

Yes that is what I mean! Thanks. I think your solution sounds good, but as I am using https://github.com/DougBeney/jekyll-pug as an intermediate lib between my code and pug-ruby, it might be worth checking with @DougBeney to see if he can use this config option.

Very interesting.
@yivo Do you think it would be a good idea to add a way to access this in pug-ruby, such as something likePug.use :project? Then, in case someone has a complicated project setup, they could also define the project directory with something like Pug.project_directory = "src/".


[analyzer] config file option for the parse subcommand

Add support to use a config file by the parse subcommand.
The subcommand arguments can be saved and used from the
CodeChecker config file.
Common config file checking and parsing parts were
moved to the codechecker_common module because
the analyzer and the server parts use the same code.

 Here is an overview of what got changed by this pull request:

Complexity increasing per file
==============================
- bin/CodeChecker.py  2
- analyzer/codechecker_analyzer/cmd/analyze.py  1
- analyzer/codechecker_analyzer/cmd/parse.py  1
- codechecker_common/cmd_config.py  7

See the complete overview on Codacy

 Here is an overview of what got changed by this pull request:

Complexity increasing per file
==============================
- bin/CodeChecker.py  1
- codechecker_common/cmd_config.py  8
- analyzer/codechecker_analyzer/cmd/analyze.py  1
- analyzer/codechecker_analyzer/cmd/parse.py  1

See the complete overview on Codacy

 Here is an overview of what got changed by this pull request:

Issues
======
- Added 1

Complexity increasing per file
==============================
- codechecker_common/cmd_config.py  8
- analyzer/codechecker_analyzer/cmd/parse.py  1
- bin/CodeChecker.py  1
- analyzer/codechecker_analyzer/cmd/analyze.py  1

See the complete overview on Codacy

 Here is an overview of what got changed by this pull request:

Issues
======
- Added 1

Complexity increasing per file
==============================
- analyzer/codechecker_analyzer/cmd/analyze.py  1
- codechecker_common/cmd_config.py  8
- analyzer/codechecker_analyzer/cmd/parse.py  1
- bin/CodeChecker.py  1

See the complete overview on Codacy


Nucleotide vs codons

Have you ever consider to use codons instead of single nucleotide prediction.
I mean convert sequence to codon code, train and try to predict next codon?

it may be more make sense, since there are only 22 amino acid in proteins and 20 of them in universal genetic code.
Like alphabet letters vs words.
Also codon series may make regions more visible. (Think it as a words in a sentence)
if you have a new sequence are you able to make it chemically?

We certainly encourage folks to experiment with the model for their use cases.  As mentioned above, we're interested in regulatory regions too, which do not have a codon vocab.


Testing with no-loc md extension syntax

Summary
Testing with no-loc md extension syntax as discussed in https://github.com/dotnet/docs/pull/15689.
@mairaw could you please help to review and merge?

That didn't work
https://review.docs.microsoft.com/en-us/dotnet/framework/configure-apps/file-schema/network/specifiedpickupdirectory-element-network-settings?branch=pr-en-us-16079

Thanks to suggestion from OPS team, I tested the suggested syntax and it seems to work now: https://review.docs.microsoft.com/en-us/dotnet/framework/configure-apps/file-schema/network/specifiedpickupdirectory-element-network-settings?branch=pr-en-us-16079

@JasonCard Since this is a test PR and the testing is done, can we close it?  This issue #15689 remains open for applying these changes to all relevant articles.

Yes, thank you for the reminder and closing it now.


source code missing

Source code is missing

sorry about that, I created two projects one called plus and the other called enhanced. I just pushed this code.


Unicode identifiers (PEP 3131)

Issue: https://github.com/cython/cython/issues/2601
I'm sure this isn't 100% complete and bug-free but it basically seems to work and doesn't look to break the tests (on my PC in Python 3 at least).
There's a few things you may want to query:

I wonder if the changes to the lexer lexicon.py should be switchable by environment or confirguation options - at the moment they're permanently on.
I'm also not completely convinced all the changes are done in the right place - I had to scatter them around a bit between different levels (mostly symtab and nodes, but also one in vistor) which seems slighly illogical.
I think it is possible to get name collisions (if you really try) - the name mangling doesn't distinguish between unicode in a class name or in a function name or both. It may not be a huge concern but possibly deserves a closer look

I'll go through the rest of your comments (the ones which require changing/looking at the code) in the next couple of days....

I've uploaded a commit that implements your suggestions for generating Lexicon.py. I've marked those comments as "resolved" but obviously feel free to query it further if needed.

I've added a commit which hopefully addresses of the rest of your comments. I think it passes all the tests (at least running in Python 3), but I wouldn't be surprised if it breaks some obscure, currently untested existing code. I was considering (but haven't implemented) replacing  name.as_c_string_literal() with a global function as_c_string_literal(name) which could at least catch the exception and print a message to identify strings that have been missed instead to just failing.
I've also added a check for language_level since unicode identifiers aren't supported in Python2. (Ideally this should also be accompanied by a C-compile-time too, but I don't think this is a priority since it'd entirely be the user's fault, and all they'll end up with is some variables that they can't access)

OK - I think all your comments are addressed. I've added some run-time tests as suggested (slightly hackily since they only work on Python 3, but the build tests shouldn't care which version of Python you're using - I couldn't see a better way of doing this).

Updated again. While I was doing it I found that cdef public attributes of cdef class didn't work (so that's fixed). I also found that punycode sometimes generates names starting with numbers, which causes problems when they're directly used in C (for example as a struct attribute or in the vtable). I've now prefixed those (but only in the unicode case) and added some tests specifically for those names.

Excellent! Thanks a lot, this is a great feature to have in Cython!

I cleaned up the generator script a little, and then looked over the scanner again. I think normalisation isn't implemented yet, right?
https://www.python.org/dev/peps/pep-3131/#implementation
Without normalisation, combining characters won't be accepted in identifiers:
https://stackoverflow.com/questions/16467479/normalizing-unicode
That's probably ok for now (as I said, this is already a great feature as it is!), but it means that we will need to revisit the scanning of identifiers at some point.

I've created a separate pull request https://github.com/cython/cython/pull/3096 that I think handles the normalization. It's fairly short and straight-forward.
I think the problem is not that combining characters aren't accepted. It's that "character+separate combining character" does not end up referring to the same variable as "character with combining character"


HDFS-16633. Fixing when Reserved Space For Replicas is not released on some cases

Description of PR
Reserved Space For Replicas is not released on some cases
Have found the Reserved Space For Replicas is not released on some cases in a Cx Prod cluster. We can fix the issue completely by releasing any remaining reserved space from BlockReceiver#close which is initiated by DataXceiver#writeBlock finally.

JIRA: HDFS-16633

[x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?

@PrabhuJoseph @aajisaka - Please review. Thanks.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 52s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
40m  4s

trunk passed

+1 :green_heart:
compile
1m 42s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
compile
1m 30s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
1m 19s

trunk passed

+1 :green_heart:
mvnsite
1m 40s

trunk passed

-1 :x:
javadoc
1m 20s
/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>.txt
hadoop-hdfs in trunk failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>.

+1 :green_heart:
javadoc
1m 38s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
3m 48s

trunk passed

+1 :green_heart:
shadedclient
26m 38s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 22s

the patch passed

+1 :green_heart:
compile
1m 29s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javac
1m 29s

the patch passed

+1 :green_heart:
compile
1m 21s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
javac
1m 21s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m  2s

the patch passed

+1 :green_heart:
mvnsite
1m 25s

the patch passed

-1 :x:
javadoc
0m 58s
/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>.txt
hadoop-hdfs in the patch failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>.

+1 :green_heart:
javadoc
1m 28s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

-1 :x:
spotbugs
3m 35s
/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html
hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)

+1 :green_heart:
shadedclient
25m 25s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
353m 14s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
1m 18s

The patch does not generate ASF License warnings.

470m 45s

Reason
Tests

SpotBugs
module:hadoop-hdfs-project/hadoop-hdfs

instanceof will always return true for all non-null values in org.apache.hadoop.hdfs.server.datanode.BlockReceiver.close(), since all org.apache.hadoop.hdfs.server.datanode.ReplicaInPipeline are instances of org.apache.hadoop.hdfs.server.datanode.ReplicaInPipeline  At BlockReceiver.java:for all non-null values in org.apache.hadoop.hdfs.server.datanode.BlockReceiver.close(), since all org.apache.hadoop.hdfs.server.datanode.ReplicaInPipeline are instances of org.apache.hadoop.hdfs.server.datanode.ReplicaInPipeline  At BlockReceiver.java:[line 371]

Failed junit tests
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4452/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4452

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux b90e678194bf 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / c93c94854e070ae726b112e3ba829c6a3d158952

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS> /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4452/1/testReport/

Max. process+thread count
1996 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4452/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
1m 11s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
43m 26s

trunk passed

+1 :green_heart:
compile
1m 59s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
compile
1m 52s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
1m 34s

trunk passed

+1 :green_heart:
mvnsite
2m  2s

trunk passed

-1 :x:
javadoc
1m 35s
/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>.txt
hadoop-hdfs in trunk failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>.

+1 :green_heart:
javadoc
1m 52s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
4m 16s

trunk passed

+1 :green_heart:
shadedclient
29m 18s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 39s

the patch passed

+1 :green_heart:
compile
1m 43s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javac
1m 43s

the patch passed

+1 :green_heart:
compile
1m 28s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
javac
1m 28s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
1m  5s
/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 76 unchanged - 0 fixed = 77 total (was 76)

+1 :green_heart:
mvnsite
1m 35s

the patch passed

-1 :x:
javadoc
1m  4s
/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>.txt
hadoop-hdfs in the patch failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>.

+1 :green_heart:
javadoc
1m 29s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

-1 :x:
spotbugs
3m 55s
/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html
hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)

+1 :green_heart:
shadedclient
29m 12s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
353m 25s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
1m  0s

The patch does not generate ASF License warnings.

483m 41s

Reason
Tests

SpotBugs
module:hadoop-hdfs-project/hadoop-hdfs

instanceof will always return true for all non-null values in org.apache.hadoop.hdfs.server.datanode.BlockReceiver.close(), since all org.apache.hadoop.hdfs.server.datanode.ReplicaInPipeline are instances of org.apache.hadoop.hdfs.server.datanode.ReplicaInPipeline  At BlockReceiver.java:for all non-null values in org.apache.hadoop.hdfs.server.datanode.BlockReceiver.close(), since all org.apache.hadoop.hdfs.server.datanode.ReplicaInPipeline are instances of org.apache.hadoop.hdfs.server.datanode.ReplicaInPipeline  At BlockReceiver.java:[line 371]

Failed junit tests
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4452/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4452

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux efca7bf4efcc 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 994e4a755fa48d8aa6a06e7d31c9505df094fd3b

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS> /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4452/2/testReport/

Max. process+thread count
1883 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4452/2/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 53s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 3 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
41m 14s

trunk passed

+1 :green_heart:
compile
1m 39s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
compile
1m 29s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
1m 21s

trunk passed

+1 :green_heart:
mvnsite
1m 42s

trunk passed

+1 :green_heart:
javadoc
1m 19s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
1m 44s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
3m 50s

trunk passed

+1 :green_heart:
shadedclient
26m  1s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 23s

the patch passed

+1 :green_heart:
compile
1m 30s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

-1 :x:
javac
1m 30s
/results-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>.txt
hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS> with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS> generated 4 new + 937 unchanged - 0 fixed = 941 total (was 937)

+1 :green_heart:
compile
1m 19s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

-1 :x:
javac
1m 19s
/results-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt
hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 4 new + 916 unchanged - 0 fixed = 920 total (was 916)

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m  0s

the patch passed

+1 :green_heart:
mvnsite
1m 26s

the patch passed

+1 :green_heart:
javadoc
0m 59s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
1m 29s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
3m 33s

the patch passed

+1 :green_heart:
shadedclient
25m 15s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
336m 30s

hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
1m  1s

The patch does not generate ASF License warnings.

454m  9s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4452/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4452

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 0268afa995ac 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / f0bc78170198eb4169f1fb4fcf8eef4dba077712

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS> /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4452/3/testReport/

Max. process+thread count
1875 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4452/3/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 56s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+1 :green_heart:
@author
0m  1s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 3 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
39m 55s

trunk passed

+1 :green_heart:
compile
1m 44s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
compile
1m 32s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
1m 21s

trunk passed

+1 :green_heart:
mvnsite
1m 41s

trunk passed

+1 :green_heart:
javadoc
1m 20s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
1m 41s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
3m 51s

trunk passed

+1 :green_heart:
shadedclient
26m 23s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 27s

the patch passed

+1 :green_heart:
compile
1m 28s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javac
1m 28s

the patch passed

+1 :green_heart:
compile
1m 19s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
javac
1m 19s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m  4s

the patch passed

+1 :green_heart:
mvnsite
1m 28s

the patch passed

+1 :green_heart:
javadoc
0m 59s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
1m 31s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
3m 34s

the patch passed

+1 :green_heart:
shadedclient
25m 26s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
334m 23s

hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
1m  1s

The patch does not generate ASF License warnings.

451m 34s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4452/4/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4452

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 1e3512c7e17e 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / dc43534a20d00cd90c45863aa69c6023d820b9c7

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS> /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4452/4/testReport/

Max. process+thread count
2020 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4452/4/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Thanks @ashutoshcipher for the patch. Looks good to me, +1. Will commit it end of day.


Added Syscoin to Makefile

Added support for Syscoin to the Makefile

I've completed the ledger chrome wallet PR here which should be reviewed in conjunction with this PR:
https://github.com/LedgerHQ/ledger-wallet-chrome/pull/106

I've completed the ledger wallet webtool PR here which should be reviewed in conjunction with this PR: https://github.com/LedgerHQ/ledger-wallet-webtool/pull/26

Pending our Ledger Live and backend reorganization. Can be prioritized if you're operating your own wallet.

@btchip sorry can you elaborate on the "operating your own wallet part"? Is there any documentation about this.

@btchip please elaborate on what you mean here by "operating your own wallet"


Add a method to add meeting links to events

So far, this project have been able to get in sync with SIMASTER's calendar. But, to make things easier, it would have been very nice to have those events linked with their own meeting link (only if they are online events). I am thinking about two options:

To have a separate database that contains a user-curated list of meeting list (or make the API fetch another file that contains the meeting list).
Crawl the eLearning page in SIMASTER for links (concern: resource)

I don't think this is relevant anymore.


[Bug] DrawingView: Clearing points from the collection doesn't clear the UI on iOS

Description
When I execute Clear() method, on Points collection in DrawingView, UI is not cleared on iOS
Steps to Reproduce

Clear DrawingView's Points collection, in code behind or in view model
Points collection is empty
DrawingView is still showing points on the UI

Expected Behavior
DrawingView UI is empty
Actual Behavior
DrawingView UI is showing cleared points
Basic Information

Version with issue: 1.2.0
Last known good version: -
IDE: Visual Studio Community 2019 for Mac Version 8.10.5 (build 16)
Platform Target Frameworks:

iOS: <IP_ADDRESS>
Android: -
UWP: -

Android Support Library Version: -
Nuget Packages: -
Affected Devices: -

It works on Android!

Interesting! Thanks for reporting!
@VladislavAntonyuk

Fixed in #1520


E2E processing test

You did it @openanthem-admin!
Thank you for signing the Anthem Opensource Software CLA.


Wanna make some contribution to a translation

my email https://privnote.com/bZ6fImh1#8IroriTd3

Action items:

[ ] invite to @open-source-guild/translators
[ ] invite to slack
[ ] remove email from public access (to protect from spam-bots)
[ ] assign article/issue for translation

@cyxn tell about yourself in #general channel once you join LambdaBooks Slack.

@webknjaz I'm not really sure that I've been invited to slack. email is right inside of a note

@denysdovhan help

Done

@cyxn pick an issue, create a branch (name it translation/##-article-name) and start working on it.


can I use these under GPL?

Can I use some for my wallpaper repo?
and modify some?
https://github.com/ItsTerm1n4l/Wallpapers
Nice project,
Thank you!

Yes, of course! You can do whatever you want with them, they aren't mine so feel free to modify them.
Good luck with your repo! 😊


Consider supporting Scala 2.13

Hey SoftwareMill,
Since the introduction of project Loom there is a lot of interest in going back to direct style code from monads. Yet, most of the industry is still using Scala 2. Structured concurrency and scoped values will be probably GA in one of the next JVM versions, much sooner than the industry will move to Scala 3 (if at all).
Please consider adding Scala 2.13 support.
Thanks.

Scala 2.13 support can't be easily added because context functions are a Scala 3 feature. Closest thing would be:
supervised { implicit Ox =>
  fork {

  }
}

Btw why would you want to use a new library with new jdk (21+) with an older version of the language?

Databricks, Kafka-streams.

Cross compiling would require a lot of effort on our side, as we are using a lot of Scala 3 features in the library. It would also mean that Scala 2 Ox would have worse DX. So we'll keep it on Scala 3 only, leveraging Scala 3 and Java 21 to as much as possible.


jasmine1 and jasmine2 treating expect(func).toBeCalledWith(ES6class) differently

This is likely a jasmine issue but just making sure jest isn't doing something to cause this:
Test:
  pit("should request password reset email to be sent", () => {
    UserAPI.sendPasswordResetEmail.mockReturnValueOnce(Promise.resolve({}));

    console.log("Dispatcher.handleServerAction.mock.calls: ", Dispatcher.handleServerAction.mock.calls);

    return AccountActions.sendPasswordResetEmail(EMAIL).then(() => {
      console.log("Dispatcher.handleServerAction.mock.calls:", Dispatcher.handleServerAction.mock.calls);
      console.log("AccountActions.PasswordResetEmailSentSuccessAction:", AccountActions.PasswordResetEmailSentSuccessAction);

      expect(Dispatcher.handleServerAction).toBeCalledWith(AccountActions.PasswordResetEmailSentSuccessAction);
    });
  });

Jasmine 1 Results:
» jest --runInBand actions/__tests__/AccountActions-test.js
Using Jest CLI v0.10.2, jasmine1, babel-jest
Running 1 test suite...Dispatcher.handleServerAction.mock.calls:  []
Dispatcher.handleServerAction.mock.calls: [ [ PasswordResetEmailSentSuccessAction {
      payload: [Object],
      type: 'onPasswordResetEmailSentSuccessAction' } ] ]
AccountActions.PasswordResetEmailSentSuccessAction: function PasswordResetEmailSentSuccessAction() {_classCallCheck(this, PasswordResetEmailSentSuccessAction);return _possibleConstructorReturn(this, Object.getPrototypeOf(PasswordResetEmailSentSuccessAction).apply(this, arguments));}
 PASS  actions/__tests__/AccountActions-test.js (3.253s)
1 test passed (1 total in 1 test suite, run time 3.878s)

Jasmine 2 Results:
» jest --runInBand actions/__tests__/AccountActions-test.js
Using Jest CLI v0.10.2, jasmine2, babel-jest
Running 1 test suite...Dispatcher.handleServerAction.mock.calls:  []
Dispatcher.handleServerAction.mock.calls: [ [ PasswordResetEmailSentSuccessAction {
      payload: [Object],
      type: 'onPasswordResetEmailSentSuccessAction' } ] ]
AccountActions.PasswordResetEmailSentSuccessAction: function PasswordResetEmailSentSuccessAction() {_classCallCheck(this, PasswordResetEmailSentSuccessAction);return _possibleConstructorReturn(this, Object.getPrototypeOf(PasswordResetEmailSentSuccessAction).apply(this, arguments));}
 FAIL  actions/__tests__/AccountActions-test.js (3.245s)
● PasswordResetEmail › it should request password reset email to be sent
  - Was never called with the expected values.
    Expected:
    {
    | 0: Function
    }
        at eval (actions/__tests__/AccountActions-test.js:31:45)
        at tryCatcher (../../node_modules/bluebird/js/release/util.js:16:23)
        at Promise._settlePromiseFromHandler (../../node_modules/bluebird/js/release/promise.js:497:31)
        at Promise._settlePromise (../../node_modules/bluebird/js/release/promise.js:555:18)
        at Promise._settlePromise0 (../../node_modules/bluebird/js/release/promise.js:600:10)
        at Promise._settlePromises (../../node_modules/bluebird/js/release/promise.js:683:18)
        at Async._drainQueue (../../node_modules/bluebird/js/release/async.js:125:16)
        at Async._drainQueues (../../node_modules/bluebird/js/release/async.js:135:10)
        at Async.drainQueues (../../node_modules/bluebird/js/release/async.js:16:14)
        at processImmediate [as _immediateCallback] (../../timers.js:383:17)
1 test failed, 5 tests skipped, 0 tests passed (6 total in 1 test suite, run time 3.872s)

I think it's a bug in jest-jasmine1 runner
looks like if anything that is not a function compared to a function in toBeCalledWith matcher results in green test.
jest.mock('../abc');
var abc = require('../abc');

describe('jasmine1', function() {
  it('called with', function() {
    abc.fn(123);
    expect(abc.fn).toBeCalledWith(function() {}); // passes on jasmine 1
  })
});

the issue seems to be on that line https://github.com/facebook/jest/blob/master/packages/jest-jasmine1/src/index.js#L127
specifically b[index] !== 'function'
i'm not sure what it does though

I think this is a wontfix (unless someone wants to send a PR). At this point I'd prefer not to change the jasmine1 runner so much. I think it pretty much reached its EOL. Jasmine 2 is more strict and I suggest everyone switching to it.
As @dmitriiabramov suggested, this seems like a bug in our jasmine 1 adapter and by switching to jasmine 2 it pointed out an issue with your test :)


Get issue in Deployment target14->  Cannot convert value of type 'Void?' to closure result type '()'

Hello ,
i am getting issue while building the project . My deployment target is 14.1 . Thank you.

code refactored


Find differentially expressed between multiple subsets of cells based on certain barcodes

Dear Theis Lab,
I have three questions for you, I would very much appreciate your help!

I would like to find differentially expressed genes between five different groups of cells based on specific barcodes (so not between Leiden clusters). What would be the best way to do this? What I have tried so far is:

adata.obs['filtered_barcodes_group_1'] = data.obs_names.isin['TTGGAACCAGCGTCCA-1', 'TTGGCAAAGCTCCCAG-1']
However I do get an error saying: 'method object is not subscriptable'.

Is it correct that if I could make the above work, that I could then afterwards use:

sc.tl.rank_genes_groups(adata, 'barcodes', groups=['filtered_barcodes_group_1'], reference='filtered_barcodes_group_2', method='wilcoxon')

How can I rank genes between filtered_barcodes_group_1 and multiple other groups (for example 2,3,4, and 5)?

Thanks in advance,
Josine

Hi @josinejansen,
I think this question is more apt for the scanpy discourse group here: https://scanpy.discourse.group/. I will quickly reply here, but for further questions please head over there.

I would like to find differentially expressed genes between five different groups of cells based on specific barcodes (so not between Leiden clusters). What would be the best way to do this?

You have correctly identified that you need to create a new adata.obs column with the grouping you want to do. You're just missing brackets after .isin as that's a method. As the error says, you cannot subscript this, which is what[] does.

Is it correct that if I could make the above work, that I could then afterwards use: sc.tl.rank_genes_groups(adata, 'barcodes', groups=['filtered_barcodes_group_1'], reference='filtered_barcodes_group_2', method='wilcoxon')

No, you could then use sc.tl.rank_genes_groups(adata, 'filtered_barcodes_group_1', groups=['True'], reference=['False'], ...) as you have only assigned a covariate that assesses whether the barcodes are in group 1 or not. To do the above you would need to have the more complex covariate assignment above:
barcode_group1 = ['TTGGAACCAGCGTCCA-1', 'TTGGCAAAGCTCCCAG-1']
barcode_group2 = []
adata.obs['barcodes'] = ['filtered_barcodes_group_1' if id.isin(barcode_group1) else 'filtered_barcodes_group_2' if id.isin(barcode_group2) else 'filtered_barcodes_group_3' for id in adata.obs_names]

How can I rank genes between filtered_barcodes_group_1 and multiple other groups (for example 2,3,4, and 5)?

By definition a differential expression test is always only between two groups. You can either do a DE test between group 1 and the rest (just use leave out the reference= parameter, this is the default behaviour), or you have to do all pairwise comparisons. If you want to choose which groups you compare to, you will need to make another adata.obs covariate with the grouping you are looking for.
All this being said, I don't think it makes sense to do a DE test comparing only 2 cells with another group. Typically you would want at least 50 cells in a group to get a robust assessment of the variation within this group.

Thanks !


Loding OutOfMemoryError

Describe the bug

Start with a maximum memory of 1024 MiB.

I am reporting here because it happens with Alfheim Lighting Engine, although it is listed as HEI.
When I unplug Alfheim, it never crashes with OutOfMemoryError, and when I introduce it, it errors starting up.
Incidentally, I started to use it as a changeover from 燐/Hesperus.

1.12.2-outmem.zip
Reproducibility

[X] I reproduced this issue with as few other mods as possible installed.
[ ] I am unable to reproduce this issue consistently.

Screenshots and/or videos
No response
Logs
https://mclo.gs/biGPeks
Debug Logs
No response
Crash Reports
No response
Alfheim Version
1.3
Forge Version
<IP_ADDRESS>60
MixinBooter Version
9.1
Java Version
1.8.0_412
Java Virtual Machine
Hotspot
Operating System
ArchLinux & Windwos11
GPU Vendor
NVIDIA
Code of Conduct

[X] I agree to follow this project's Code of Conduct

probably due to #46, fixed in GregTechCEu/GregTech#2475


ModernMT 3.3 does not load models at all

I've somehow managed to train two engines in ModernMT 3.3. I got some Java errors in the beginning, that disappeared after a restart. Now I get "Loading models." The log says "Loading FastAlign models." Then absolutely nothing happens. The GPU fans are not spinning, there are no errors... just absolutely nothing, and nothing in the logs either.
How come?

@LoekvanKooten
the step "Loading FastAlign models" (as well as few others) does not use GPU, but only CPU.

Okay, but that still does not explain why modernMT does not move on. After 10 minutes it still says Loading models, while in the mmt7 directory, exactly the same model starts up straight away.

Hi Loek!
A quick note: as stated in the release info, v3.3 aligner models are not compatible with previous versions. So if you are trying to use old models with new code, Aligner models won't load.
Good news: Aligner models are not required for translation! So you can edit the engine.conf file and disable the Aligner (more info in our wiki)

I should add that 3.0 does not load models trained in 3.3, but then again, neither does 3.3 itself. What I'm trying to say is that the eligner models created by 3.3 itself cannot be loaded by 3.3.
As an intermediary solution I'll try to turn them off using the above instructions, but it's still strange this happens, I think?

Mmmmh... with  it does indeed run. Maybe I didn't start with an empty directory during training, so that there are still old aligner remnants in it?
 has no influence on the translation quality or whatsoever? It's only used during training? Please forgive me for not being aware of the inner workings of the software.

Correction: one of the engines does load, but the other one does not and gets stuck at the very same step. I double-checked: it gets stuck as Loading FastAlign, no error is generated and the config explicitly mentions aligned enabled="false"

LOL, and now it does start. I changed absolutely nothing. The behavior is inconsistent and apparently not related to memory issues. I think we need a more verbose log, but I already use -v2

There's another issue. Even when the models, trained in 3.3, are loaded correctly, the ports are closed immediately. Using netstat -tuplen | grep java I can see that java is listening when the engines are being started, but as soon as they have been started, java stops listening and all I get is connection refused.
The only engine that is working in 3.3 is the one that was not trained in 3.3 but in 3.0: this one keeps listening. When I then use exactly the same port settings for an engine trained in 3.3, the ports lock down immediately after start.
Conclusion: java does not like the engines trained in 3.3 and refuses to serve them.
These are absolute showstoppers at the moment. I have no choice but to move back to 3.0.

Hi @LoekvanKooten
Thanks for the report, I was able to replicate the bug. I'm investigating right now!

Hi @LoekvanKooten
thanks again for your help, we have just published a new release (v3.3.1) that fixes both problems: FastAlign models and neural networks models.
Let us know if this solves your issue.

I just trained my first engine in 3.3. It now loads and serves (without needing to use aligner = false or anything else for that matter). Thank you for the quick fix!

Great! Thanks again for your report!


Request Body must be marked as required where applicable

Bileto is currently upgrading openapi generator to the latest version 7.7.0 and its Java generator got fixed logic to handle json nullable/optional cases. Now the generator produces request body as optional where it doesn't make any sense, e.g.

POST /offers - you have to have some request parameters to provide
POST /bookings/{bookingId}/fulfillments
POST /bookings/{bookingId}/refund-offers
PATCH /bookings/{bookingId}/purchaser - not sure if providing null requests means to delete the purchaser alltogether
PATCH /bookings/{bookingId}/passengers/{passengerId} - does providing null passenger data means to delete all passenger details? or the once not needed for pricing?
POST /bookings-search
POST /places

The patch is to add required: true for any body object that must be provided.
Formally this is breaking change, but we consider it to be a bug since empty requests are not valid use case.

TODO josef branch for 4.0 and 3.4


⚠ Bofrost.de has degraded performance

In 625bd61, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1328 ms

Resolved: Bofrost.de performance has improved in eea01eb after 8 minutes.