Does not support wildcards in topic,  like + or #

First of all, thank for creating this Angular mqtt module.
I was trying to subscribe topic like event/#, then it crashed.
Checking the code, since the message comes from the topic maybe event/ping, but in store array, it only keeps event/# which means there is no stream for topic event/ping
    MqttService.prototype.updateTopic = function (topic, message) {
        var msg;
        try {
            msg = JSON.parse(message);
        }
        catch (ex) {
            msg = message;
        }
        this._store[topic].stream.next(msg);     <======== Here
    };

Not sure how to fix though.

Hi, firstly, thank you to the person who made this Angular mqtt module.
Had a similar problem, included is the workaround I used to be able to use the '#' wildcard, will later look at implementing something to allow the '+' as well...
    MqttService.prototype.updateTopic = function (topic, message) {
        var msg;
        try {
            msg = JSON.parse(message);
        }
        catch (ex) {
            msg = message;
        }

        var topics = Object.keys(this._store);
        var self = this;
        topics.forEach(function(i) {
            if(self._store[i].grant.topic === '#') {
                self._store[i].stream.next(msg);
            }
            else if(self._store[i].grant.topic === topic)
            {
                self._store[i].stream.next(msg);
            }
            else {
                var topicinstore = i.split('/');
                var updatedtopic = topic.split('/');
                var isany = false;
                for(var x = 0; x < topicinstore.length; x++)
                {
                    if(topicinstore[x] !== '#' && updatedtopic.length > x)
                    {
                        if(topicinstore[x] === updatedtopic[x])
                        {
                            isany = true;
                        }
                        else {
                            isany = false;
                        }
                    }
                }
                if(isany == true && topicinstore[topicinstore.length -1] === '#') {
                    self._store[i].stream.next(msg);
                }
            }
        });
        //this._store[topic].stream.next(msg);
    };

Please do not hesitate to use this if it helps or even to improve it.


Fix Ubuntu Script for 20.04

Describe problem solved by this pull request
I set up a new Ubuntu 20.04 VM and ran into a bunch of issues. I made some changes to the setup script to avoid all of them.
Describe your solution
The commit messages clearly describe the steps.
Test data / coverage
I retested the changes after resetting the VM to a snapshot just after installation.
Additional context
The additional python package requirement for matplotlib with an exact old version added in https://github.com/PX4/Firmware/pull/15203 was the main reason the script failed on 20.04.

FWIW (since I can't merge) looks good to me.

Looks good. I still want to get this into the containers to keep everything synced nicely. https://github.com/PX4/containers/pull/267

This creates a warning for me: sudo: dmidecode: command not found

This creates a warning for me: sudo: dmidecode: command not found

@julianoes In docker? With normal Ubuntu 20.04 desktop it's preinstalled. Maybe I need to add it for minimal/docker cases.

@julianoes I added the package now before I forget: https://github.com/PX4/Firmware/pull/15388


suns.py does not work on a solaredge inverter

Here is the traceback when I try to get sunspec data out of our solaredge inverter with the suns.py command:
$ suns.py -i <IP_ADDRESS> -a 1 -T 30

Timestamp: 2017-10-20T15:29:41Z
Traceback (most recent call last):
  File "/usr/bin/suns.py", line 88, in <module>
    sd.read()
  File "/usr/lib/python2.7/dist-packages/sunspec/core/client.py", line 477, in read
    self.device.read_points()
  File "/usr/lib/python2.7/dist-packages/sunspec/core/client.py", line 101, in read_points
    model.read_points()
  File "/usr/lib/python2.7/dist-packages/sunspec/core/client.py", line 246, in read_points
    raise SunSpecClientError(e)
sunspec.core.client.SunSpecClientError: Modbus read error: Connection error: [Errno 111] Connection refused

Note that the inverter modbus seems to work though as I'm able to retrieve data through a perl tcp modbus client:
$ perl mbtget -r3 -n 100 -a 40001 <IP_ADDRESS>
values:
  1 (ad 40001): 28243
  2 (ad 40002):     1
  3 (ad 40003):    65
  4 (ad 40004): 21359
  5 (ad 40005): 27745
  6 (ad 40006): 29253
  7 (ad 40007): 25703
  8 (ad 40008): 25888
  9 (ad 40009):     0
 10 (ad 40010):     0
 11 (ad 40011):     0
 12 (ad 40012):     0
 13 (ad 40013):     0
 14 (ad 40014):     0
 15 (ad 40015):     0
 16 (ad 40016):     0
 17 (ad 40017):     0
 18 (ad 40018):     0
 19 (ad 40019):     0
 20 (ad 40020): 21317
 21 (ad 40021): 13110
 22 (ad 40022): 14384
 23 (ad 40023):     0
 24 (ad 40024):     0
 25 (ad 40025):     0
 26 (ad 40026):     0
 27 (ad 40027):     0
 28 (ad 40028):     0
 29 (ad 40029):     0
 30 (ad 40030):     0
 31 (ad 40031):     0
 32 (ad 40032):     0
 33 (ad 40033):     0
 34 (ad 40034):     0
 35 (ad 40035):     0
 36 (ad 40036):     0
 37 (ad 40037):     0
 38 (ad 40038):     0
 39 (ad 40039):     0
 40 (ad 40040):     0
 41 (ad 40041):     0
 42 (ad 40042):     0
 43 (ad 40043):     0
 44 (ad 40044): 12336
 45 (ad 40045): 12339
 46 (ad 40046): 11826
 47 (ad 40047): 12337
 48 (ad 40048): 13824
 49 (ad 40049):     0
 50 (ad 40050):     0
 51 (ad 40051):     0
 52 (ad 40052): 14131
 53 (ad 40053): 12593
 54 (ad 40054): 12343
 55 (ad 40055): 13123
 56 (ad 40056):     0
 57 (ad 40057):     0
 58 (ad 40058):     0
 59 (ad 40059):     0
 60 (ad 40060):     0
 61 (ad 40061):     0
 62 (ad 40062):     0
 63 (ad 40063):     0
 64 (ad 40064):     0
 65 (ad 40065):     0
 66 (ad 40066):     0
 67 (ad 40067):     0
 68 (ad 40068):     2
 69 (ad 40069):   101
 70 (ad 40070):    50
 71 (ad 40071):   102
 72 (ad 40072):   102
 73 (ad 40073): 65535
 74 (ad 40074): 65535
 75 (ad 40075): 65534
 76 (ad 40076):  2374
 77 (ad 40077): 65535
 78 (ad 40078): 65535
 79 (ad 40079): 65535
 80 (ad 40080): 65535
 81 (ad 40081): 65535
 82 (ad 40082): 65535
 83 (ad 40083): 15959
 84 (ad 40084): 65534
 85 (ad 40085):  5012
 86 (ad 40086): 65534
 87 (ad 40087): 24435
 88 (ad 40088): 65534
 89 (ad 40089): 18503
 90 (ad 40090): 65534
 91 (ad 40091):  6531
 92 (ad 40092): 65534
 93 (ad 40093):    30
 94 (ad 40094):  7612
 95 (ad 40095):     0
 96 (ad 40096):  4280
 97 (ad 40097): 65532
 98 (ad 40098):  3785
 99 (ad 40099): 65535
100 (ad 40100): 16202

Any idea what is wrong with suns.py here?

I have encountered similar problems with older inverters (SolarMax) using pysunspec.
I'm guessing these inverters simply don't support the sunspec modbus standard.
After some research I found this python lib for solarmax communication (it is the same as mentioned by @bbinet - but in python).
python-solarmax
All tested with python 2.7 and working fine.
I have created a python 3 branch which still needs some testing.
Hope this helps you.

Have you tried a different device ID? For example it seems SMA uses ID: 126 rather than Id: 1

I came across this exact same error on a brand new solaredge inverter:
SE6000H-US000BNU4 with firmware 4.7.19
Looking at the wireshark logs, I found that the code does a series of 9 modbus reads before closing the socket. The error occurs when the code tried to open a new socket 40ms later. The inverter responds with the RST flag. Maybe the inverter only allows a single connection, and we are trying to open a new socket before it can fully clean up after the last one?
As a workaround, I added the following line at line 333 in core.py to introduce a delay.
time.sleep(0.1)
This seems to work 99% of the time now. I now see that it opens and closes the socket 5 times and took about 990ms to read all values.
A better long term solution would be to perform all modbus reads with the same socket to avoid this situation entirely.

I came across this exact same error on a brand new solaredge inverter:
SE6000H-US000BNU4 with firmware 4.7.19
Looking at the wireshark logs, I found that the code does a series of 9 modbus reads before closing the socket. The error occurs when the code tried to open a new socket 40ms later. The inverter responds with the RST flag. Maybe the inverter only allows a single connection, and we are trying to open a new socket before it can fully clean up after the last one?
As a workaround, I added the following line at line 333 in core.py to introduce a delay.
time.sleep(0.1)
This seems to work 99% of the time now. I now see that it opens and closes the socket 5 times and took about 990ms to read all values.
A better long term solution would be to perform all modbus reads with the same socket to avoid this situation entirely.

I have the same issue, adding time.sleep(0.1) in pysunspec/sunspec/core/client.py line 333 helps

Hopefully manufacturer specific issues are being pointed out to the manufacturers.  Even if it is determined that there is an issue with pysunspec maybe they could help out with a PR to fix it.

Hopefully manufacturer specific issues are being pointed out to the manufacturers. Even if it is determined that there is an issue with pysunspec maybe they could help out with a PR to fix it.

I have already sent several service requests to SE.  One of them was that the phase-to-phase voltage of the meter connected to the inverter and read from the inverter is roughly -230V, when I read it directly from the meter it is about 400V.  For today, the request has not been processed.  Posted 6 months ago.  So I wouldn't count on SolarEdge on this issue :)

As of April 2020, SolarEdge has not fixed this problem.
Model:  SE7600H-US000BNU4
Communications CPU:     4.8.28
I added code in /sunspec/core/modbus/client.py to ensure at least a 50ms delay between disconnect and connect.
Not thoroughly tested yet.
I think that the pysunspec library is the appropriate place to fix this.  Manufacturers may have hardware limitations on the speed at which connections can be re-made.  Wouldn't it be nice if this was a parameter in the "common" block?
As @fgnn mentions, the sunspec library is connecting and disconnecting multiple times to read data; his suggestion of using the same connection for multiple operations has merit.  When doing a device.read(), the library connects, reads the first module's data, disconnects, then immediately reconnects to read the next module's data.  This is because the modbus.client.read() function checks to see if a connection is open, and if not opens one for just this read operation, then disconnects.  When doing a device.read(), that code should handle the connect/disconnect instead of pushing it to the modbus layer.

Hi,
I tried to locate and write the code "time.sleep(0.1)" but get still the same result as bbinet: (In both pysunspec/sunspec/core/client.py and pysunspec/sunspec/core/modbus/client.py
pi@solaredgepi:~/pysunspec $ suns.py -i <IP_ADDRESS> -P 1502 -a 1 -T 60
Traceback (most recent call last):
File "/usr/local/bin/suns.py", line 71, in
sd = client.SunSpecClientDevice(client.TCP, options.a, ipaddr=options.i, ipport=options.P, timeout=options.T)
File "/usr/local/lib/python2.7/dist-packages/sunspec/core/client.py", line 775, in init
self.device.scan(progress=scan_progress, delay=scan_delay)
File "/usr/local/lib/python2.7/dist-packages/sunspec/core/client.py", line 218, in scan
self.modbus_device.connect()
File "/usr/local/lib/python2.7/dist-packages/sunspec/core/modbus/client.py", line 678, in connect
raise ModbusClientError('Connection error: %s' % str(e))
sunspec.core.modbus.client.ModbusClientError: Connection error: [Errno 111] Connection refused
The inverter can communicate to an another pi, (home assistans). I dont know how I managed that... so the inverter configured to communicate.

@pkgdata I haven't researched thoroughly, but I suspect the sunspec library is trying to open the client twice.  The SolarEdge documentation specifically states that the device only supports ONE connection at a time.  I think (again more research needed) that the reason putting in a delay works is that it gives time for the device to complete and disconnect before the next connection begins.  I encountered the problem if I used device.read(), but avoided the problem if I read each module individually.
I'm not ready to publish my work-around, but had to use class variables to ensure an adequate delay; instance variables didn't work.  (Thus the belief that sunspec is opening multiple instances of the client.)
I don't believe the problem is with SolarEdge.  Your other Pi may be slow enough or using the library in a way that doesn't encounter the problem.

@MojaveTom , thank you for answer...
As some have written before, it seems to work when making a time.sleep (0.1)
But I'm not a programmer and when I type this line in line 333 doesn't seem to have helped. (in addition, the 2 client.py files are two completely different files, and line 333 looks different)
in /home/pi/pysunspec/sunspec/core/client.py
def read_points(self):
"""Read all points in the model from the physical device.
"""
    if self.model_type is not None:
        # read current model
        try:
            end_index = len(self.read_blocks)
            Row 333: ........... if end_index == 1:
                data = self.device.read(self.addr, self.len)
            else:
                data = b''
                index = 0
                while index < end_index:
                    addr = self.read_blocks[index]
                    index += 1
                    if index < end_index:
                        read_len = self.read_blocks[index] - addr
                    else:
                        read_len = self.addr + self.len - addr
                    data += self.device.read(addr, read_len)
            if data:
                # print('data len = ', len(data))
                data_len = len(data)/2
                if data_len != self.len:
                    raise SunSpecClientError('Error reading model %s' % self.model_type)

in /home/pi/pysunspec/sunspec/core/modbus/client.py
else:
return
else:
raise ModbusClientError('Client serial port not open: %s' % self.name)
    if sys.version_info > (3,):
        resp = bytes(resp, 'latin-1')

    return resp
row 333.......
def _write(self, slave_id, addr, data, trace_func=None):
    resp = ''
    len_remaining = 5
    len_found = False
    except_code = None
    func = FUNC_WRITE_MULTIPLE
    len_data = len(data)
    count = int(len_data/2)

Are these the right lines?
Should I put time.sleep (0.1) in both?

@pkgdata Putting a delay in /pysunspec/sunspec/core/modbus/client.py is the more effective place; this is the lowest level module to actually interact with the TCP connection.  Putting delays at a specific line number is risky since as you noticed your sources may be different.  I'm sure you just want to get your project going, as I did, so I'll attach my modified file.  Realize that this is a POOR EXCUSE for finding the real problem and fixing that.  This just ensures a .05 sec delay between socket (TCP) disconnect and connect.  As I said, this is a band-aid to let the SolarEdge inverter finish up one connection before starting another.  The real solution is to not try to make multiple connections so quickly, instead just re-use an already open connection.  I have only tested this code on a Mac Mini, I have no idea if it will work on your Pi.
The parameter that sets the delay time is called MIN_RECONNECT_TIME_SEC defined near the top of the file.  The code modifications are in the class ModbusClientDeviceTCP; functions connect() and disconnect() which are near the middle of the file.  The name of the file indicates which file from the distribution is to be replaced with the modified file (name changed to the name of the replaced file). [Looks like I can only include a zip of the file, so you have to unzip it before using it.]
Some day when I'm looking for a distraction, I'll dig into the sunspec code and fix it correctly.
Good luck with this,
Tom
Modified-sunspec-core-modbus-client.py.zip


[WIP] Fix block sync problem

:white_check_mark: Build AElf <IP_ADDRESS>0 completed (commit https://github.com/AElfProject/AElf/commit/e147a2317d by @zhxymh)

:x: Build AElf <IP_ADDRESS>7 failed (commit https://github.com/AElfProject/AElf/commit/f0df011d6b by @zhxymh)

:white_check_mark: Build AElf <IP_ADDRESS>3 completed (commit https://github.com/AElfProject/AElf/commit/16bfe1d7d5 by @zhxymh)

:white_check_mark: Build AElf <IP_ADDRESS>5 completed (commit https://github.com/AElfProject/AElf/commit/b6230d9878 by @zhxymh)

@shiwk @rosona also, reviewers need to check the rules

:white_check_mark: Build AElf <IP_ADDRESS>2 completed (commit https://github.com/AElfProject/AElf/commit/78316e19c3 by @zhxymh)

add retry logic to grpc https://github.com/grpc/grpc/issues/15623#issuecomment-398487843
fetch blocks from multiple n nodes, save the fast one, and random fetch from n - 1 nodes, save the fast one, and continue ...
save blocks in one queue, and process block in another queue.

test this branch online until it works, solve this problem before 23:59 pm.

switch online test environment to this branch

switch online test environment to this branch

We are already using this branch.

:x: Build AElf <IP_ADDRESS>3 failed (commit https://github.com/AElfProject/AElf/commit/731e803e73 by @zhxymh)

:x: Build AElf <IP_ADDRESS>5 failed (commit https://github.com/AElfProject/AElf/commit/e4e1dc7413 by @zhxymh)

:x: Build AElf <IP_ADDRESS>3 failed (commit https://github.com/AElfProject/AElf/commit/ee1756172b by @zhxymh)

:x: Build AElf <IP_ADDRESS>8 failed (commit https://github.com/AElfProject/AElf/commit/da142ed958 by @zhxymh)

:x: Build AElf <IP_ADDRESS>4 failed (commit https://github.com/AElfProject/AElf/commit/6588ad8e24 by @jason-hoopox)

:x: Build AElf <IP_ADDRESS>2 failed (commit https://github.com/AElfProject/AElf/commit/b4a6efbb5b by @zhxymh)

:x: Build AElf <IP_ADDRESS>1 failed (commit https://github.com/AElfProject/AElf/commit/81b9d97f66 by @samfrb)

:x: Build AElf <IP_ADDRESS>5 failed (commit https://github.com/AElfProject/AElf/commit/dfa92effa4 by @samfrb)

:x: Build AElf <IP_ADDRESS>3 failed (commit https://github.com/AElfProject/AElf/commit/cd8f176e4f by @samfrb)

:x: Build AElf <IP_ADDRESS>6 failed (commit https://github.com/AElfProject/AElf/commit/d9f30cc300 by @samfrb)

:x: Build AElf <IP_ADDRESS>0 failed (commit https://github.com/AElfProject/AElf/commit/7d2f05a80e by @samfrb)

:white_check_mark: Build AElf <IP_ADDRESS>1 completed (commit https://github.com/AElfProject/AElf/commit/06a906daa4 by @zhxymh)

:x: Build AElf <IP_ADDRESS>5 failed (commit https://github.com/AElfProject/AElf/commit/364f1eadeb by @samfrb)

Codecov Report

Merging #1710 into dev will decrease coverage by 0.2%.
The diff coverage is 84.49%.

@@            Coverage Diff             @@
##              dev    #1710      +/-   ##
==========================================
- Coverage   85.16%   84.96%   -0.21%
==========================================
  Files         434      444      +10
  Lines       16020    16417     +397
==========================================
+ Hits        13644    13948     +304
- Misses       2376     2469      +93

Impacted Files
Coverage Δ

src/AElf.OS/OSAElfModule.cs
100% <ø> (ø)
:arrow_up:

src/AElf.Kernel.Types/Helpers/TimestampHelper.cs
100% <100%> (ø)

src/AElf.WebApp.Application.Net/NetAppService.cs
100% <100%> (ø)
:arrow_up:

src/AElf.OS.Network.Grpc/GrpcPeerInfo.cs
100% <100%> (ø)

src/AElf.WebApp.Application.Net/Dto/PeerDto.cs
100% <100%> (ø)
:arrow_up:

...s.DPos/DPoSAnnouncementReceivedEventDataHandler.cs
75.36% <100%> (-24.64%)
:arrow_down:

src/AElf.OS.Network.Grpc/AuthInterceptor.cs
92.85% <100%> (ø)
:arrow_up:

src/AElf.OS.Network.Grpc/GrpcNetworkModule.cs
100% <100%> (ø)
:arrow_up:

...ernel.Core/Blockchain/Events/BlockAcceptedEvent.cs
100% <100%> (ø)
:arrow_up:

src/AElf.OS.Core/Network/NetworkOptions.cs
71.42% <100%> (ø)
:arrow_up:

... and 42 more

Codecov Report

Merging #1710 into dev will decrease coverage by 0.2%.
The diff coverage is 84.49%.

@@            Coverage Diff             @@
##              dev    #1710      +/-   ##
==========================================
- Coverage   85.16%   84.96%   -0.21%
==========================================
  Files         434      444      +10
  Lines       16020    16417     +397
==========================================
+ Hits        13644    13948     +304
- Misses       2376     2469      +93

Impacted Files
Coverage Δ

src/AElf.OS/OSAElfModule.cs
100% <ø> (ø)
:arrow_up:

src/AElf.Kernel.Types/Helpers/TimestampHelper.cs
100% <100%> (ø)

src/AElf.WebApp.Application.Net/NetAppService.cs
100% <100%> (ø)
:arrow_up:

src/AElf.OS.Network.Grpc/GrpcPeerInfo.cs
100% <100%> (ø)

src/AElf.WebApp.Application.Net/Dto/PeerDto.cs
100% <100%> (ø)
:arrow_up:

...s.DPos/DPoSAnnouncementReceivedEventDataHandler.cs
75.36% <100%> (-24.64%)
:arrow_down:

src/AElf.OS.Network.Grpc/AuthInterceptor.cs
92.85% <100%> (ø)
:arrow_up:

src/AElf.OS.Network.Grpc/GrpcNetworkModule.cs
100% <100%> (ø)
:arrow_up:

...ernel.Core/Blockchain/Events/BlockAcceptedEvent.cs
100% <100%> (ø)
:arrow_up:

src/AElf.OS.Core/Network/NetworkOptions.cs
71.42% <100%> (ø)
:arrow_up:

... and 42 more


Increase parent height programatically does not affect elements

Initial HTML:

<vue-draggable-resizable :parent="true">

Hello world

</vue-draggable-resizable>

Okay, it is rendered. Now i want to increase height of drag-parent element:
let dragParent = document.getElementById('drag-parent');
const t = dragParent.offsetWidth + 300
dragParent.style.width = t.toString() + "px";

And now, height of drag-parent is 600px, but drag element i can only in height of 300px.
If i add new one element dynamically, it will count with new height 600px.

Any feedback?

See #133


make compatible with python versions <3.9

using in python v3.8 gives following error message
nmq407@/projects/petar/fgf1/01_resolve$ python /projects/petar/fgf1/code/MindaGap/mindagap.py microscope_image.tiff 3 40 --edges False
Traceback (most recent call last):
  File "/projects/petar/fgf1/code/MindaGap/mindagap.py", line 110, in <module>
    parser.add_argument("-v", '--version', action=argparse.BooleanOptionalAction, default = False, help="Print version number.")
AttributeError: module 'argparse' has no attribute 'BooleanOptionalAction'

This is because BooleanOptionalAction class was added to the argparse module in Python 3.9. The behavior can be preserved and made compatible with older versions with the proposed change.

Thanks @pvtodorov!


Add the Pages class and massively refactor PageResult

The Pages class is the next level up in the PaginatedResult
abstraction hierarchy above PageResult. It provides:

a single constructor that takes a (copied) closure that retrieves
successive pages from the API service.
iterator-returning begin() and end() methods.

The refactor for PageResult includes turning it into a nested class in
PageResult and removing all ElementAccessor members and parameters:
ElementAccessor is intended to be a lightweight, stateless, uniform wrapper, and is
now used purely as a default-constructed throwaway.

What is the benefit of making PageResult nested vs a standalone class?

The nested class obviates duplication of the templates and template constraints and means there aren't as many forward declarations necessary. It's purely a maintenance and convenience decision.

Open review bump
@vam-google @michaelbausor @coryan

I'm not convinced that a nested class is the way to go. Is the benefit for maintenance significant? I think that the PageResult class should logically be considered separate from a Pages class. It is "lower" in the hierarchy of classes being implemented for pagination, and could be returned directly from an RPC method on the class, letting users interact with it without reference to a Pages class.

I'd like to not duplicate the template boilerplate if at all possible, and I'd like to keep the accessor and next-page retrieval in the template. I'll try some cleverness and report back with how onerous the predicted maintenance burden becomes.

I'd like to not duplicate the template boilerplate if at all possible, and I'd like to keep the accessor and next-page retrieval in the template. I'll try some cleverness and report back with how onerous the predicted maintenance burden becomes.

Result is not as bad as I had feared. There's some duplication of the template, but it should be managable.

Open review bump
@vam-google @michaelbausor @coryan


Add new template fish Shell

New

[x] Template - New .gitignore template

Details
For more information, see: https://fishshell.com/

Hey @erdaltsksn , thanks for contributing!
One question from someone that doesn't use Fish (ZSH user here): Aren't these files stored in the home path? Do you version control your entire home folder?
I don't mind adding this template at all. I'm just curious actually.

Aren't these files stored in the home path?

They use $XDG_CONFIG_HOME actually. Default path for them is $HOME/.config/fish.
Personally, I clone my dotfiles inside .config and create symbolic links to the folders, but I saw some people prefer to add the whole .config folder under version control. I don't do that to be safe than sorry.


LengthVariable allow 0?

I'd like to ignore zero values from the LengthVariable linter.
This is my config:
  LengthVariable:
    enabled: true
    allowed_lengths: [0]

But I still get this:
Length literals like 0 should only be used in variable declarations;...
How can I get the effect I require?

It looks like this might be a limitation in how LengthVariable works. We'd be happy to look at a PR fixing this!

Easy fix for this would be allowed_lengths: ["0"]
When you put smth like allowed_lengths: [0, 20px], config['allowed_lengths'] outputs [0, "20px"].
I have no idea how to fix it in code though.


Update Canada city boundaries

@maning close here when tileset is updated

This is done, same tile id as before.

@maning Studio says source is missing

Can you give the tileset id?

Updated tileset: https://www.mapbox.com/studio/tilesets/maning.worldcities/

Done http://mapbox.github.io/osm-navigation-map/#8.1/49.2029/-123.0050


Poposal: Controller.Json() overloads are confusing in 3.0, expose extensions methods.

In previous versions of ASP.NET Core, the overloads for rendering Json were:
public virtual JsonResult Json(object data, JsonSerializerSettings serializerSettings);
public virtual JsonResult Json(object data);

Here, JsonSerializerSettings is Newtonsoft.Json.JsonSerializerSettings.
Since decoupling the dependency in 3.0 though, it's now:
public virtual JsonResult Json(object data, object serializerSettings);
public virtual JsonResult Json(object data);

While this makes sense to decouple from Newtonsoft specifically, it makes for a poor user experience. For example, I couldn't readily find what type I'm supposed to be using for the serializerSettings argument.
Is there harm in adding additional overloads that are specific? For example, the Microsoft.AspNetCore.Mvc.NewtonsoftJson package could add a:
public static Json(this Controller controller, object data, JsonSerializerSettings serializerSettings);

It'd expose a specific overload for the user to use. The same for System.Text.Json.
If all of this isn't doable for API compat reasons, at the very least the docs should be updated. The out-of-the-box experience with System.Text.Json is very confusing and leaves a user lost on what they're supposed to be doing.

Unfortunately extension methods don't really help with this. We already considered this during the initial design and rejected it for the following reason.
When you use an extension method on this you have to qualify it. So it means that in the following example you won't see the extension method offered as an example.
public class MyController : Controller
{
    public IActionResult Foo()
    {
        return Json(...); // Can't resolve Json(this Controller controller, object data, JsonSerializerSettings serializerSettings);
    }
}

You'd have to write:
public class MyController : Controller
{
    public IActionResult Foo()
    {
        return this.Json(...); // Can't resolve Json(this Controller controller, object data, JsonSerializerSettings serializerSettings);
    }
}

Ah that's a great point. I guess it's a docs issue then, though the API is unfriendly/unclear and that's unfortunate. My current workaround is creating an overload on my base controller class:
protected JsonResult Json(object data, System.Text.Json.Serialization.JsonSerializerOptions serializerOptions = null)
    => serializerOptions != null
    ? base.Json(data, serializerOptions)
    : base.Json(data);

...just in case that helps anyone else. Note calling Json(data, (anything)null) throws an error in the default Json(object, object) overload, so you have to make sure not to call it with one.

@pranavkm - that last part sounds like something we should fix.

Null overload issue was fixed in https://github.com/aspnet/AspNetCore/pull/12207 - thanks!

@rynowak Since we can't fix things in this way and the documentation has improved it seems I should just close this out, any objections?

Yep, this can be closed.


محفوظة فعالي قابلي في تركيا اضنه خدمات أرباح الإنترنت youtube.com/@ahmetcemmalfirebase loginError Code: 50058
Request Id:

https://github.com/google-pay/wallet-samples/issues/39

Hello,
If you are experiencing an issue with the Google Wallet samples or have a question about how to use them, please create a new issue and be sure to include any relevant comments.
Thank you!


Add Spotless run configuration for format

Issue

https://github.com/DroidKaigi/conference-app-2023/issues/1034

Overview (Required)
I add Spotless run configuration for format.
As I had asked before, I reinstalled Spotless.
https://github.com/DroidKaigi/conference-app-2023/issues/1034#issuecomment-1703854345
Is what I did correct?
Screenshot (Optional if screenshot test is present or unrelated to UI)

We are using detekt like this.
./gradlew detekt --auto-correct
It uses ktlint internally. So we don't have to introduce spotless. 👀

@takahirom
Thank you for your reply!
So, I should have added a configuration file for detekt.
I removed spotless and added a configuration file to run for detekt.
Please check. 🙏🏼


ICMP packets generated by local node can't be encoded

Since the local node is not part of the routing table, the shared secret can't be retrieved there so this needs a bit of separate logic

On a closer look, this is only a problem for time exeeded messages when doing a traceroute check

Not a priority atm


field refreshToken is not async safe

consider this query
obtainPayload {
    payload {
      origIat
      exp
      username
    }
    refreshExpiresIn
    # no refresh token here
    token
  }

Works fine.
But this one:
mutation obtainJWT {
  obtainPayload {
    payload {
      origIat
      exp
      username
    }
    refreshExpiresIn
    refreshToken //  <<<
    token
  }
}

gives this error
"errors": [
    {
      "message": "You cannot call this from an async context - use a thread or sync_to_async.",
      "locations": [
        {
          "line": 11,
          "column": 7
        }
      ],
    }
]

This is possibly the line.
https://github.com/KundaPanda/strawberry-django-jwt/blob/17d520efcb1f2e2cfefc2744d283b7c93ea8d540/strawberry_django_jwt/refresh_token/shortcuts.py#L24

@KundaPanda Any updates on that?

Yes I have tracked down some code that might cause this, looking into it now.

Would you mind to create a release for that?


Redspread link

This is a...

[ ] Feature Request
[x] Bug Report

Problem:
https://redspread.com fails with a "Secure Connection Failed" error, removing the https:// has it redirect to https://coreos.com
Proposed Solution:
should probably just go to https://coreos.com at this point
Page to Update:
https://kubernetes.io/...

I signed CLA.


Handling of compressed files

Dear Togo,
I am working a lot with both VASP, pymatgen and phonopy. Very often, I am gzipping my files (simplify data transfer/safe storage etc). Pymatgen can work with gzipped files directly by using monty https://guide.materialsvirtuallab.org/monty/ and zopen instead of open.
Would there be interest to add such a support for phonopy as well?
Thank you very much.
Best
JG

I don't see exactly what you expect. But I think compression is a good idea. HDF5 files support compression.
In my usage, I usually dump phonopy data (e.g. displacements, forces, NAC params) into phonopy_params.yaml-like-file using API, Phonopy.save(), or command line options, and then use xz command to compress it. Phonopy supports .gz and .xz in reading the yaml file:
https://github.com/phonopy/phonopy/blob/063e48505b866468bbd84eedd914cf7b1b34ac32/phonopy/interface/phonopy_yaml.py#L643-L669
References to API and command options:
https://phonopy.github.io/phonopy/phonopy-module.html#shortcut-to-load-input-files-phonopy-load
https://phonopy.github.io/phonopy/phonopy-load.html
https://phonopy.github.io/phonopy/setting-tags.html#summary

Thank you for the answers!
I would like to have the option to start the creation of the force constants or BORN file from compressed vasprun.xml's.

Do you mean each vasprun.xml file is compressed?

How about having the following function?
https://github.com/phonopy/phonopy/blob/f354d82f3673999dc93855bd07add0f2ebf23ac9/phonopy/file_IO.py#L927-L949

Thank you for the suggestion. I'm not yet sure how this would be used in the most efficient way. Isn't it simpler to replace "open" with a function or method that can also handle compressed files?

I'm not yet sure how this would be used in the most efficient way.

What is the most efficient way?
I can not see your use-case. I understand you have it in your mind, but your scope may be limited to your use case but I probably have to support people in wider scope. You can explain more details, otherwise no chance for me to understand what we should do.

I understand. Would it be okay if I simply draft a pull-request at some point to indicate and explain my thoughts better?

My question is probably not very much about the code, but the user-case. If I know the use case, I may be able to solve it directly or indirectly.
So if your use-case is, for example,
% phonopy -f vasprun-1.xml.xz  vasprun-2.xm.xz  vasprun-3.xml.xz ..

Then the code may be written as
myio = get_module_to_decompress(filename)
with myio.open(filename) as f:
    ...

If this is the requirement, a build-in simple solution is better than relying on an external library. Dependency is not as controllable as build-in code (https://abseil.io/resources/swe-book/html/ch21.html).

https://github.com/phonopy/phonopy/pull/199.

@JaGeo, please let me know if it works. Testing user-interface is uneasy. Do you think I should write documentation for this?

The forces can be collected when the vasprun.xml's are gzipped (phonopy -f .... ) via the command-line when all yaml files are unzipped. (Or is there a way to work with a gzipped yaml-file?)
Creation of born file is also working when I use phonpy-vasp-born vasprun.xml-gz.
What does not seem to work is: phonopy-vasp-born  --outcar OUTCAR.gz POSCAR.gz.

It might be good to document it.

Thanks for your test.

Or is there a way to work with a gzipped yaml-file? I guess only via the API.

You are good at finding it. This is a little bit more tricky to handle. Phonopy-commands (phonopy and phonopy-load) are out of API and are standing on very fragile code base to support various complicated options.
I push an update to develop branch with fingers crossed.
https://github.com/phonopy/phonopy/commit/014b6538a904009e1d6195d83b0f403359abdc99

This has worked out. I can work with the compressed yamls now.
Only thing left that does not seem to work is reading a compressed POSCAR.gz.

Why do you compress POSCAR? It will not be so large.

True. It's fine like this from my side. Thank you in any case!


Ensure commits are properly linked to the primary/destination account

This update addresses an issue with the reliability of git config user.name and git config user.email when running bash script.sh.
To ensure the correct attribution of commits, I have updated the script to include the following changes:

Added GIT_COMMITTER_NAME= and GIT_COMMITTER_EMAIL= to explicitly set the committer's information for each individual commit.
Included the --author flag for each commit. This is essential because GitHub differentiates the author from the committer. Without specifying --author, the commit may show one person as the author (often linked to my primary account) and another as the committer (probably set by git config user.name in my experience). For this case, the commits may not count as the primary account's contribution because it has a different committer.

Additionally, I removed the use of the # symbol in commit messages, as this inadvertently links to issues filed in the repository.

I like it, although the change from numerals to text (which makes sense) might break backward compatibility. This may not be a big issue at this point, but it would still be good to announce it in the README or changelog (which doesn't exist yet).
Here, it's validated by the message to ensure the commit isn't duplicated: https://github.com/isturiz/sync-graph/blob/main/src/script.js#L36
So, changing it could cause duplicate commits if you run it in the repository where you're applying the commits.
One solution would be to delete the repository and rerun the process (or simply delete the .git directory and force push to override the history).
What do you think about? @apappas1129

Not sure what you're referring to "the change from numerals to text". Maybe that was unintentional? Can we address that? which line in my changes?

Perfect so for this PR.
The change from # to No. remains interesting for the reason you mentioned regarding issues.

Yes, it definitely needs to be fixed. The commits are causing conflicts with the issues.


Question： Why STM did not support range get yet

I have read the STM code in etcdv3 , and gonna to use in my project.
But here is a big problem there: I need the range get in STM .
Why there is no range get in STM ? Can I add this feature with a pull request?

@xiang90

@arthurkiller it's written that way because there's a potential for a Range to fetch a very large number of keys which can lead to a large number of false conflicts due to a huge readset and very large transactional compare blocks. How do you intend to use STM?

I need the STM to handle the potential conflicts while doing the update in a parallel way. We think this should let the user to decide if a large number false is acceptable.
I have also make up a STM-like facilities , it provide the isolation ability between the read committed & repeatable read Cause we really need the range get and doesn't care the failure much.
Etcd can be used in many situations and the failure acceptable is one of them I think.

@arthurkiller Would an extension to the STM interface that takes a range [begin, end) and returns a slice of key/value pairs like Range(begin, end string) [][2]string be enough?

@heyitsanthony  Actually I only need the prefix option and don't care the implementation

@heyitsanthony  would like a PR ? I can do this

@arthurkiller sure

@heyitsanthony I want to implement the AVL tree / dictionary tree instead of the map(hash map). Cause the tree can handle the prefix matching problem with less cost. How do you think this? This may add a dependency in the package

@arthurkiller there's already an interval tree in pkg/adt/interval_tree.go. I think that kind of approach will be very complicated though. An easier way: when calling Range(begin, end string) [][2]string, the STM runtime would cache the result for the begin/end pair and that's it.
One problem with Range and STM (and why I'm hesitant to diverge from the traditional single-key model): suppose a transaction T1 makes a range request [a, d), retrieving {a, b}. Concurrently, another transaction T2 inserts "c" and commits, giving {a, b, c}; there's now a read conflict on T1. How do you intend to resolve it?

This makes me confused. Question is that Serilizable read should not allow phantom read right?

@heyitsanthony  You mean T1 do the read {a,b,c} but only get the {a,b} revison and T2 write the {c} then T1 should get a conflict ???

@heyitsanthony  I see ... Seems snapshot can solve this?

@heyitsanthony  Should serializable read add the revision check when write?

Should serializable read add the revision(global) check when commit? That's not a good way , cause we get a global RWLock over etcd version.But it can avoid phantom read ,right ?

That would catch the read conflict, but would have so many false conflits it would be effectively unusable. Plus, that kind of test isn't supported in the transaction API and probably shouldn't be; if the txn model is going to be extended it should include range tests (e.g., Range(a, b).modrev < r).
Can you give a concrete example of how you're planning on using etcd / what your data model is going to be? Maybe STM isn't the best fit.

@heyitsanthony  Sure. We make up a config manager service over etcd. And here is a problem. We need to identify&manage a kind of keys(this key is the config key ,they may have a common prefix) get same prefix.We also defined some status while operating these keys.And those operation should be concurrent safety
Now I do this with a additional key to hold the revision while do the transections to isolate the transections.And it really take effect.But till I read the STM ,I realized my isolation facilities is just between the read-committed & serializable-read.
Maybe here is a another way to handle this

@arthurkiller was this the revision of the entire etcd store? that would have a lot of conflicts...
Maybe use a distributed lock? The one in clientv3/concurrency/mutex.go can guard concurrent transactions with IsOwner().

@heyitsanthony  Yep, seems STM is not suit for this kind of problem.
So, what kind of problem is suit for STM??

@arthurkiller it's suitable for problems which can be solved with ordinary software transactional memory. If the algorithm accesses keys like addresses in memory, it's OK; range queries are not part of that model.

@heyitsanthony  Thx dude. I will think about this.

Reopening since the txn API can do this as of #8025 and people still ask for this STM extension.

Also interested in this :)


Support for WSL sideloading

I prefer to do most of my dev work / terminal interactions through WSL. This has worked out fairly well by mounting the c drive so that I can run visual applications like VS Code, but use an ubuntu terminal for things like actual yarn build. I would strongly prefer to not maintain a two dev environments (in and out of WSL).
I've tried using the sideload command in a plugin dev workflow, but the root path for the sideload directory is determined through os.home() which isn't quite the desired folder when operating through WSL, and can't be symlinked from my working directory:
Error: EXDEV: cross-device link not permitted, link '/mnt/c/Users/rniem/<dev path>/excel-plugin/my-office-add-in-manifest.xml' -> '/home/ricky/Library/Containers/com.microsoft.Excel/Data/Documents/wef/my-office-add-in-manifest.xml'
Ideally this would be configurable so that WSL support can work with the mounted directory.

I'm not sure how we would configure this.


Upgrade EVM pallet to FRAMEv2

I'm going to temporarily revert this PR because we want to release Frontier to crates.io.
pallet attributes macro was technically released with Substrate 3.0, but it later still went through some changes. With that it's almost impossible to have a version that support both Substrate 3.0 and Substrate master. Using the old decl_modules macro we can completely avoid this problem.


I'm getting an error while using the example

My code is this
use simple_ssdp::MulticastAddr;
use simple_ssdp::service::Service;
use simple_ssdp::service::ServiceDescription;

const UUID: &str = "ed5f0be2-8f63-11ee-b9d1-0242ac120002";

pub async fn ssdp_master() -> ! {
    let desc = ServiceDescription {
        usn_uri: format!("uuid:{}", UUID),
        service_type_uri: "AccessTime:Multicast".to_string(),
        expiration: 100,
        location: "https://<IP_ADDRESS>/api/v1/adopt".to_string(), // TODO get a servername from conf
    };

    let service = Service::new(desc);
    service.listen(MulticastAddr::Loopback).await.unwrap();
    panic!();
}

Basically the same as the example. But I've put the necessary use statements in, I've changed the usn_uri, and I've also put panic at the end since the function doesn't return, and I don't think the service should terminate.
But I'm getting this error:
2024-08-28T16:45:49.945+01:00 - DEBUG Start listening for SSDP discovery messages...
thread 'tokio-runtime-worker' panicked at /home/redacted/src/ssdp.rs:16:51:
called `Result::unwrap()` on an `Err` value: Os { code: 98, kind: AddrInUse, message: "Address already in use" }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

It's looking like the service starts, but some time after starting, throws this error.
Have you seen this before or know how to remedy this?

Okay, obviously my brain was not working terribly well yesterday; I was just spawning this asynchronous function twice, which is what gave rise to my error.
That said, I'm wondering if we shouldn't set_reuse_address on the socket, so that several SSDP servers may run on the same host. At least some other implementations do this.
What do you think; have I run into a bug, "missing feature", or something else?

Thanks for your comment, to be fair I am not entirely sure as I am not really good in Rust by any means, I just needed this as a part of a bigger project. I've never come across your problem yet but from my perspective reuse address sounds like the right thing to do, do you want to test it and make a PR?


Added ability to pass a unique id

So that multiple reactive tables can appear on the same page, this request adds in the ability to supply a unique id per table instance.

Hi, I always solved this myself by putting the datatable itself in a unique wrapper. Datatable creates a unique is for itself on creation anyway. There's no need to change the add an ID to the wrapper class because of blaze's this.$ call.
What were you trying to achieve here that didn't work otherwise? I'm going to close this now because I think there are better solutions


No image arg to config file

Suggested by @chaimain (thanks !),

Move the --no-image argument in the onsite_mc_r0_to_dl3 script to the config_MC_prod.yml file. Change the script argument by an entry in the configuration file.

Also;

Quickstart and launch workflow explanations clarified in the README.
Started some unit tests.

Thanks @chaimain !


Rename IInheritable::InsertParent to improve clarity of intent

This is a followup commit for 168d28b.
By renaming IInheritable::InsertParent(com_ptr) and
InsertParent(size_t, com_ptr) into AddLeastImportantParent(com_ptr)
and AddMostImportantParent(com_ptr) respectively, we can improve
the clarity of our code's intent without the need for comments.
PR Checklist

[x] I work here
[x] Tests added/passed

I'm not sure I understand the parenting and priorities well enough to reason whether this is clearer or not. There's both a parent/child tree and an importance value assigned to each of them depending on whether they're push_front or push_back'd into the list of parents? Yes?

Yeah you could also call it PushParentBack and PushParentFront.
I prefer this renamed version for 2 reasons:

It's agnostic over the underlying container. The current API practically only works with a std::vector. But what if we want to reverse it, to have the most important one at the end?
Currently the code only calls either exactly InsertParent(somePointer) or InsertParent(0, somePointer). The positional parameter is actually never really used and thus can be replaced with something that's a bit more explicit.

I believe the correct term for our inheritance tree is a "ordered arborescence", which is a directed acyclic graph, where child lists are ordered and there's only one path from a child to its parent (or grandparents).


PHP-container: composer public keys missing

OS type
Linux
Drupal version
9
Codebase
mounted codebase
Describe the bug
docker-compose run php composer self-update --update-keys
reports
Public keys stored in /home/wodby/.composer
However,
key.dev.pem and key.tags.pem
have not been saved in /home/wodby/.composer after self-update of composer from 2.0.11 to 2.0.12. and not after a separate key update with manual insertion of the keys. I keep getting the following results (same from inside the php container):
docker-compose run php composer diagnose
Checking composer.json: OK
Checking platform settings: OK
Checking git settings: OK
Checking http connectivity to packagist: OK
Checking https connectivity to packagist: OK
Checking github.com rate limit: OK
Checking disk free space: OK
Checking pubkeys: FAIL
Missing pubkey for tags verification
Missing pubkey for dev verification
Run composer self-update --update-keys to set them up
Checking composer version: OK
Composer version: 2.0.12
PHP version: 8.0.3
PHP binary path: /usr/local/bin/php
OpenSSL version: OpenSSL 1.1.1k  25 Mar 2021
cURL version: 7.74.0 libz 1.2.11 ssl OpenSSL/1.1.1k
zip: extension present, unzip present
Output of docker info
Client:
Context:    default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
Server:
Containers: 21
Running: 11
Paused: 0
Stopped: 10
Images: 13
Server Version: 20.10.5
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.10.0-0.bpo.4-amd64
Operating System: Debian GNU/Linux 10 (buster)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.841GiB
Name: node.mydomain.tld
ID: [...]
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
<IP_ADDRESS>/8
Live Restore Enabled: false
Contents of your docker-compose.yml
version: "3.6"
networks:
drupal:
[...]
nginx:
image: wodby/nginx:$NGINX_TAG
container_name: "${PROJECT_NAME}_nginx"
depends_on:

php
traefik
networks:
drupal
environment:
NGINX_STATIC_OPEN_FILE_CACHE: "off"
NGINX_ERROR_LOG_LEVEL: debug
NGINX_BACKEND_HOST: php
NGINX_SERVER_ROOT: /var/www/html/web
NGINX_VHOST_PRESET: $NGINX_VHOST_PRESET
NGINX_DRUPAL_FILE_PROXY_URL: testsite1.mydomain.tld
volumes:
/etc/localtime:/etc/localtime:ro
./:/var/www/html:cached
labels:
"traefik.enable=true"
"traefik.http.routers.${PROJECT_NAME}_nginx.entrypoints=web"
"traefik.http.routers.${PROJECT_NAME}_nginx.rule=Host(testsite1.${PROJECT_BASE_URL})"
"traefik.http.routers.${PROJECT_NAME}_nginx.middlewares=https-redirect@file"
"traefik.http.routers.${PROJECT_NAME}_nginx-https.entrypoints=websecure"
"traefik.http.routers.${PROJECT_NAME}_nginx-https.rule=Host(testsite1.${PROJECT_BASE_URL})"
"traefik.http.routers.${PROJECT_NAME}_nginx-https.middlewares=secHeaders@file"
"traefik.http.routers.${PROJECT_NAME}_nginx-https.tls=true"
"traefik.http.routers.${PROJECT_NAME}_nginx-https.tls.certresolver=le-live"
php:
image: wodby/drupal-php:$PHP_TAG
container_name: "${PROJECT_NAME}_php"
networks:
drupal
environment:
PHP_SENDMAIL_PATH: /usr/sbin/sendmail -t -i -S mailhog:1025
volumes:
./:/var/www/html:cached

Contents of your .env
PROJECT_NAME=testsite1
PROJECT_BASE_URL=mydomain.tld
DB_NAME=xxxxx
DB_USER=xxxxx
DB_PASSWORD=xxxxx
DB_ROOT_PASSWORD=xxxxx
DB_HOST=mariadb-drup
DB_PORT=3306
DB_DRIVER=mysql
MARIADB_TAG=10.5-3.12.2
DRUPAL_TAG=9-4.28.0
PHP_TAG=8.0-dev-4.24.4
NGINX_TAG=1.19-5.13.5
NGINX_VHOST_PRESET=drupal9
REDIS_TAG=6-3.7.2
Logs output docker-compose logs
Attaching to testsite1_php
php_1           | [06-Apr-2021 09:11:51] NOTICE: fpm is running, pid 1
php_1           | [06-Apr-2021 09:11:51] NOTICE: ready to handle connections
php_1           | <IP_ADDRESS> -  06/Apr/2021:09:16:44 +0000 "GET /index.php" 401

Ah, I didn't know that that commented out lines were printed in huge bold fonts, sorry about that.


Ready check not triggering General Ready Check

Description
Within the last release or two, ready checks no longer trigger the built in trigger.
Additional information
Probably from this commit, timeframe lines up.
https://github.com/quisquous/cactbot/commit/03af1ac061511e796d32e41819f819989713a1e5
https://github.com/quisquous/cactbot/issues/4294

Hmm, seems like ready check can also be on code 0139, looking at log files from today. I'm not sure what the differentiator is, I had two ready checks back to back from someone else in the party (same person both times) go to 0139 and 0239. Running a quick grep shows:
$ grep -P '(?: has initiated|You have commenced) a ready check' /.../FFXIVLogs/*.log | cut -d'|' -f3 | sort | uniq -c
     23 0039
     68 0139
     46 0239

Hmm, seems like ready check can also be on code 0139, looking at log files from today. I'm not sure what the differentiator is, I had two ready checks back to back from someone else in the party (same person both times) go to 0139 and 0239. Running a quick grep shows:
$ grep -P '(?: has initiated|You have commenced) a ready check' /.../FFXIVLogs/*.log | cut -d'|' -f3 | sort | uniq -c
     23 0039
     68 0139
     46 0239

Yea I just checked, all the ones that didn't trigger for me are 0139.

My logs show the same,  I'm not sure what 0139 is, but I don't see any codes but those three.  Thanks for the investigation!


Group entries with the same phenotype in 1 row

Could be merge with the branch release/86 as well

Resubmitted to the branch postreleasefix/86


Tab characters in TMemo render as a square

While FMX does not directly allow you to type a tab char (#9), if you try to display text using Skia, in a Tmemo, the tab characters are rendered using the missing glyph (square) shape.

Additional info: https://issues.skia.org/issues/40043415

Fixed on new release v6.0.0-beta5


Simplify usage of typed-checked nodes

Why?
It is not straightforward to build logic after type-checking the Val program. The user needs to keep around a TypedProgram object and use it in parallel with walking over the AST. This creates a lot of boliderplate code
We want to remove the use of boilerplate code, and make the code that reasons on typed-checked node as easy as possible to reason about.
How?

add type-checked nodes that reflect all the properties of the original nodes, but keeps the type information also around.
allow clients to use TypedXXX to represent type-checked nodes

Testing

all existing unit tests pass

Dropping in the favour of #131


Fix tap $this bug

[x] Added or updated tests
[ ] Documented user facing changes
[ ] Updated CHANGELOG.md

Changes
Updates the HigherOrderTapProxyExtension such that statements using $this  are now properly understood as well. See https://github.com/nunomaduro/larastan/pull/575#issuecomment-642527659

Thank you!


Multiple operators in the same precedence group is still a bit glitchy

For example, 1 + 2 + 3 gives 5 and 2 + 2 + 2 gives 4.

Fixed by 7517d19


Status of this extension?

@dcdh what is the status of this extension?
I don't see it has any specific Quarkus code in it to make Shedlock work?  This looks like its just the default extension creation.

I have same question !

PR https://github.com/quarkiverse/quarkus-shedlock/pull/3 looks like it has some code.


stuff

This disables a thing that puts the form into perpetual react updates, and correctly returns true when the participants form is valid.

edit: it now fixes an issue with creating a new grant.

I tested this locally, and on my first attempt, I could not get grant creation to work. In particular the glitchy text input (which I think is related to the infinite loop that you all were referring to yesterday) is still occurring for me-- I'll post some screen capture videos in slack.
Here are some other general notes:
I don't think the admin / curator user should be allowed to enter text for the 'date of grant' -- not sure when this changed but it needs to be fixed. (Way too many things that could go wrong with a general admin user entering a datetime here)
typo: "Longline" should be "Logline" (if necessary, check Nene's comment on this google doc)
Love the form validation errors being displayed, thank you 👍 (I haven't checked that this is working for all relevant fields, but I saw it working for at least one)
So, I don't think this should be merged yet. Please let me know if I'm missing something. I'll try creating a grant again, typing more slowly

https://user-images.githubusercontent.com/102297373/210791663-cfc62f8a-dbaa-4ec2-a095-4470283d04cd.mov
https://user-images.githubusercontent.com/102297373/210791702-103c8529-fb77-4ac4-aa98-8e72c09e8590.mov

The text capture bug still exists in the current codebase, and is lessened with this PR, so it’s not a regression.
If “longline” is supposed to be “logline” then that’s a typo in Hasura. It should be fixed as a separate branch.
“admin curator shouldn’t enter text for date of grant” I thought I explained that, apologies if I overlooked it: it is supposed to be a pre-fill read-only field, but the text capture bug was leading that to an infinite loop.

Since this does enable the creation of grants, and that’s holding up a lot of important work, and none of this PR is a regression, I would like it to be merged so we can move forward with testing and fixing other things (yes, including deeper investigation of the text capture glitch)


Run SonarCloud only on pull requests to master

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: chintandoshi24 sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.


README update

The readme should be updated with the new instructions to run the server locally

done as of commit ef7a0c0

thanks, looks good!


feat: infimum of filter coproduct and filter product

We prove that if f₁, f₂ are filters on a type α and g₁, g₂ are filters on a type β, then f₁.coprod g₁ ⊓ f₂ ×ˢ g₂ ≤ f₁ ×ˢ g₂ ⊔ f₂ ×ˢ g₁.

bors r+


ActivityLog does not log sync activities

Kindly help me out. I have a project am working on but I discovered from my repository it does not log any SNYC activities I made call to. Can someone assist pls. I am using spatie/laravel-activity package using the model log

Hi @bankork – sorry for the late reply.
As you said; you can use withProperties() to save some extra data alongside the activity. I've looked into it but I don't think there's an easier way to log changes made when you sync related models.
Good luck and if you have any more questions feel free to open a new issue.

can you make it easier?
it will be perfect when it logs automatically

@hamza-masoud relation logging is discussed in "endless" issues and like always: there's no way for us to log them automatically as there's no event we could listen for.


Siglent: Retrigger after waveforms have been captured

So this code works fine with USBTMC, but hangs when using Lxi. It seems to be because there are 2 different threads issuing lxi requests:

the AcquireData thread does a retrigger
the gui thread calls GetVoltageRange

How is this done with LeCroy and the lan driver?
Is the issue that the Lxi library isn't reentrant?
If so, what's the solution? Do we need a new Transport method that combines SendCmd and Reply and make that atomic?

By regrabbing the m_mutex before retriggering, concurrent Lxi library requests are avoided again, and things work. I think this is the right solution.
But it raises the question: isn't there a m_mutex grab missing in LeCroy code?
https://github.com/azonenberg/scopehal/blob/f790a73e598dd15c3321f790bf3cf115e1625d3f/scopehal/LeCroyOscilloscope.cpp#L1461

Good catch re the mutex grab in the LeCroy driver. Yes, that's a bug.


Feature request: Invisible marker with label only

Hi @tombatossals , it's me again, the static map enthusiast.
I'd like to request for the above feature as it can be an useful feature, say I want to show the nearby point of interests and using the current markers, it gets a little too clustered. Will be great if I can set the markers to invisible and just showing the label.
Thanks in advance.

This is already possible, simply use a "void" style on your marker:
var marker = {
    lat: xxx,
    lon: yyy,
    style: {}, // No style to display nothing except popup
    label: {
        message: 'hello !',
        show: true
    }
  };

worked for me...

@style-x7 Does @claustres solution match your needs? I'm closing this issue. In case feel free to comment and re-open it again. Thx


Changed all the html into go using go-elem

Created templates package in ./hscontrol/templates. Moved the registerWebAPITemplate into the templates package as a function to be called.
Replaced the apple and windows html files with go-elem.

[X] have read the CONTRIBUTING.md file
[ ] raised a GitHub issue or discussed it on the projects chat beforehand
[ ] added unit tests
[ ] added integration tests
[ ] updated documentation if needed
[ ] updated CHANGELOG.md

@nblock if you have some time, can you do a quick skim of the content for windows and apple?

@amha-mersha can you delete the HTML templates so we are sure noone will update them in error?

@nblock if you have some time, can you do a quick skim of the content for windows and apple?

Sure, but I got this while building this commit:
error: hash mismatch in fixed-output derivation '/nix/store/xj7li14vnd7v6rnmlj5k7v2rqgm7hxw7-headscale-1e8fc3e-go-modules.drv':
         specified: sha256-+8dOxPG/Q+wuHgRwwWqdphHOuop0W9dVyClyQuh7aRc=
            got:    sha256-U5u4tRcCQ+aRIFzOOm0KWRX4BitWKwiHtq5m2DadNtA=
error: 1 dependencies of derivation '/nix/store/f2ds1pcql2haf3s2z3m68mfikcga7lix-headscale-1e8fc3e.drv' failed to build

@nblock I think it is now fixed

@nblock I think it is now fixed

That was quick, thx!
Found one additional location where the space is lost:

how about now @nblock . sorry to take your time for this things

Linter and TestAuthKeyLogoutAndRelogin is expected, but can you have a look at updating the build flake sha?

A rebase should also fix the failing integration tests.

So should I first rebase and commit

yep that sounds best

I have rebased and pushed again

Think the nix flake needs to be update again, haha, we are very close!

do I update that and how

In the failing build there is a new hash that needs to replace the old one one flake.nix, so a search replace should be good.
Sent from Proton Mail for iOS
On Thu, Oct 3, 2024 at 13:44, Amha Mersha @.> wrote:   do I update that and how —
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because your review was requested.Message ID: @.>

I fixed it up and will merge it when the test runs :) Thanks @amha-mersha, if you feel like more challenges there is one linked in the issue for OIDC page that is trickier because it has more CSS.

Thank you @kradalby for taking the time and guiding me. And for the other one, sure I look into it and get back to you

Could you share me the issue, I couldn't find it.

https://github.com/juanfont/headscale/issues/2115 @amha-mersha


[crash] Something went wrong

Describe the bug
Not able to create the project.
Steps to reproduce
I have established a directory containing a collection of .yaml files and I am attempting to load them as a folder, but I am encountering difficulty in loading the files.
Stacktrace
Cannot read properties of undefined (reading 'filePath')
TypeError: Cannot read properties of undefined (reading 'filePath')
   at file:///C:/Program%20Files/Kubeshop/Monokle/resources/app.asar/build/bundle.main.js:108586:195
   at a (file:///C:/Program%20Files/Kubeshop/Monokle/resources/app.asar/build/bundle.main.js:99126:462)
   at file:///C:/Program%20Files/Kubeshop/Monokle/resources/app.asar/build/bundle.main.js:99126:681
   at Object.useSyncExternalStore (file:///C:/Program%20Files/Kubeshop/Monokle/resources/app.asar/build/bundle.main.js:93158:72013)
   at exports.useSyncExternalStore (file:///C:/Program%20Files/Kubeshop/Monokle/resources/app.asar/build/bundle.main.js:94333:6580)
   at exports.useSyncExternalStoreWithSelector (file:///C:/Program%20Files/Kubeshop/Monokle/resources/app.asar/build/bundle.main.js:99126:753)
   at useSelector (file:///C:/Program%20Files/Kubeshop/Monokle/resources/app.asar/build/bundle.main.js:93821:342)
   at SaveEditCommandModal (file:///C:/Program%20Files/Kubeshop/Monokle/resources/app.asar/build/bundle.main.js:108586:134)
   at Xh (file:///C:/Program%20Files/Kubeshop/Monokle/resources/app.asar/build/bundle.main.js:93158:64935)
   at dj (file:///C:/Program%20Files/Kubeshop/Monokle/resources/app.asar/build/bundle.main.js:93158:76756)
   at Wk (file:///C:/Program%20Files/Kubeshop/Monokle/resources/app.asar/build/bundle.main.js:93158:118731)
   at Vk (file:///C:/Program%20Files/Kubeshop/Monokle/resources/app.asar/build/bundle.main.js:93158:113844)
   at Uk (file:///C:/Program%20Files/Kubeshop/Monokle/resources/app.asar/build/bundle.main.js:93158:113773)
   at Jk (file:///C:/Program%20Files/Kubeshop/Monokle/resources/app.asar/build/bundle.main.js:93158:113630)
   at Fk (file:///C:/Program%20Files/Kubeshop/Monokle/resources/app.asar/build/bundle.main.js:93158:110727)
   at jg (file:///C:/Program%20Files/Kubeshop/Monokle/resources/app.asar/build/bundle.main.js:93158:48408)
   at file:///C:/Program%20Files/Kubeshop/Monokle/resources/app.asar/build/bundle.main.js:93158:108281
   at node:internal/process/task_queues:140:7
   at AsyncResource.runInAsyncScope (node:async_hooks:204:9)
   at AsyncResource.runMicrotask (node:internal/process/task_queues:137:8)

I managed to initiate a project and commence editing the files. However, an obstacle arose when the correct path for .monokle was absent. By relocating all the necessary files to the directory where .monokle resided, the project began displaying the files.

I also noticed that the desktop version keeps freezing and doesn't recover, so I had to end the Monokle desktop forcible.

thanks for reporting this @Exnadella - which Monokle version are you using?

Hi @olensmar I am using monokle desktop and version it is displaying is 2.1.

thanks @Exnadella - hm... how large are these files in total? is it possible for you to share the files you are loading (over a private channel of course) for debugging purposes?

Hi @olensmar There are 3 folders and 28 files, size on the disk is 8 KB.  with the .git folder we have 53 files and 17 folders.  We can have a call today to take a look.


How to deploy ?

I tried to wrap this project as a docker image, in fact it was able to run, but since I was using my own domain name; I had a problem with Github OAuth2 authorization:
The page always prompts me to log in to Github (OAuth2 authorization). When I click the button, I am prompted to authorize the success, but the page does not reflect anything; I observed that the authorization callback address does not seem to be my own domain name.

I have created a dockerfile and now it looks fine (if the Github public api request limit is not reached);
refs:  #67

It's doing oauth through Netlify, I don't belive that part can be selfhosted.
If you want to make it work you need to create a

create a GitHub oauth app
create a site on Netlify
add the github app client id and client secret to the netlify site
set the site id form netlify in src/github.js

Netlify access is slow in China due to firewall reasons.
Is there a solution that can no longer rely on Netlify?
For example, running git-history on vps.


Call a Java method from Rust code

Is there a fully contained example to call a Java method from Rust code?
Some thing like an example found on https://github.com/rustyhorde/rsjni

There is an integration test that calls various types of Java methods: https://github.com/prevoty/jni-rs/blob/master/tests/java_integers.rs
It shows how to start a JVM from Rust, how to attach the current native thread, and call a Java method. If Java calls Rust, the native thread is already attached and you can just do the latter.

I've written a sample to do this.  fasihrana/rustjni_example.
Hopefully this will help someone looking for a clear cut way.
My suggestion would be to add the pub fn jvm() -> &'static Arc<JavaVM> found in the util/mod.rs file, to the actual API, taking in the arguments to start the jvm.

Closing this issue since I have a working example.


⚠ Bofrost.es has degraded performance

In f844ba8, Bofrost.es (https://www.bofrost.es) experienced degraded performance:

HTTP code: 200
Response time: 1446 ms

Resolved: Bofrost.es performance has improved in dcc106b after 28 minutes.


(Feature) #1442 User should be able to export his wallet to other wallets

Description
Create a layout for the new ExportWallet screen, apply styles for elements according to the design. Implemented all buttons functionality. Change the icons on the side menu.
Comment old BackupWallet Screen.
About #1442
How Has This Been Tested?
Open the side menu and see the new 'export wallet' item. Click on it and you should be redirected to the new screen.
Checklist:

[x] PR title matches follow: (Feature|Bug|Chore) Task Name
[x] My code follows the style guidelines of this project
[x] I have followed all the instructions described in the initial task (check Definitions of Done)
[x] I have performed a self-review of my own code
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[x] I have added reference to a related issue in the repository
[x] I have added a detailed description of the changes proposed in the pull request. I am as descriptive as possible, assisting reviewers as much as possible.
[x] I have added screenshots related to my pull request (for frontend tasks)
[ ] I have pasted a gif showing the feature.
[x] @mentions of the person or team responsible for reviewing proposed changes

Screenshots:

Browser
Simulators

@yaroslav-fedyshyn-nordwhale
I've updated the Fuse logo as a vector, please replace it.
In order to review the final screen, please upload a print screen.
Thanks

@yaroslav-fedyshyn-nordwhale
I've updated the Fuse logo as a vector, please replace it.
https://xd.adobe.com/view/b6910538-bb24-4b05-6eba-f8ceea3cf290-c0a0/
In order to review the final screen, please upload a print screen.
Thanks

Done. See screenshots attached to the PR description.

@yaroslav-fedyshyn-nordwhale
Please go over all the things that need to be fixed:

text is overlapping the white circle around the avatar/fuse logo.
"copy address" should be in a single line.
"copy" = "copy key"
double-check the settings menu icons relative positions and sizes.

New screenshots after design fixes:

@yaroslav-fedyshyn-nordwhale
The export screen looks good.
But the settings icons are still not aligned pixel perfect.

Fixed side menu icon positioning and sizes:

@LiavGut waiting for you


Next auth client redirect back to spa not api

Currently the transforms required for nextAuth have been implemented but when setting the NEXTAUTH_URL to another url ex. localhost:3000 the provider still tries to load the season from Vite's dev port 5173.
this can be fixed by setting the basePath parameter in the SessionProvider component to the full URL of the api http://localhost:3000 but when the user clicks the login button and completes the auth flow the user will be redirected to http://localhost:3000 instead of http://localhost:5173 the redirect query parameters are being set properly when talking to the api but once the flow is completed its not respected.. this could be due to the only part changing is the port and not the hostname

Fixed by using vite's dev server proxy https://vitejs.dev/config/server-options.html#server-proxy
https://github.com/arbassett/create-t3-app-extra/blob/7104e433782f4ee06d6bb46ca0ad9816f5c9c245/packages/cli/src/installers/vite:client/trpc.ts#L279


Remove analyze properties when removing connector

From https://github.com/prestodb/presto/pull/12575

@davidlao2k i migrated your PR from prestodb.
Apparently this is the second time you're fixing a bug like this. Thanks for this!
A question -- how do you find these?

i have a feature in my fork that support loading/unloading catalogs at
runtime. perhaps I should contribute a test next time.
On Tue, Apr 2, 2019 at 12:59 PM Piotr Findeisen<EMAIL_ADDRESS>wrote:

@davidlao2k https://github.com/davidlao2k i migrated your PR from
prestodb.
Apparently this is the second time you're fixing a bug like this. Thanks
for this!
A question -- how do you find these?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/prestosql/presto/pull/575#issuecomment-479174091, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADGWMZ8skcLyPIkje0PaP5qpTkhHnCerks5vc7a3gaJpZM4cYyyt
.


[Backport 2.x] Abstracting outbound side of transport (#13293)

Abstracting outbound side of transport

Making outbound handler protocol dependent via inbound handler

Fixing precommit

Addressing comments

Fixing precommit

Cleaning up code

Addressing comments

Cleaning up

Addressing comments

Abstracting InboundHandlerTests

Abstracting TransportLoggerTests

Description
Backport #13293 to 2.x
Related Issues
Resolves #[Issue number to be closed when this PR is merged]

Check List

[x] New functionality includes testing.

[x] All tests pass

[x] New functionality has been documented.

[x] New functionality has javadoc added

[x] API changes companion pull request created.
[x] Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
[x] Commits are signed per the DCO using --signoff
[x] Commit changes are listed out in CHANGELOG.md file (See: Changelog)
[x] Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

❌ Gradle check result for 6b356e5: FAILURE

https://github.com/opensearch-project/OpenSearch/issues/12308
https://github.com/opensearch-project/OpenSearch/issues/13473


Determine a data organization strategy for S3

In current hubverse repos, the model-output data is organized like:
model-output/
├─ team1/
│  ├─ yyyy-mm-dd-team1.csv
│  ├─ yyyy-mm-dd-team1.csv
├─ team2/
│  ├─ yyyy-mm-dd-team2.csv
│  ├─ yyyy-mm-dd-team2.csv

The cloud work done to date mirrors this structure in AWS S3 buckets.
Our current hypothesis is that this file organization isn't aligned with how most people what to query the data. For example, when hubUtils connects to an S3 bucket and uses Arrow to present information, the performance of that operation depends on how well the query maps to how the data is physically stored.
Thus, the decision for how we want to organize the data once it's in the cloud is important.

Depending on where we land with this, we might need a related task for ensuring that hubUtils will work with the new organization.

Capturing some notes from Luke Mullany about common use cases:

I think it is in fact hard to say what the most common use cases might be.. However, I am thinking perhaps the submission date is first, and then the targets, and then the modeling team last.
I think I would suspect getting an entire set of data for a particular submission date (i.e. all models and all targets) would be priority for many analyses. .. The next filter would probable by on the specific type of target..
I guess I suggest the modeling team last, because although modeling team-specific analyses are certainly important, in the context of "hub",  the whole purpose is to leverage/compare/contrast multiple models and their forecasts
Note all the above depends critically on all model submissions within a given "round" having identical "submission_dates".. If not, replace the above with some logic that replaces "submission_date" with something that does identify the "round"..

for many repos, partitioning on more than round may be overkill, and lead to a huge number of small files.
for example, on my local machine, I just now cloned the flusight 1 repo, read in all 4,026 submissions (i.e. across numerous flu seasons and models), combined into a single data.table, and creating a hive-partitioned dataset using arrow.. (with submission date and target)
now this would create >3000 files, many of which are quite small where total amount of metadata is starting to dominate relative to actual data of interest!. Granted, this is true partly because the submission dates in flusight are not sufficiently specific to round (see my other comment).
if i recreate the partition only on submission_date or only on target, obviously the number of files and the total size of the combined partitioned dataset will be much smaller.

One level of partition indeed sounds about right. And definitely much easier to implement in a generalisable fashion than trying to add further partitions. Would be interesting to benchmark different queries on the two hubs (one standard partitioned and the one you repartitioned on submission date).

Okay, prefacing with the fact that the flusight repo is very small - only 29,600,000 rows, here are some initial benchmarks on the following task:

create handle to the partitioned dataset --> open_dataset()
filter on a particular model --> filter(model == "Delphi-Stat")
group by either submission date or target --> group_by(submission_date) or group_by(target)
summarize the mean of the value [ I realize this is meaningless; just for illustration purposes] --> summarize(meanvalue = mean(value, na.rm=T))
collect the result set --> collect()

We compare three partitioning options

partitioned on target (labelled by targ in benchmark table)
partitioned on submission date (labelled by sdate in benchmark table)
partitioned on target and submission date (labelled by both in benchmark table)

When grouping variable is target:

cmds1 = alist(
  targ = open_dataset("flusight1_targ/") %>%
    filter(model=="Delphi-Stat") %>%
    group_by(target) %>%
    summarize(meanvalue = mean(value, na.rm=T)) %>%
    collect(),
  sdate = open_dataset("flusight1_date/") %>%
    filter(model=="Delphi-Stat") %>%
    group_by(target) %>%
    summarize(meanvalue = mean(value, na.rm=T)) %>%
    collect(),
  both = open_dataset("flusight1_date_targ/") %>%
    filter(model=="Delphi-Stat") %>%
    group_by(target) %>%
    summarize(meanvalue = mean(value, na.rm=T)) %>%
    collect()
)

Unit: milliseconds
  expr       min        lq      mean    median        uq       max neval
  targ   64.7840   82.7043  198.1586   91.2229  100.0729  674.7837    10
 sdate  133.0033  136.7588  139.9850  140.2119  142.3588  147.1574    10
  both 1155.5496 1199.8241 1261.3381 1216.4838 1283.8325 1569.8637    10

When grouping variable is submission_date:

cmds2 = alist(
  targ = open_dataset("flusight1_targ/") %>%
    filter(model=="Delphi-Stat") %>%
    group_by(submission_date) %>%
    summarize(meanvalue = mean(value, na.rm=T)) %>%
    collect(),
  sdate = open_dataset("flusight1_date/") %>%
    filter(model=="Delphi-Stat") %>%
    group_by(submission_date) %>%
    summarize(meanvalue = mean(value, na.rm=T)) %>%
    collect(),
  both = open_dataset("flusight1_date_targ/") %>%
    filter(model=="Delphi-Stat") %>%
    group_by(submission_date) %>%
    summarize(meanvalue = mean(value, na.rm=T)) %>%
    collect()
)

Unit: milliseconds
  expr       min        lq      mean    median        uq       max neval
  targ   72.2872   83.7387  222.6143   88.8084  103.2500 1037.7222    10
 sdate  126.4035  135.8261  178.6519  139.8237  147.7848  535.2068    10
  both 1191.7232 1204.7832 1233.7476 1237.5590 1251.3735 1290.6913    10

So, in this admittedly very limited example, its clear that the doubly partitioned dataset (partitioned on both target and submission date) causes some slow down, and that the smaller of the two single partitions (i.e. target) is faster, even when the grouping variable is mismatched (i.e. when grouping by submission_date, the smaller partition was still faster)

Nice!
Would be interesting to also try a query that collects forecasts for say the last n rounds for a given output type.

yeah, that might be common; probably a good implementation is via identifying the last n round indicator, and then joining.. something like this:
get_last_n_rounds <- function(dataset,target_type, n=5) {
  inner_join(
    open_dataset(dataset) %>% filter(target == target_type),
    open_dataset(dataset) %>% distinct(submission_date) %>% arrange(desc(submission_date)) %>% slice_head(n=n),
    by="submission_date"
  )
}
cmds4=alist(
  targ = get_last_n_rounds("flusight1_targ/", target="Season onset") %>% collect(),
  sdate = get_last_n_rounds("flusight1_date/", target="Season_onset") %>% collect(),
  both = get_last_n_rounds("flusight1_date_targ/", target="season_onset") %>% collect()
)
microbenchmark(list=cmds4, times=10)

Unit: milliseconds
  expr       min        lq      mean    median        uq       max neval
  targ  469.5469  491.2216  528.6626  498.8077  536.0588  746.3325    10
 sdate  490.5871  501.6316  543.4063  515.6881  554.7503  728.3711    10
  both 1859.2467 1893.0724 2014.3022 1980.3282 2027.8139 2365.7026    10

but you all may have different ideas.. arrow as far as I know does not support a number of alternative ways you might consider doing this (i.e. via n(), row_number(), slice_head(n=, by=..), etc, etc.

Actually, I think that partitioning by "round" (above "submission date"),  might be the best choice, but only for retrospective storage
If the storage is being accumulated over time, then I think it make sense to have a separate (possibly partitioned) dataset for each round.  That is, the data are accumulated for a round, and that round contstitutes a dataset, that could be partitioned by target. Then the next round is accumulated sometime later, and that round constitutes a dataset, which again could be partitioned by target, etc, etc.

One thought on this is that for the scenario hub examples, there are few rounds and each one has a lot of data. For a typical the forecast hub (at least so far) there are a lot of rounds, and relative to the entire project, each round would have relatively little data.
I agree that it probably makes sense to standardize storage and partitioning structure for all hubs in the cloud.
I don't have a clear sense of the trade-offs for having multiple datasets for a hub (one per round) vs. trying to partition on round. @annakrystalli was saying something about how she thought that "appending" a round to an existing dataset could be an expensive procedure computationally. (The entire partition would have to be rebuilt?)
From the end user perspective, I think we pretty definitely want to keep the ability to issue one "query" or "collect statement" to gather data from multiple rounds at once.

Yes @nickreich  - its that issue of appending (i.e. one can't really append, but rather needs to load-append-re-partition) that led me to the comments above re separate datasets per round..
re the tradeoffs, would be good to do some testing with an actual S3 bucket, and actual large size repo.. If there were substantial trade offs (i.e. with a single round-partitioned dataset being optimal), then one might rethink our consensus that one single approach is best. That is, we might consider a different more optimzed storage structure for "archived/ no-longer-active" hubs vs "currently active" hubs

I'm not super familiar with the ins and outs of arrow-style datasets, but if we did have separate datasets per round, would that kind of set-up be compatible with a single "collect-style" argument to load data from multiple rounds? If not, what would the right strategy be to support single-argument-style queries on the data? E.g. would we need build a user-facing function that would wrap a few collect calls for different rounds?

actually, i've been thinking about this a bit more, and perhaps there is a way that we can "append" to a single dataset, without reloading the prior data, appending the new, and rewriting the updated datastet.

Let's say we have an existing single dataset containing the entire hub's data collected thus far partitioned by round (here round is a string date, but it could be anything (i.e. "R1", "R2" etc, or "1", "2", "3", etc..) [note, this is just some data from Scenario Modeling Hub (not the FluSight repo  mentioned throughout the benchmarking in this thread]
It might look something like this, with say 10 completed rounds (in this case, labelled by date string from 2021-05-02 to 2022-07-19

if we run
open_dataset("active_hub") %>% glimpse()

, we set that there are 9 million rows across 10 parquet files:

Now, lets say we have some new data (new_data) from the next round, lets say "2022-10-19", and we want to add/append this to the same arrow multi-file dataset.
We can simply do:
# write the new data to the same dataset
write_dataset(new_data, "active_hub", partitioning = "round")

and now out dataset looks like this:

and  similarly if we again run
open_dataset("active_hub") %>% glimpse()

we get

The single dataset has been properly appended, without re-reading or re-partitioning the existing data

sure, but within a round, we only have to row bind all the submissions from a single round, and then append that single new round to an arrow dataset..   I guess what I'm saying is that the process seems trivial regardless of whether it is an active or closed/complete hub

It seems like this may have already been settled -- but responding to Anna's comment above alluding to a possibility of partitioning by target, wanted to note that although every hub has a concept of a "target", we don't impose any requirement that model outputs record a field called target (e.g., if the hub only collects forecasts for one target, dropping that into the model output files would be superfluous). So if we're aiming to build a "one solution for all hubs" kind of answer here, we likely don't want to hard code target as a part of that.

Thanks @elray1 !
Ideally nothing is hardcoded and any re-partitioning is configurable by users. I was just referring to the examples we had been discussing above, with target being one example given.

@lmullany , myself and @bsweger actually have a call set up for tomorrow! I was going to suggest you should join us anyways!

Linking to notes from the chat the @lmullany and @annakrystalli and I had this morning (Anna and Luke, please correect anything I got wrong): https://docs.google.com/document/d/1NgEZ1i_mXukoOMrgPWFKWm8xZKdFs6G1sWxeOIfvmT4/
Our small working group is coming at this problem from different angles, which is very useful (imo)
A high-level question that emerged is: what are hoping to gain by re-organizing/munging model-output data in the cloud? What would make the additional dev and maintenance of moving parts worthwhile?

given this discussion this morning, I agree that simply mirroring the structure, but writing to parquet, might be pretty high on the list on candidate approaches.
if we are going to index a  db, or partition for some optimization, round-driven queries (both of @elray1 examples) I think are very common (whether that's for an analysis/viz of a single round, or a set of recent rounds, etc).

Just adding the link to the initial benchmarking here too: https://partition-benchmarking.netlify.app/

To add on to @elray1 's comments above about common queries. I think a common secondary analysis of forecast data involves pulling down all forecasts of one target, for all models, for perhaps all or a subset of rounds. E.g. for an analysis of accuracy of forecasts of deaths, I just want "k step ahead incident hospitalizations" (for all possible values of k) from the past 20 weeks. Because this is a secondary analysis and not some interactive web app, "instantaneous" speed is not critical.

Another comment here is that another partition that we could consider would be by output_type. One advantage here is that it might move us more towards a zoltar-style "one-table-per-output-type" kind of model which could have advantages later and help us get around some of the downstream data-type challenges that we have when output_type_id can take both character and string values (e.g., here).

Just to note that unfortunately I don't think splitting across output types would solve the data type issue if accessing the data as an arrow dataset as the dataset as a whole uses a single schema across all files. If there are data type differences across files on the same column you need to explicitly define a single data type to be used for those columns or opening the dataset fails.

Having said that, if you know you are only interested in a single output type, perhaps you could just connect to the output type sub directory (rather then the whole model output dir) and set the required data type for the output type Id column.

Closing this one because we've verified that writing the model-output files as parquet (and added columns for round_id, team, and model) works with the existing hubData tools.
At this point, we need more real-world usage patterns before deciding to re-organize the data. We had talked about potentially using Zoltar query history as a data point (and thanks @matthewcornell for volunteering to help with that!), but let's wait until we get some hubverse-centric feedback about this.


Update OCP build configuration template

The OCP build configuration template provided in the project contains an override for the base image to be used for the build.  The default override is the older CentOS based image.  The associated Dockerfile has since been updated to use a Fedora base image, and the two images use different package managers which causes issues when a none Fedora based image is used.
https://github.com/BCDevOps/backup-container/blob/621bb53f9d1d2a577a3d923df37a207c16310d57/openshift/templates/backup/backup-build.yaml#L70-L74
https://github.com/BCDevOps/backup-container/blob/621bb53f9d1d2a577a3d923df37a207c16310d57/docker/Dockerfile#L2
The line that causes the build to fail is here:
https://github.com/BCDevOps/backup-container/blob/621bb53f9d1d2a577a3d923df37a207c16310d57/docker/Dockerfile#L37
ToDo:

Update the OCP build configuration template to use the Fedora based image as the default base image for the build.

cc @jleach, @StevenBarre, @hannah-macdonald1


Improve bulk assignment flow

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

Bulk assignment has many issues currently:

you can click "save and continue" when you haven't selected any assignments
you can't filter by "studio", which will be required as those are the bootcamp participants
the ui is not clear as to how to add assignments

q for both troy and michael - if you create an assignment and don't mark it as common, there is no way to retrieve it in a future assignment. Would you prefer that all assignments created in the bulk assign screen are automatically created as common? Or the check box for "common assignment" already checked?

potential change - participant dashboard should have a filter and not just sorting

potential new view - studio bootcamp dashboard
q for michael - do you want them to submit answers? right now they can only mark completed

moodle as an alternative for assignments? in bootcamps


Peer deps

:+1: You'll probably want to also specify these as devDependencies too so the tests will pass :)

I checked it with<EMAIL_ADDRESS>and looks like it still installs peerDeps. I also saw your commits in the fork so I assume you had some problems with it?

hmmm… no, I didn't actually try it. I just have the habit of putting them in devDeps since they won't be installed in the future. Interesting that it's not a problem.

Sounds logical :) Let me update it

Thanks :)

Thank you!


It this plugin running on one thread?

When I use Logstash without the filter my Logstash agent takes up around 270% CPU, which is normal (it has 4 cores).
When I enable the throttle plugin it takes up 100% CPU and is not able anymore to process the load.
It seems that the throttle plugin caps the CPU usage in some way, perhaps working singlethreaded. I would like the plugin to take full advantage of all my available cores.
Red Hat Enterprise Linux Server release 6.7 (Santiago)
logstash 2.2.0

Your observations are correct, see https://github.com/logstash-plugins/logstash-filter-throttle/pull/1 and related issue https://github.com/elastic/logstash/pull/1469 for the conversation.
Currently there is nobody planning on fixing this, so if you want to...

logstash-filter-throttle is multithreaded since v4.0.0 (pull request: https://github.com/logstash-plugins/logstash-filter-throttle/pull/5). Issue can be closed.


searchbar .searchbar-not-empty needed to show clear button

we have clear button in html, but no required class searchbar-not-empty for this button to be visible. Need some how to detect content of input in vue component?

Close. Found that we should implement search somehow. In case of custom search, add customSearch = true to component. Then it behaves as expected.


Optimize performance of getVirtualItems

getVirtualState is getting the height of the container every onScroll (in addition to a lot of other repeated calculations) when the values likely haven't changed. I believe this is causing elements to be reflowed by the browser, and even if not, is doing the same work over and over.
I will try to optimize the method after #14 which is required to know if things need to be recalculated.

Also possible related: #1, #3

Great catch; this should be able to be cached, as long as we recalculate it on componentWillReceiveProps if the container has changed.


docs: update Alpine package repo in README

Hi! I'm the maintainer of the Alpine Linux package for seaward. Seaward is now in Alpine's community repository (aports commit).
On an unrelated note, Alpine currently patches the Cargo.lock file to use version 0.2.155 of the libc crate instead of 0.2.147. When you next release a new version of seaward, it would be great if you could cargo update the libc crate first so we no longer have to patch it.
Thanks!

Sure thing, thank you!