fix when endpoint has no region, need a suggestion

Proposed changes
solve the problem
 mysql> CREATE CATALOG iceberg PROPERTIES (
    'type'='iceberg',
    'iceberg.catalog.type'='rest',
    'uri' = 'http://<IP_ADDRESS>:8888,
    "AWS_ACCESS_KEY" = "admin",
    "AWS_SECRET_KEY" = "password",
    "AWS_REGION" = "us-east-1",
    "AWS_ENDPOINT" = "http://minio:9000"
);
show databases;

ERROR 1105 (HY000): IllegalArgumentException, msg: java.lang.IllegalArgumentException: The value of property fs.s3a.endpoint.region must not be null

Problem summary
Describe your changes.
Checklist(Required)

[ ] Does it affect the original behavior
[ ] Has unit tests been added
[ ] Has document been added or modified
[ ] Does it need to update dependencies
[ ] Is this PR support rollback (If NO, please explain WHY)

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 35.09 seconds
stream load tsv:          422 seconds loaded<PHONE_NUMBER>9 Bytes, about 169 MB/s
stream load json:         24 seconds loaded<PHONE_NUMBER> Bytes, about 93 MB/s
stream load orc:          59 seconds loaded<PHONE_NUMBER> Bytes, about 17 MB/s
stream load parquet:          30 seconds loaded 861443392 Bytes, about 27 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230428085804_clickbench_pr_137054.html


module 'reciprocalspaceship.algorithms' has no attribute 'compute_intensity_from_structurefactor'

running rs version 0.10.1, rs.algorithms.compute_intensity_from_structurefactor() returns the error 'module 'reciprocalspaceship.algorithms' has no attribute 'compute_intensity_from_structurefactor'.

That method isn't available in the latest PyPI/conda-forge release -- I'll update it in the near future. To get access now, you can do a local install from an up-to-date clone of the repo: https://hekstra-lab.github.io/reciprocalspaceship/userguide/installation.html#installation

I'm closing this because the method is now available in the 0.10.2 release on PyPI (and shortly after on conda-forge). In general though, please keep in mind that a local install may be necessary if you want to access a feature that is not yet available in an official release


Update readme.md

Added Distributed Systems 3rd Edition by van Steen and Tanenbaum.

LGTM

Thx. Great work with the list.


add sample github actions integration to go, gradle and pytest

What

Example integration of Launchable to GitHub Actions in go-test, gradle and pytest.

@awilkes

Looks good to me, but don't we also need to add the steps from https://docs.launchableinc.com/sending-data-to-launchable/ensuring-record-tests-always-runs ?

if: always is needed when test step and record test step are separated. Since I wrote them in one step, we don't have to have if: always here.
I feel it is better to show variety of ways to install Launchable, so I made a separated version and if always.
https://github.com/launchableinc/cli/pull/483/commits/668126599728fea77b8c18efe18f81f8357ce3d2


[WARN]: Unable to access microphone. [Errno -9996] Invalid input device (no default output device)

Running the client on the localhost always hits this issues, Server Message says "Message from Server: TensorRT-LLM not supported on Server yet. Reverting to available backend: 'faster_whisper'"
But the server is working and running

I fixed that by installing voice control using
sudo apt-get install pavucontrol

I fixed that by installing voice control using sudo apt-get install pavucontrol

can not fix


When using CommonsChunkPlugin, the common bundle built contains hash list of all entry points

Hi,
First of all, thanks for the awesomeness that webpack is. I've been using it for few weeks only but I found that it provides all I expect from a bundler (and that I did not found anywhere else).
The current issue is the only thing that is currently preventing me from using it on production.
Here is the context:
I intend to use webpack for a big existing website to replace the current old-school bundle solution. This website is not a webapp and it is updated several times a day.
Our current bundle solution is built on a configuration file listing all bundles and the files they should embed. This allow us to have full control over what goes in a bundle and to maintain a common bundle for all pages. Then, each page is loaded with the common bundle and a page-specific bundle.
I tried to achieve the same with webpack and I used this configuration file:
var webpack = require('webpack'),
    path = require('path');

var configuration =
    {
        entry:
        {
            page1: './js/page1.js',
            page2: './js/page2.js',
            page3: './js/page3.js',
        },
        output:
        {
            path: 'build',
            publicPath: 'build/',
            filename: '[name].[chunkhash].js',
            chunkFilename: '[id].[chunkhash].js'
        },
        plugins: [
            new webpack.NewWatchingPlugin(),
            new webpack.optimize.CommonsChunkPlugin('common.[chunkhash].js'),
            function()
            {
                this.plugin("done", function(stats)
                {
                    require("fs").writeFileSync(
                        path.join(__dirname, 'webpack_files_map.conf.json'),
                        JSON.stringify(stats.toJson().assetsByChunkName)
                    );
                });
            }
        ],
        resolve:
        {
            root: path.join(__dirname, 'js')
        }
    };

module.exports = configuration;

This configuration file produce correctly one bundle per page and a common bundle.
The behavior that is bothering me is that the common bundle embed the hash for all entry points:
/******/ 	__webpack_require__.e = function requireEnsure(chunkId, callback) {
...
/******/ 			script.src = __webpack_require__.p + "" + chunkId + "." + {"0":"ec91e42ab36483e18cd9","1":"73705fff499a2fecdeeb","2":"5d017c0e038c150db29b"}[chunkId] + ".js";

This line of the common bundle make the common bundle implicitly dependent of ALL others bundles. So a modification of a file in page1.js that is not present in the common bundle will surely invalidate the common bundle because of the hash of page1.js that is written in the common bundle.
This may seems like a detail but the true purpose of a common bundle is to have a bundle that is long-cached and that do not change too often. But making this common bundle dependent of all others bundle kind of ruin this long-term strategy.
What I'd like is a way to specify this hashes externally (through a global var exposed in window) so that the common bundle is not dependent of others bundle addition / modification / deletion.
The code above could then become:
/******/ 	__webpack_require__.e = function requireEnsure(chunkId, callback) {
...
/******/ 			script.src = __webpack_require__.p + "" + chunkId + "." + WEBPACK_BUNDLE_HASHES + [chunkId] + ".js";

The WEBPACK_BUNDLE_HASHES content could be extracted from the JSON stats file that is produced at the end of the compilation phase (I am not sure if this is already possible).
Do you think that this makes sense and that it could be done?
Thanks.

Makes sense. This is how soundcloud does it.
You could do it this way:
        plugins: [
            new webpack.NewWatchingPlugin(),
            new webpack.optimize.CommonsChunkPlugin('common.[chunkhash].js'),
            new webpack.optimize.CommonsChunkPlugin('inline.js'),
            function()

and inline the content of inline.js into the HTML page as <script> tag.

For long time caching with [chunkhash] you should also use the records option, to make sure files don't change too often...

cc @diurnalist

@happypoulp yes, I had this same concern, and created the (perhaps badly-named) chunk-manifest plugin as a result. I should note that I apparently didn't write tests for that plugin yet - boo on me for that. But, maybe it will be helpful for you/others.
It will output all the chunk hashes to a JSON file that you can consume and use however you want. You just need to set it to some global variable (configurable) so webpack can read it at runtime.
@sokra your idea to use two CommonsChunkPlugin instances is interesting though. I'll have to see what that would look like, not sure if it's worth the extra bytes on the initial request or not. And, +1 to using the records option to keep module ids in sync across builds.

Thanks for the quick reply!
@sokra I tried your solution and even if it does extract hashes as desired, it came with the cost of a "big" inline.js because of the inclusion in it of the webpackBootstrap code. I'd like to be able to inline only hashes of bundles.
@diurnalist your solution is a perfect match for what I am looking for. The json produced is small and easily embeddable and the common plugin is stripped of other bundles hashes.
I also added the recordsPath options following your advices and I see a more consistent bundles update that I had before.
I'll go with @diurnalist solution and close the issue, thanks a lot!

@sokra if i don't to wrote the inline.js tag in html. Have some solution?

Write js inline at script tag? How?


[VRM - Bug]: no efficient battery usage

Contact Details
No response
VRM portal ID
b827eb6ff17f
Country / region
Germany (de)
B max
26
TB max
6,5
FB max
7
TG max
30
FG max
30
Battery costs
0,02
Buy price
(p+0.21)*1.19
Sell price
0,115
feed-in possible
Yes
feed-in control
Yes
What happened?
The battery was discharged and charged at the wrong time. First the electricity was too cheap to discharge from the battery and then the electricity was too expensive to charge from the grid :-/

sell into grid and battery is empty 11:31

When did it happen?
23/10/2023 04:00
Screenshots

I can confirm this from another point of view:
Weather is bad, battery is low, but there is some PV power available. Instead of charging all the power into the battery, Dynamic ESS is feeding some of it into the grid.
With a fixed sell-rate of 7.5cent per kWh this makes absolutely no sense.

VRM portal id: 48e7da859663
feed-in possible: Yes
feed-in control: Yes
Screenshots taken at 2023-10-27 09:30am GMT+2

Grid is very cheap and dynamic ESS like to use batteries ... WHY ?

I'm curios how victron further develops this function.
I think it's not everytime easy to understand DESS's decisions, as you also need to take into account how the PV and consumption forecast looks like.
I'm now trying to go for a different approach: In summer-time, I have enough PV to survive without any grid. In winter time, I know more or less precisely how much energy I need to drive our house and heating. So I do not need a forecast that get's confused when things like heatpump or EV starts consuming. I know, without EV, I need between 10 and 20kWh per day.
My battery has 30kWh. So its enough for at least 24hrs.
With that I can calculate, depending on the PV forecast, how much enery I need from the grid. And with an hourly grid price, I can calculkate when it makes sense to charge my battery to which SOC.
As you EVs are parking the whole night in the garage, they can wait for a low price and start charging at this time to a minimum that is required to survive the next 24hrs.
I think this will do it, and will be very transparent and understandable compared to DESS decisions :-) Will see how this works out for me. Just started to develop this solution.

I think this will do it, and will be very transparent and understandable compared to DESS decisions :-) Will see how this works out for me. Just started to develop this solution.

Hello Alex,
Good to know that you are looking for a transparent solution, because I have the same problem as you, finding an easy way to combine the solar forecast with knowing the amount of electricity required per day. Please let me know if I can help you provide an additional testing environment ;-)
Jens

No battery load today:

SOC is too low:

before I go to bed it was planned to load the battery between 2 - 4 o'clock and now noting :-/

Same here. I noticed it several times now, that charging was scheduled in the most expensive slot. And the battery was at 75% already. My second point is charging when sun is shining. In my opinion it should always be better (at least in germany) to charge with maximum kw. Energy out of battery is always cheaper than from grid. I had it several times, that battery was charged with 300w-500w, when 1700w was easy possible.

btw charging now!?

same here:

Why would I feed to grid if battery has a SOC of ~41% (30kWh total capacity)... Turned off DESS once again...
I doubt that people will use DESS if it's not explainable why the system behaves like this.

@Je68le
working on a simple solution. Current status:
If current price drops below daily average, I signal via "SnmartGrid" my heating system that price is low and that it's a good idea to heat or prepare warmwater NOW. Next step could be: if battery SOC is that low that it will not survive the next hours (or night or day ... not that clear for now): let the battery charge from grid if price is also below daily average. The tricky part is: how to calculate if it will not survive. I think I will be based on a specified SOC level as it's not really reliable estimateable when heating with heatpump or EV charger will turn on.
I also doubt that my solution will be 1:1 applicable for others. But maybe it's kind of inspiring.

ESS stopps from my point of view, one hour to early to use the battery:

Grid is at 9 pm at 35 cent which would be ok for today, but ESS stops at 8 pm 36 cent
we have still over 60 % into the battery

new VRM ID c0619ab4ae0e

Can you try again with running version 0.0.14 with firmware 3.30~7 (or above when released)? This adds extra strategies for not blaming the wrong forecasts on the grid all of the time.
Closing this issue. Please re-open a new issue if the problem persists.


ZeRO 0, 1, 2, 3 produce different results

Previously I observed that ZeRO 1 produces significant worse performance than ZeRO 2 in the finetuning. Similar observation is also mentioned in https://github.com/microsoft/DeepSpeed/issues/757. I created a simple test to see how loss changes with different ZeRO stages. The test code snippet (test_zero.py) is pasted as follows (the code is run by using deepspeed test_zero.py --zero ${ZERO}):
import os
import json
import argparse
import torch
import deepspeed
from torch import nn
from torch.utils.data.distributed import DistributedSampler

class SimpleModel(torch.nn.Module):
    def __init__(self, hidden_dim, empty_grad=False, zero=0):
        super(SimpleModel, self).__init__()
        self.linear = torch.nn.Linear(hidden_dim, hidden_dim)
        mlp = [self.linear]
        mlp.append(torch.nn.Linear(hidden_dim, hidden_dim//2))
        for _ in range(6):
            l = torch.nn.Linear(hidden_dim//2, hidden_dim//2)
            mlp.append(l)
        mlp.append(torch.nn.Linear(hidden_dim//2, hidden_dim))
        l = torch.nn.Linear(hidden_dim, hidden_dim)
        l.weight = self.linear.weight
        l.bias = self.linear.bias
        mlp.append(l)
        if zero == 3:
            deepspeed.zero.register_external_parameter(self, self.linear.weight)
            deepspeed.zero.register_external_parameter(self, self.linear.bias)
        self.mlp = nn.Sequential(*mlp)
        if empty_grad:
            self.layers2 = torch.nn.ModuleList([torch.nn.Linear(hidden_dim, hidden_dim)])
        self.cross_entropy_loss = torch.nn.CrossEntropyLoss()

    def forward(self, x, y):
        hidden_dim = x
        hidden_dim = self.mlp(hidden_dim)
        return self.cross_entropy_loss(hidden_dim, y)

def create_config_from_dict(tmpdir, config_dict):
    config_path = os.path.join(tmpdir, 'temp_config.json')
    with open(config_path, 'w') as fd:
        json.dump(config_dict, fd)
    return config_path

def get_data_loader(model, total_samples, hidden_dim, device):
    batch_size = model.train_micro_batch_size_per_gpu()
    train_data = torch.randn(total_samples, hidden_dim, device=device, dtype=torch.half)
    train_label = torch.empty(total_samples,
                              dtype=torch.long,
                              device=device).random_(hidden_dim)
    train_dataset = torch.utils.data.TensorDataset(train_data, train_label)
    sampler = DistributedSampler(train_dataset)
    train_loader = torch.utils.data.DataLoader(train_dataset,
                                               batch_size=batch_size,
                                               sampler=sampler)
    return train_loader

def get_args(tmpdir, config_dict):
    parser = argparse.ArgumentParser()
    parser.add_argument("--local_rank", type=int, default=0)
    parser.add_argument('--zero', type=int, default=0)
    args = parser.parse_args()  #args=''

    config_dict["zero_optimization"]["stage"] = args.zero
    print('config_dict["zero_optimization"]', config_dict["zero_optimization"])
    config_path = create_config_from_dict(tmpdir, config_dict)

    args.deepspeed_config = config_path
    return args

def print0(msg):
    if torch.distributed.get_rank() == 0:
        print(msg, flush=True)

rank = int(os.environ['RANK'])
print('seed:', 2222 + rank)
torch.random.manual_seed(2222 + rank)

config_dict = {
    "train_batch_size": 32,
    "train_micro_batch_size_per_gpu": 4,
    "steps_per_print": 1,
    "zero_allow_untested_optimizer": True,
    "optimizer": {
        "type": "LAMB",
        "params": {
            "lr": 0.02,
            "weight_decay": 0.01,
            "bias_correction": True,
            "eps": 1e-6
        }
    },
    "gradient_clipping": 1.0,
    "fp16": {
        "enabled": True,
        "initial_scale_power": 10
    },
    "zero_optimization": {
        "stage": 1,
        "overlap_comm": True,
        "contiguous_gradients": False,
        "reduce_bucket_size": 20
    }
}
#        "initial_scale_power": 15
args = get_args('/tmp/', config_dict)
hidden_dim = 4

model = SimpleModel(hidden_dim, empty_grad=False, zero=args.zero)

model, _, _,_ = deepspeed.initialize(args=args,
                                     model=model,
                                     model_parameters=model.parameters(),
                                     dist_init_required=True)

def print_params(tag, model):
    if torch.distributed.get_rank() == 0:
        for n, p in model.named_parameters():
            print0("{} {}:{}".format(tag, n, p))

data_loader = get_data_loader(model=model,
                              total_samples=1000,
                              hidden_dim=hidden_dim,
                              device=model.device)
#print_params('pre-train', model)
for n, batch in enumerate(data_loader):
    loss = model(batch[0], batch[1])
    #if torch.distributed.get_rank() == 0 and model.is_gradient_accumulation_boundary():
    model.backward(loss)
    model.step()
    if torch.distributed.get_rank() == 0 and model.is_gradient_accumulation_boundary():
        print("{}, LOSS: {}".format(n, loss.item()))
    #print_params('step={}'.format(n), model)
    if n == 4: break

The following shows the results:
ZeRO 0:
0, LOSS: 1.6396484375
1, LOSS: 1.4296875
2, LOSS: 1.4267578125
3, LOSS: 1.529296875
4, LOSS: 1.623046875

ZeRO 1:
0, LOSS: 1.6396484375
1, LOSS: 1.4296875
2, LOSS: 1.427734375
3, LOSS: 1.5322265625
4, LOSS: 1.626953125

ZeRO 2:
0, LOSS: 1.6396484375
1, LOSS: 1.4306640625
2, LOSS: 1.427734375
3, LOSS: 1.529296875
4, LOSS: 1.623046875

ZeRO 3 gives me error
UnboundLocalError
    : param.grad.data = dest_tensor_full_buffer.datalocal variable 'dest_tensor_full_buffer' referenced before assignment
UnboundLocalError
: local variable 'dest_tensor_full_buffer' referenced before assignment
UnboundLocalError: local variable 'dest_tensor_full_buffer' referenced before assignment
    accumulate=True if self.micro_step_id > 0 else False)
  File "/usr/local/lib64/python3.7/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 453, in partition_gradients
    accumulate=accumulate)
  File "/usr/local/lib64/python3.7/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 787, in _partition_gradients
    accumulate=accumulate)
  File "/usr/local/lib64/python3.7/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 855, in _partition_gradient
    param.grad.data = dest_tensor_full_buffer.data
UnboundLocalError: local variable 'dest_tensor_full_buffer' referenced before assignment

As can be seen, ZeRO 0 and 2 produce exactly the same result, while ZeRO 1 gives different losses. And, ZeRO 3 test fails.
@tjruwase FYI

@szhengac Set hidden dim > 35 and zero level 3 should work

@szhengac, thanks for this excellent repro recipe. I am able to repro on my side.
@aced125, thanks for sharing a workaround and for your hard work on #940. However, with hidden_dim=36, I am seeing the same UnboundLocalError with zero-3. On the other hand, with hidden_dim=36, the zero-1 regression seems to disappear.
These are very helpful clues that you have both provided to help with this investigation.

@szhengac, so I have fixed zero-3. Below is the full results, with hidden_dim=4.  I want to clarify that zero-0 and zero-2 don't match exactly, they are different on steps 1 & 2. I think my observation matches your results, right?

step
zero-0
zero-1
zero-2
zero-3

0
1.639648438
1.639648438
1.639648438
1.639648438

1
1.4296875
1.4296875
1.430664063
1.430664063

2
1.426757813
1.427734375
1.427734375
1.427734375

3
1.529296875
1.532226563
1.529296875
1.529296875

4
1.623046875
1.626953125
1.623046875
1.623046875

@szhengac . I will continue investigation on the zero-1 regression, but please do share if you have other evidences.

@tjruwase Thanks for the quick fix for ZeRO 3. I didn't note that steps 1 and 2 have different losses for ZeRO 0 and 2. I think this is due to that the behavior of LAMB is different when optimizer state partition is adopted. After I change the optimizer to Adam, I can obtain the following:

step
zero-0
zero-1
zero-2

0
1.23046875
1.23046875
1.23046875

1
1.533203125
1.533203125
1.533203125

2
1.3779296875
1.3779296875
1.3779296875

3
1.3134765625
1.3134765625
1.3134765625

4
1.220703125
1.2216796875
1.220703125

You can see that now ZeRO 0 and 2 have exactly the same numbers, while the loss at step 4 is different for ZeRO 1.

@szhengac, I get different observations for Adam on 8GPUs, see below. Of course some the differences in our observations can be due to environment, i.e., initialization and paddings.
| zero-0 | zero-1 | zero-2 | zero-3
-- | -- | -- | -- | --
0 | 1.639648438 | 1.639648438 | 1.639648438 | 1.639648438
1 | 1.399414063 | 1.399414063 | 1.399414063 | 1.399414063
2 | 1.442382813 | 1.44140625 | 1.442382813 | 1.438476563
3 | 1.525390625 | 1.524414063 | 1.525390625 | 1.514648438
4 | 1.537109375 | 1.537109375 | 1.537109375 | 1.52734375

@szhengac, the fix for the UnboundLocal has been merged, perhaps you can close this issue when you verify. Thanks.

So in your testing, ZeRO 1 and 3 have different losses from ZeRO 1 and 2. Do you have idea? As ZeRO 1 produces very bad in my finetuning task, it is good to figure it out and have consistent result.

@szhengac, I agree that ZeRO 1 requires further investigation, which I plan to do. However, it seems this particular issue was opened for a ZeRO 3 error. Please feel free to open a separate issue to track ZeRO 1 regression, where we can continue the investigation. Does that make sense?

@szhengac, on second thoughts. Let's just use this issue to at least understand the source of the different results.

@szhengac, thanks for this excellent repro recipe. I am able to repro on my side.
@aced125, thanks for sharing a workaround and for your hard work on #940. However, with hidden_dim=36, I am seeing the same UnboundLocalError with zero-3. On the other hand, with hidden_dim=36, the zero-1 regression seems to disappear.
These are very helpful clues that you have both provided to help with this investigation.

@tjruwase out of interest what did you do to fix the UnboundLocalError here?

@aced125, I saw your question about the fix. Let me know if you want more details after reviewing the PR.

@tjruwase here is the bug with, for example, nn.Conv
Please replace the SimpleModel class with:
class SimpleModel(torch.nn.Module):
    def __init__(self, hidden_dim, empty_grad=False, zero=0):
        super(SimpleModel, self).__init__()
        self.linear = torch.nn.Linear(hidden_dim, hidden_dim)
        mlp = [self.linear]
        mlp.append(torch.nn.Linear(hidden_dim, hidden_dim//2))
        for _ in range(6):
            l = torch.nn.Linear(hidden_dim//2, hidden_dim//2)
            mlp.append(l)
        mlp.append(torch.nn.Linear(hidden_dim//2, hidden_dim))
        l = torch.nn.Linear(hidden_dim, hidden_dim)
        l.weight = self.linear.weight
        l.bias = self.linear.bias
        mlp.append(l)
        if zero == 3:
            deepspeed.zero.register_external_parameter(self, self.linear.weight)
            deepspeed.zero.register_external_parameter(self, self.linear.bias)
        self.mlp = nn.Sequential(*mlp)
        if empty_grad:
            self.layers2 = torch.nn.ModuleList([torch.nn.Linear(hidden_dim, hidden_dim)])
        self.cross_entropy_loss = torch.nn.CrossEntropyLoss()
        self.conv1d = nn.Conv1d(1, 1, kernel_size=3, stride=1, padding=1, bias=True)

    def forward(self, x, y):
        hidden_dim = self.conv1d(x.unsqueeze(1)).squeeze(1)
        hidden_dim = self.mlp(hidden_dim)
        return self.cross_entropy_loss(hidden_dim, y)

and run
deepspeed test_zero.py --zero 3

Also - add CPU offloading

Actually - I seem to be getting a different error (on A100), when running the above:
RuntimeError: p.type().is_cuda() INTERNAL ASSERT FAILED at "/home/ubuntu/anaconda3/envs/torch/lib/python3.7/site-packages/deepspeed/ops/csrc/lamb/fused_lamb_cuda.cpp":49, please report a bug to PyTorch. p must be a CUDA tensor

@aced125, this expected as CPU offloading means the optimizer will execute on the CPU, and so you need CPU-implementation of optimizer. However, deepspeed/ops/csrc/lamb/fused_lamb_cuda.cp is CUDA-implementation of LAMB. We currently don't have CPU-implementation of LAMB.

@szhengac, I have a fix for zero-1 issue in this branch, can you give it a try?

@tjruwase The current option is to disable reduce-scatter?

@szhengac, yes. Reduce-scatter appears to be broken in zero-1, and it might take a while to fix. Does it work for you now?

@tjruwase Yes. This workaround works for the test cases.

Thanks for the confirmation. We are trying to fix reduce-scatter asap, bandwidth permitting. But we hope this workaround can unblock until then.

@szhengac, did you also check if zero-3 does not match others?

ZeRO 3 does not match if I increase number of steps to 8.

@szhengac, thanks for the sharing that update. I will look further. Can you also share results?

| zero-0,1,2 | zero-3
-- | -- | --
0 | 1.23046875 | 1.23046875
1 | 1.533203125 | 1.533203125
2 | 1.3779296875 | 1.3779296875
3 | 1.3134765625 | 1.3134765625
4 | 1.220703125 | 1.220703125
5 | 1.462890625 | 1.4619140625
6 | 1.263671875 | 1.263671875
7 | 1.234375 | 1.23828125
8 | 1.392578125 | 1.3955078125

@szhengac, I don't see any difference between zero 3 and others with Adam optimizer. Are your results from LAMB or Adam?

@tjruwase I use Adam.

Did you increase the number of iterations to 8?

And, my deepspeed version is '0.3.15+e414435'

@szhengac, sorry this was my error. I was running on 2gpus. However, with 8 gpus I see differences between zero 3 and others.

It looks like they just merged this fix not too long ago https://github.com/microsoft/DeepSpeed/pull/968 and incorporated into the official release 9 days ago https://github.com/microsoft/DeepSpeed/releases
I did pip install deepspeed -U and it started working

@szhengac, I think the ZeRO3 regression is now fixed. PR should be merged soon. Thanks for helping to find this corner case. I extended your test case to 17 iterations.
| 0 | 1 | 2 | 3
-- | -- | -- | -- | --
1 | 1.683594 | 1.683594 | 1.683594 | 1.683594
2 | 1.853516 | 1.853516 | 1.853516 | 1.853516
3 | 1.154297 | 1.154297 | 1.154297 | 1.154297
4 | 1.125977 | 1.125977 | 1.125977 | 1.125977
5 | 1.227539 | 1.227539 | 1.227539 | 1.227539
6 | 1.537109 | 1.537109 | 1.537109 | 1.537109
7 | 1.385742 | 1.385742 | 1.385742 | 1.385742
8 | 1.322266 | 1.322266 | 1.322266 | 1.322266
9 | 1.381836 | 1.381836 | 1.381836 | 1.381836
10 | 1.415039 | 1.415039 | 1.415039 | 1.415039
11 | 1.389648 | 1.389648 | 1.389648 | 1.389648
12 | 1.552734 | 1.552734 | 1.552734 | 1.552734
13 | 1.501953 | 1.501953 | 1.501953 | 1.501953
14 | 1.351563 | 1.351563 | 1.351563 | 1.351563
15 | 1.404297 | 1.404297 | 1.404297 | 1.404297
16 | 1.589844 | 1.589844 | 1.589844 | 1.589844
17 | 1.422852 | 1.422852 | 1.422852 | 1.422852


Update beam dataflow python container versions.

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

Run Python PreCommit

:/ It looks like the Python Precommit is related to the same streaming issue with the jenkins test project. I would run a test against a different project and specifying this container as --sdk_container_image, or installing the SDK in editable mode and then running a pipeline. If you run the gradle tests, they don't really test the default image.


Cmd+D (select multiple search occurences) is not working

VSCode Version: 1.39.1
OS Version: macOS Mojave (10.14.6)

Steps to Reproduce:

Occurs at random times
I need to restart vscode to fix it

Does this issue occur when all extensions are disabled?: Yes/No
Yes

It looks like this only happens when another Electron app is running

I think I found the cause, while another Electron app is opened, cmd+D toggles the developer tools on the other screen :)

@tetsuo I am not sure I understand the steps, nor the problem. Can you please break down the step and add more details? What Electron app do you open? Which Electron app has focus when you press cmd+D?

@alexandrudima
Hey I am using an Electron application which seems to be adding global shortcuts using electron's GlobalShortcut API.
Not sure if this is expected, but it looks like they are overriding VScode's own shortcuts.

@tetsuo Thank you for following through.
I think this is expected. If another application "steals" cmd+D, then no other running application, including VS Code receives the key combination anymore.
This is considered unfriendly, so my recommendation is to contact the other Electron application author and ask them for a configuration option to disable their global keybindings...


Refactor: Remove dependencies from zif_abapgit_definitions

zif_abapgit_definitions shall not depend on other classes or interfaces.
Current dependencies:

zif_abapgit_object
zif_abapgit_xml_input
zcl_abapgit_repo

Proposal for 1+2: Move to zcl_abapgit_objects or a new interface like zif_abapgit_objects (but that name could easily be confused with zif_abapgit_object ).
 TYPES:
    BEGIN OF ty_deserialization,
      obj     TYPE REF TO zif_abapgit_object,
      xml     TYPE REF TO zif_abapgit_xml_input,
      package TYPE devclass,
      item    TYPE ty_item,
    END OF ty_deserialization .
  TYPES:
    ty_deserialization_tt TYPE STANDARD TABLE OF ty_deserialization WITH DEFAULT KEY .
  TYPES:
    BEGIN OF ty_step_data,
      step_id      TYPE ty_deserialization_step,
      order        TYPE i,
      descr        TYPE string,
      is_ddic      TYPE abap_bool,
      syntax_check TYPE abap_bool,
      objects      TYPE ty_deserialization_tt,
    END OF ty_step_data .
  TYPES:
    ty_step_data_tt TYPE STANDARD TABLE OF ty_step_data
                                WITH DEFAULT KEY .

Proposal for 3: Move to zif_abapgit_repo_srv:
  TYPES:
    ty_repo_ref_tt TYPE STANDARD TABLE OF REF TO zcl_abapgit_repo WITH DEFAULT KEY .

currently its not optimal, but its also not a big problem AFAIK
1+2: can try moving it to the classes, but it might fail, due to the abapmerg'ed report

the problem comes when you want to reuse zif_abapgit_definitions somewhere else (the installer ;-) ) and it drags the complete abapgit along via these dependencies. I will give it a try.

okay 👍


Update the examples to 0.13.1 and above

Since we now have a new NN plugin system, let's update the examples:

Use the official release files to install WasmEdge and plugins in the GitHub Actions CI. Do not build from the source.
Create CI actions for TF, TFLite, PyTorch, OpenVINO examples. We could have 4 separate GitHub Actions yaml files -- one for each example.
Update the wasi-nn crate in all examples to the latest (no more unsafe)
Update the README for each example to be consistent with the successful CI.
Note in the PyTorch and TFLite READMEs how to run on GPUs.

Do we still need this issue?

I will close this. We will need a different set of docs.


addressed circular dependency on fargate profiles #680

Issue #, if available: #680
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/do-e2e-tests


Detangle trait solving from SInto

This PR makes it so trait solving never calls into SInto, as that was causing headaches with binders. There is still more work to do to clarify trait solving but that's less of a priority.

As usual, the charon failure is expected, I have a fix ready


EoS: Implement FindCoordinator v1

See #165
This PR implements the FindCoordinator V1 protocol as described in the issue. This protocol will now be consumed by the broker in its findGroupCoordinator call. I suppose this could have gone behind some sort of feature flag (like allowExperimentalV011) but AFAIK it should play find with the existing code.
For backwards compatibility I chose to maintain the groupId option and map it to the new coordinatorKey option.
I've had some trouble getting the full test suite to run consistently on my local machine, but the relevant tests appear to be passing 😬
This is my first PR touching the protocols & the first starting work on the EoS stuff, so please let me know if there's anything that should be done differently 😄 Thanks!

@ianwsperber awesome work. I've added a couple of commits to accept the new argument on the broker.
About the tests, they have been working really well for me in the local environment. What I usually do is to run the Docker container on a separate terminal and then run ./scripts/createScramCredentials.sh to create the credentials. Then you just have to use yarn test:local to run all tests or yarn test:local:watch src/<> for some specific part you are developing. Maybe we should document this workflow on the readme.

I chose to maintain the groupId option and map it to the new coordinatorKey option.

This is a sensible decision

Thanks for the review @tulios! If you don't get to it first I could add those testing steps in the next day or so. Maybe it'd be helpful to create a CONTRIBUTING.md to collect this an related info?


Impement cameras pagination

fix #130

https://github.com/montevideo-tech/hybrid-meetup-platform/issues/130

ParticipantsCollection
It is the collection of participants container,
Receives as props, the appropriate width and height, calculated according to the size of the screen.
It has a pagination and it has a maximum of 25 participants per page.
Receives as a prop participantsPerPage (the number of participants that also fit per page according to the width and height of the screen) and participantsCount (the number of remote participants)
ParticipantLayout
It is in charge of rendering the videos with the optimal size based on the calcOptimalBoxes function.


Shape Up - Cycle 2: Editor Improvements

Shape Up - Cycle 2: Editor Improvements

New components + fields specification

🤖 Generated by Copilot at 455b9b7
This pull request adds new features and improvements to the @dcl/inspector package, which is a tool for inspecting and editing entities in a Decentraland scene. It updates the dependencies, adds support for more entity components, enhances the GLTF inspector, and introduces a new color field component. It also adds some type definitions, utility functions, and unit tests for the new components.
Leftovers:

Proper numbers fields UI (we already have a design by mel)
Proper "uvs" UX (currently disabled since there is no UI for this)
"fontAutoSize" & "lineCount" for TextShape in babylon (I don't know if this is actually working on unity explorer, but...)
"avatarTexture" & "videoTexture" UI + full implementation (need spec for this)

There seems to be an issue updating the engine when the EntityInsector is not open. When I add a MeshRenderer component to an entity, it is not rendered until I select it:
https://github.com/decentraland/js-sdk-toolchain/assets/2781777/a4aa8b85-e597-4482-b7fe-5a40345ecf3e

Also, when I change values of the albedo, emissive and reflectivity colors, every time I change one, the previous one is reseted:
https://github.com/decentraland/js-sdk-toolchain/assets/2781777/96801e54-e07f-4ccd-b404-619423ba1bcf

Is it okay that the default value for the emissive is 2? It brights a lot, this cube almost looks like an sphere:

I know I can tweak it, but to me it's strange that this is the default value of a newly created material.

This looks like an issue on the babylon renderer, we are always rendering the bounding mox of a MeshRenderer component when it is selected, even when it's not a cube. Like here this sphere shows a box around it:

I noticed that this PR is not being able to build the Cloudflare Pages deployment. This was fixed on master, you can pull to get it fixed 🙏

There's something that I notice when I test this, and is that there are many components that require a texture (or a path to a texture to be precise). This is not an issue on VSCode because the user can access the file system to add the texture, but on the Web Editor this would not be possible.
Our current import flow only allows the user to import GLB/GLTF files, so there's no way for the Web Editor users to use many of these components.
Im not saying we need to address that issue on this PR, but I think it's important to allow users to import other type of files as as soon as possible after merging this, otherwise they wont be able to fully use the feature

@cazala

MeshRenderer not shown on babylon ✅
Colors getting reset ✅
Emissive default value ✅
That's the current default behaviour, but we don't notice it since we only allowed for cubes, thus the boundingBox was "invisible" (but it was there). We can remove it, but I think we should keep it. 🤔
Rebased PR ✅
Agree on updating the Web Dditor

There seems to be an issue with the TextShape component. When I apply it to an entity, I can't click anything else that is further away from the camara, as if it is blocking it (even tho they are not behind the text).
After inspecting the normals, you can see there is a huge plane attached to the entity with the TextShape component.
https://github.com/decentraland/js-sdk-toolchain/assets/2781777/69089648-6c96-402d-ac93-dc407516a533
There are also a few CSS issues with the TextShapeInspector, some paddings missing:

Also on some components like MaterialInspector the TextField when it is on error state, the red strip-lined outline is not centered:


Pull Request Failure

@josuebrunel Hello! I'm hoping you're at least aware I've forked your repository. More than that, I've made some additions to the code too. I tried to push the changes I made back to my forked repository. I also tried to submit a pull request of my changes to your repository... and I'm not sure I did either of those things correctly. Let me know if there's something I need to do for you to view my changes.
Two more things: 1) Once you let me know you can see my updates, I'm happy to discuss what other functions should be included in the fantasy sports program. Also, I tried running your code in a virtual env and got an error again, which means I wasn't actually able to test any of the code I wrote in this pull request. While I believe most of it should work, there were a few things I wasn't sure of (like the filters in the players functions). Of course once I get your code running, I'll test any code I attempt to add.
Thank you!

@unpairestgood hello pal !!

I'm hoping you're at least aware I've forked your repository

Yes i'm

More than that, I've made some additions to the code too. I tried to push the changes I made back to my forked repository. I also tried to submit a pull request of my changes to your repository... and I'm not sure I did either of those things correctly. Let me know if there's something I need to do for you to view my changes.

I figured out what you did. When you want to contribute to a project :

You fork it
You make the changes on the forked repository
As soon as your changes are made and tested, you can submit a pull request

The only thing you did wrong, was to create your own repository and make the changes there. That is why you can't submit a pull request.
Therefore, the thing to do is to make the changes in the forked repository.

Once you let me know you can see my updates, I'm happy to discuss what other functions should be included in the fantasy sports program

I did see your changes

I wasn't actually able to test any of the code I wrote

Ok i see. Could you try this please :
$ git clone https://github.com/unpairestgood/yahoo-fantasy-sport
$ cd yahoo-fantasy-sport
$ virtualenv env
$ source env/bin/activate
$ pip install -r dev_requirements.txt
$ ./run_test.sh

It should work ( let me know if i doesn't )
####reminder : work on the forked repository not on this one.
Keep me posted, going for some :bicyclist:

Josue, I was following those instructions and everything was running smoothly until the last line wouldn't run. Then I realized it should be ./run_tests.sh (not run_test), yes? I believe I ran the test script successfully, though there does still seem to be an error. Here's what the output looked like:

...

Also thanks for explaining what I was doing wrong with regards to working on the forked repository. I haven't tried yet, but I'm sure I'll get that to work and I'll make more progress today!
Best,

@unpairestgood ,
I think you have to put a oauth.json file the folder. This file must contain at least your consumer_key and your consumer_secre

I do have the oauth.json file in the folder. I wasn't sure if it should be in the yahoo-fantasy-sport folder or the env folder... and I've now tried both to no success. I'll try starting the process again from scratch, just in case I made an error at some point because I don't have any ideas why it's happening. Just in case it helps you, here's what my yahoo-fantasy-sport folder looks like:

I've been making good progress on adding functions today, though they are getting increasingly more complex when you can access sub-resources and/or include filters, for example. It'd be nice to be able to test these. Anyway, have a good one!

Just want to update, I got the yahoo-fantasy-sport tests to work. It's a little embarrassing, but I think the issue was simply I had an extra comma after my consumer_secret in the oauth file. I'm excited to finally get to test my new code...

ahahah cool. 2 important points though :

add some tests about your new features in the tests.py file. I tried to make it clear. Let me know if you need any help.
Never ever push your credential file to github or anything alike. Unless it's encrypted, just like the oauth.json.enc

I've seen your changes, good job buddy. Hope all tests go smoothly


Fix bullet point formatting in README

The extra space after the bullet point was causing the second line in the first bullet point to render as a block quote.

Thanks @mivade !


More tests + extra features [Iteration 6] && [Iteration 7]

[x] Normalize composer.json (https://github.com/localheinz/composer-normalize)
[x] Add PHPCS to travis with PSR-2 rules
[x] Composer project installation (http://php-pds.com/)
[x] Fill community profile (https://github.com/phalcon/vokuro/community)
[x] ~Database tables review and refactor through migrations~
[x] Code refactor + Unit tests
[x] Code refactor + Functional tests
[x] Fix email confirmation

Looks good!

Implemented.


Upate main README

Update Pocket Link
Add a ToC
Remove duplicate headings

@arno01 Could you re-review when you have a chance?

@arno01 Wanted to bump this PR


Lambda Powertools Idempotency Support for SQS Batch Processor

We use the Powertools SQS Batch Processor utility with our Lambda function. The documentation suggests implementing idempotency into the function logic. We were trying to leverage the Powertools Idempotent feature. There seems to be some limitations with this approach as the Idempotency feature only checks against the entire SQS batch of records instead of each individual record in the batch. So kind of defeats the purpose of the SQS Batch Processor.
Is there any way we can utilize powertools idempotency utility in our lambda with the SQS batch processor utility having the idempotency utility check at an individual record rather than the full batch.
Reading through the source code, it seems like it assumes that it will wrap the lambda handler only so it seems impossible?
Thanks.

Hey @kimberlyamandalu, thank you so much for flagging this.
We've been meaning to create or update Idempotency utility to become a decorator for any standalone function, not just Lambda handler.
Adding this to backlog as we totally should do this and appreciate any help on implementing it
cc @cakepietoast who did the original implementation for ideas.

Just had a look at the source code and it'd indeed need some major internal changes to support any function with any signature. It'd be relatively quick to support any function with a single parameter (like Batch Processor Handler), but if we want to do it right, we should support any function not just the processor handler.
@cakepietoast when you get some bandwidth, could you look into it as you're way more familiar with the implementation?

@heitorlessa Thank you very much for adding this to your backlog.
We are looking into the code as well. Will definitely share any findings or suggestions for how to implement. At the moment, what we've noticed is that, the idempotency utility is quite dependent on the Lambda Context object, but it mainly uses it to add the Lambda function name to the primary key value in the DynamoDB table, which makes sense.
@cakepietoast thanks for looking into it!

Yes! That can be easily replaced with fn.name since is a sync function

I think it was an oversight from our part.

On Mon, 16 Aug 2021 at 14:32, kimberlyamandalu @.***>
wrote:

@heitorlessa https://github.com/heitorlessa Thank you very much for
adding this to your backlog.
We are looking into the code as well. Will definitely share any findings
or suggestions for how to implement. At the moment, what we've noticed is
that, the idempotency utility is quite dependent on the Lambda Context
object, but it mainly uses it to add the Lambda function name to the
primary key value in the DynamoDB table, which makes sense.
@cakepietoast https://github.com/cakepietoast thanks for looking into
it!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/awslabs/aws-lambda-powertools-roadmap/issues/32#issuecomment-899473020,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAZPQBATI7S6SS3UI6LUGOTT5EAO3ANCNFSM5CCEWOCQ
.

Thanks for moving this so quickly to Working on it!  One thought on removing the need for Lambda Context object just to access the Lambda Function Name for building the Dynamo Key is to leverage the default Lambda Environment Variable for the Lambda Function Name - AWS_LAMBDA_FUNCTION_NAME
Source: https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars.html#configuration-envvars-retrieve

As I was getting to know better the idempotency logic I realized if I remove it it'll break people's unit tests. Relying on AWS_LAMBDA_FUNCTION_NAME also makes it inconsistent with how people expect to test things like logger.
I'll create a new decorator instead, though I'm pondering on two solutions and would appreciate your help deciding it:
A. Single argument only: Accept any function with a single argument where the argument is the actual payload we'll hash and optionally extract idempotency tokens (event_jmespath_key config)
B. Function parameter signature: Accept any function with an arbitrary number of parameters, but must have a keyword named argument named payload.
For example:

Option A

# WORKS
@another_idempotent_name_i_need_to_figure_out
def record_handler(record: Dict): ...

# WILL NOT WORK
@another_idempotent_name_i_need_to_figure_out
def record_handler(record: Dict, my_other_arg: str = ""): ...

# WILL NOT WORK
@another_idempotent_name_i_need_to_figure_out
def record_handler(record: Dict, my_other_arg: str): ...

Option B

# WORKS
@another_idempotent_name_i_need_to_figure_out
def record_handler(*, payload: dict): ...

# WORKS
@another_idempotent_name_i_need_to_figure_out
def record_handler(*, payload: dict, **kwargs): ...

# MIGHT WORK MIGHT NOT WORK, DEPENDS ON HOW IT'S CALLED
@another_idempotent_name_i_need_to_figure_out
def record_handler(payload: dict, **kwargs): ...

    # WORKS
    # record_handler(payload={}, foo="bar")

    # WILL NOT WORK
    # record_handler({}, foo="bar")

I personally prefer Option B
# WORKS
@another_idempotent_name_i_need_to_figure_out
def record_handler(*, payload: dict, **kwargs): ...

I think the flexibility this will offer allows the idempotent utility for non lambda handlers to be easily extensible, if needed.
This way, we can also have more specific arguments that can be passed to the decorator function (vs having all the input arguments in a single dictionary object as it might get confusing).
Let me know if this makes sense or if I have terribly misunderstood your point.
Thank you!!

Oh I see, you want a third option in this case then @kimberlyamandalu like:

Allow me to have a function of any signature with any number of arguments and keyword arguments

We could do that too, I didn't factor that in because of the JMESPath feature.
This feature allows you to select a portion of the incoming payload and extract what you want as Idempotency tokens.
I'll get back to the whiteboard, write a RFC and then ping it back here - meanwhile, I'll have another look at creative solutions to simply integrate the Batch utility as-is too
Thanks a lot for the help

hey @heitorlessa,
Thanks a lot for the feedback.
To clarify, I picked one of the choices you had listed for Option B (the second one that works). Wanted to make sure we were on the same page =)
Currently, the JMESPath feature is included in the Idempotency Config object that is passed to the idempotent function. We do want to have this feature retained. I think it is extremely useful.
# WORKS
@another_idempotent_name_i_need_to_figure_out
def record_handler(*, payload: dict): ...

##### SELECTED THIS OPTION #####
# WORKS
@another_idempotent_name_i_need_to_figure_out
def record_handler(*, payload: dict, **kwargs): ...

# MIGHT WORK, DEPENDS ON HOW IT'S CALLED
@another_idempotent_name_i_need_to_figure_out
def record_handler(payload: dict, **kwargs): ...

    # WORKS
    # record_handler(payload={}, foo="bar")

    # WILL NOT WORK
    # record_handler({}, foo="bar")

Hey @kimberlyamandalu I've updated Slack but updating it here too.
We've finally come to an acceptable design to keep all features. We had to make three compromises for this to work:

A new idempotent_function decorator for any synchronous function with any number of arguments and keyword arguments (a.k.a named arguments)
idempotent_function decorator requires a new parameter: data_keyword_argument.

You'll have to tell us which keyword parameter in your function signature has the data we should be performing all operations - hashing for idempotency key, extract data and/or validating parameters using JMESPath, etc.

When calling your function decorated with idempotent_function, you must ensure it uses a keyword argument or else we will raise a RuntimeError for not knowing which data to hash:

Works: record_handler(record=data)
Does not work: record_handler(data)

For the Batch utility, you don't have to worry about 3 as we are doing this on your behalf. Assuming you want to make sure you only process a SQS record that has a unique message ID once, this is how it'll look like in the next release:
from aws_lambda_powertools.utilities.batch import sqs_batch_processor
from aws_lambda_powertools.utilities.idempotency import idempotent_function, DynamoDBPersistenceLayer, IdempotencyConfig

dynamodb = DynamoDBPersistenceLayer(table_name="idem")
config =  IdempotencyConfig(
    event_key_jmespath="messageId",
    use_local_cache=True,
)

@idempotent_function(data_keyword_argument="record", persistence_layer=dynamodb, config=config)
def record_handler(record):
    return do_something_with(record["body"])

@sqs_batch_processor(record_handler=record_handler)
def lambda_handler(event, context):
    return {"statusCode": 200}

Here's a dummy example to demonstrate the constraint 3.
from aws_lambda_powertools.utilities.idempotency import idempotent_function, DynamoDBPersistenceLayer, IdempotencyConfig
import uuid

records = {
      "messageId": "059f36b4-87a3-44ab-83d2-661975830a7d",
      "receiptHandle": "AQEBwJnKyrHigUMZj6rYigCgxlaS3SLy0a...",
      "body": "Test message.",
      "attributes": {
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "1545082649183",
        "SenderId": "AIDAIENQZJOLO23YVJ4VO",
        "ApproximateFirstReceiveTimestamp": "1545082649185"
      },
      "messageAttributes": {
        "testAttr": {
          "stringValue": "100",
          "binaryValue": "base64Str",
          "dataType": "Number"
        }
      },
      "md5OfBody": "e4e68fb7bd0e697a0ae8f1bb342846b3",
      "eventSource": "aws:sqs",
      "eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:my-queue",
      "awsRegion": "us-east-2"
}

persistence_layer = DynamoDBPersistenceLayer(table_name="idem")
config =  IdempotencyConfig(
    event_key_jmespath="messageId",
    use_local_cache=True,
)

@idempotent_function(persistence_store=persistence_layer, data_keyword_argument="data", config=config)
def handler(any_arg, data, and_kwarg_too=False):
    """Dummy function with an arbitrary number of args and named args"""
    return {"some_id": f"{str(uuid.uuid4())}"}

ret = handler("blah", data=records, and_kwarg_too=True)
print(ret)

Looks perfect - thanks again for picking it up and moving on it so fast! Should be seamless for us to move to this after the next release!

Merged! Coming in today's release


Fix style in Header

Details

This PR removes an extra style in the header which was introduced in the VBA step counter.
Fixed Issues

$ https://github.com/Expensify/App/issues/6297
Tests / QA Steps

Open the app and check the DEV/STG badge is near to the avatar icon.

Tested On

[x] Web
[x] Mobile Web
[x] Desktop
[x] iOS
[x] Android

Screenshots

Web

Mobile Web

Desktop

iOS

Android

e2e is being weird here, seems like everything is done but Travis is stuck

Going to merge instead of restarting the tests.

:hand: This PR was not deployed to staging yet because QA is ongoing. It will be automatically deployed to staging after the next production release.

🚀 Deployed to staging by @Jag96 in version: 1.1.15-18 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

🚀 Deployed to production by @roryabraham in version: 1.1.16-10 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅


#16498 - Linux mint LMDE (Debian Edition) support added to install script

Fixes issue #16498
Please let me know if/how to write tests for the install script.

Please sign your commits following these rules:
https://github.com/docker/docker/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:
$ git clone -b "linuxmint" git@github.com:justsml/docker.git somewhere
$ cd somewhere
$ git commit --amend -s --no-edit
$ git push -f

Ammending updates the existing PR. You DO NOT need to open a new one.

weak LGTM
Ping @tianon @jfrazelle

@tiborvass thx... How's it looking now?

... i thought I was missing some commits before the automatic tests would pass...

@justsml argh, my bad sorry! Apparently it was good but was confused by the automatic labeling. It's now worse because there are 3 commits :(
Try this:
git checkout linuxmint
# beware: next command will make all uncommitted changes and commits in linuxmint branch go away
git reset --hard upstream/master   # assuming upstream is the docker/docker repo, otherwise replace it
git cherry-pick b012a8df34e354e73d72a4f0c70048cf020cda3b
git push -f origin         # assuming origin is your fork

@tiborvass thanks for the git commands

We should not have to add this, the other platform support function 'check forks' should cover this NOT LGTM

Hey @jfrazelle currently the check_forked() function hopes lsb_release -u is supported (it errors on my fresh LMDE install),
What are your thoughts on my updating check_forked() w/ code to handle when $lsb_release == 'linuxmint' && $lsb_release_exit_code != "0" ?
Thanks for the quick responses!!!

hmmm idk i wonder if we can push upstream to them? or how have we handled others @tianon I have forgotten?

cc @fmoliveira

Here's a basic implementation:
https://github.com/docker/docker/pull/16533

#16533 looks more interesting as it looks more generic, I'll join the discussion there first.

Linux Mint has many editions. Some are based on Ubuntu, others on Debian. So, #16533 is more reliable.

I'm going to close this in favor of https://github.com/docker/docker/pull/16533, thanks so much!


New funcionalities added

Project Requirements Achieved:

create new posts funcionality
add new comments funcionality
add new likes functionality
add current user functionality

Hello  @redmarverick kindly ensure  that your pull request has a descriptive title. Instead of saying  new functionality added, can you be more  explicit and detailed? What new functionality are you adding?

Request done @jerryowusu

Request done @jerryowusu

I thought you meant the description, now I've changed the title too.
@jerryowusu


#Provide intro for create-react-native-app and react-native init

Reference pull request: #2206
Here I try to provide intro using both create-react-native-app and react-native init.

@matthamil @davidtrogers How about this? I have provide both for CRNA and react-native init. Thank you.

Good!

Merged #2491


ghostscript: add patches for CVE-2019-3839

Motivation for this change
Resolves https://github.com/NixOS/nixpkgs/issues/61954
I'm guessing this might need to be re-targeted to staging.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[ ] NixOS
[x] macOS
[x] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Assured whether relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

Testing the backport right now. Will link here once done.
Thanks for your PR @risicle! :+1:

FWIW not needed with ghostscript 27, part of https://github.com/NixOS/nixpkgs/pull/61469 FWIW.


Logging indirection

Issue link:
https://github.com/Cloud-City-RS/cloud_city_platform/issues/781
The sole point of this PR is to do two things:

replace default android's Log. calls with our own indirection class which can forward it to Sentry, Timber and other loggers we may end up installing
to verify and ensure that logs do get sent to sentry, and ensure that we use manually typed-out logtags rather than the getClass().getSimpleName() which will break the moment we obfuscate the codebase.

oh and also to get rid of Log.x() calls and replace them with CloudCityLogger.x() calls instead.
Whether I should have ended up using CloudCityLogger in the OMNT's part of the codebase is debatable; perhaps those could have been OMNTLogger which would apply stricter logging rules. But, lets see how much crap it ends up spewing on sentry and whether we can even use the logs we end up getting there before reducing the bulk.
Oh right, this PR also implemented timber and connected it with sentry :D or should have connected it to sentry at least.

Summary by CodeRabbit

New Features

Introduced a new logging utility, CloudCityLogger, enhancing the application's logging capabilities across various components.

Bug Fixes

Improved error logging by incorporating exception details in log statements throughout the application.

Chores

Updated logging statements in multiple classes to use CloudCityLogger instead of the standard Android Log class, ensuring consistency in the logging framework.
Adjusted import statements to remove android.util.Log and include cloudcity.util.CloudCityLogger where necessary.

Documentation

Added import statements for CloudCityLogger in relevant files to support the new logging mechanism.

Tip
For best results, initiate chat on the files or code changes.
@RkShaRkz, understood. After you merge PR #50 and sync this branch, I'll remind you to address the TODO in CellUtil where the new getCellInfo() is implemented, after sanitizing the EARFCN. Would you like me to open a GitHub issue to track this task?
✏ Learnings added

@coderabbit recheck if the TODO has been addressed please


hi please explain fzf how it works

like topic thank you i built spotify player with fzf feature i dont know how to use it thank  you

https://en.wikipedia.org/wiki/Approximate_string_matching
tldr, corrects for typos a bit when you search

ok thank you


FLINK-36487 remove deprecated method createTemporaryView StreamTableE…

…nvironment

What is the purpose of the change
https://issues.apache.org/jira/browse/FLINK-36487 Remove deprecated method StreamTableEnvironment createTemporaryView
Brief change log
Removed deprecated method
Verifying this change
This change is already covered by existing tests
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no
If yes, how is the feature documented? not applicable

@flinkbot run azure


Validate route name

Previously route name could take incorrect value with spaces, or numbers at start.
@asvetlov

Was this an issue? As far as I can tell, they're only ever used with getitem` and not attribute access, so they shouldn't cause any trouble if they contain silly characters.
I'd used colons for namespacing in my route names (e.g. 'oauth2:authorize'), which I would have thought would be reasonable.

Well, initial issue was using just literal text phrases for route names,
which is definitely wrong.
I agree to relax the rule for allowing identifiers separated by dots or
columns.
On Tue, Oct 20, 2015 at 11:41 AM Alexander Dutton<EMAIL_ADDRESS>wrote:

Was this an issue? As far as I can tell, they're only ever used with
getitem` and not attribute access, so they shouldn't cause any trouble
if they contain silly characters.
I'd used colons for namespacing in my route names (e.g.
'oauth2:authorize'), which I would have thought would be reasonable.
—
Reply to this email directly or view it on GitHub
https://github.com/KeepSafe/aiohttp/pull/567#issuecomment-149478786.

--
Thanks,
Andrew Svetlov

Fixed by 0.18.1 release


about steps related to the reward

Hi Kosuke,
I've tried your model on breakout game.  The performance was amazing, the average score went up to  520 after 80M steps. It's far more better than any other model I've tried.
But the average score didn't went up much after 80M.  Sometimes game costs plenty of steps.   When the bricks left 1 or 2 pieces, the ball went between your board and the wall, but never hits the remain bricks.
Do you think if it's better to add steps of games as the penalty to calculate R? such as
R-=beta*sqrt(steps)
Thanks
BTW, I've do some changes
1 changed the ACTION_SIZE= 4, because breakout have 4 actions in ALE.
2 If lives is lost, treat it as terminal
#if not terminal_end:
if lives==new_lives and not terminal_end:
R = self.local_network.run_value(sess, self.game_state.s_t)
else:
#print("lives cost from %d to %d"%(lives,new_lives))
lives = new_lives

But the average score didn't went up much after 80M.
With default "MAX_TIME_STEP" constant in constants.py, MAX_TIME_STEP is 100M.
Learning rate is annealed to 0 along 100M steps, so learning rate around 80M will be small.
One way to try is using bigger MAX_TIME_STEP like 150M or 200M, but I guess this won't take effect.

BTW, @Itsukara has also reported same result that score stops around 80M.
http://itsukara.hateblo.jp/entry/2016/08/02/190029
Maybe he is using A3C-FF mode?
He also tried live-loss-as-terminal, and the learning speed was faster.
(Reaching score 400 around 18M steps?)
https://cdn-ak.f.st-hatena.com/images/fotolife/I/Itsukara/20160824/20160824034536.png
http://itsukara.hateblo.jp/entry/2016/08/11/003715

Do you think if it's better to add steps of games as the penalty to calculate R?
Ah I see. It might take effect.

There is another A3C implementation which reports higher breakout score.
https://github.com/ppwwyyxx/tensorpack/tree/master/examples/Atari2600
Their A3C learning code seems not yet released, but please check once their code is released.

Thanks for your reply, @Itsukara improvement looks quite good. I'm trying it now.

@congling
Sorry my mistake, his graph
https://cdn-ak.f.st-hatena.com/images/fotolife/I/Itsukara/20160824/20160824034536.png
was graph of "live loss as -1 reward."
And he said that he was using A3C-FF.
thanks @Itsukara !

Btw the code which @congling has used is not equivalent to resetting game on life lost. :p This is because all it says is that if you just lost a life, then it does not matter what the value estimate of next state is, treat it as 0.
A more correct approach to resetting simulation on lost lives is :
if terminal or (RESET_ON_LOST_LIFE and old_lives != new_lives):         terminal_end = True
on this line.

@congling  conglinghello, i am a little confused where to add the code with lives in the released code. I have problem trained with Breakout and i only reached 50 when steps are 18m. Can you give me your implemention code ? thank you.

@1601214542 hey i have this problem too! hmm. is 50 train or test score(in which you get 5 lives to spare)? how often is it?


ios textarea placeholder padding bug

bug report: http://dotwe.org/vue/7104cf9654fe0f09a87243b18b97bb74
detail: when i reinput something, placeholder padding would lost in iphone.

@zhuzhaoyuan

This issue is outdated for a long time and will be closed now. You can create a new one if you still have questions.


Implement role-based access on website

See epic here: https://github.com/CDLUC3/dmsp_frontend_prototype/issues/21
Based on roles created here: #139
Will relate to a backend ticket too
Check roles in middleware


private link for ADF is wrong

[Enter feedback here]
Should private link for ADF be privatelink.datafactory.azure.net instead of privatelink.azure.net ?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: a88af182-32aa-a8f3-359d-b92ae1cfb8c7
Version Independent ID: 01ce07fc-4bc2-def5-9ecd-a80330ad9488
Content: Azure Private Endpoint DNS configuration
Content Source: articles/private-link/private-endpoint-dns.md
Service: private-link
GitHub Login: @allensu
Microsoft Alias: allensu

@Alezis
Thanks for your feedback! We will investigate and update as appropriate.

@Alezis Just to confirm that you are referring to the below, right?
Azure Data Factory (Microsoft.DataFactory/factories) / portal | privatelink.azure.com | azure.com
This needs to be updated to privatelink.adf.azure.com and adf.azure.com. We are working on updating the document accordingly. If you have any other concerns, please let us know. Thank you!

Since we have not heard back from you we will now proceed to close this thread. We are actively working on updating the document with the new change. If there are further questions regarding this matter, please tag me in your reply. We will gladly reopen the issue and continue the discussion.


ResourceName is not printed

Godot version:

3.2.3 RC4 mono
OS/device including version:

Win10 64 home
nvidia gt440
Issue description:

The name of the resource loaded in the AudioStremPlayer is not printed on the screen.
-This also happens in 3.2.2 mono
-This also happens with the texture of a node sprite
-This also happens with GDScript in 3.2.3 RC4
Steps to reproduce:
Create an AudioStreamPlayer node
Add an audio file
Print the resource name to the screen using "Stream.ResourceName"
Minimal reproduction project:

bugResourceName.zip

Did this work with earlier versions (like 3.2.2)?

I can't test with that 3.2.2 version, I get multiple csproj errors.

Well, that's because the format has been changed between 3.2.2 and 3.2.3; you'd need to create a new minimum test project with 3.2.2.

Ok in 3.2.2 it works the same.
I mean nothing changes.

Does this occur with any other resources?

This also happens with the texture of a sprite node

Can you reproduce this in GDScript too?

If you mean, <resource>.resource_name, nothing is printed in my test. Using print (stream.resource_name) where stream is in this case an AudioStream in a function in my code.

This also happens with GDScript in 3.2.3 RC4

It seems this only happens with ExtResource types. SubResources have their resource_name property set but ExtResources do not. This has been around for a while, as it's present in 3.1.2. I made a test project in 2.1.6 and it is present there as well, although the default resource_name is the filename instead of an empty string.

This continues to happen in the last official RC.
I was trying to put the screen name of the file that the streplayer plays but then I see that it did not work and I remember that I had already reported this.

I just had this same issue. Whenever I try printing a texture's resource name it prints an empty string. It's also treated as an empty string when I try to compare it using an operator.
Using GDscript on godot 3.3

This problem still exists in: v3.5.beta1.official [b9b23d222]
The resource path is correctly printed with "resource_path", however "resource_name" does not print anything.

found it.

this is what the resource_name is trying to find.
ik im late to it but oh well

So... I thought that the resource name referred to the resource loaded as a texture, that is, the name of the image that the user assigns to the texture.
Thanks for the information!.

So... I thought that the resource name referred to the resource loaded as a texture, that is, the name of the image that the user assigns to the texture. Thanks for the information!.

No problem! i thought that too, so i searched and found this post, but i eventually realized it so yeah :D


[BUG] Soul vial can't capture a specific Ars Nouveau mob

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior
Modpack:
DW20 1.20 v1.3.1
Ars Nouveau Version:
ars_nouveau-1.20.1-4.7.4-all
6.0.21-alpha
Issue:
Trying to capture a Starbuncle (Ars Nouveau) with a soul vial from EnderIO but can't. Don't know if it's the hitbox or something but it won't trigger the capture. Other mods such as Shrink mod does capture the mobs with no issue.
I've tried Drigmy and Whirlsprig and those work without a problem
Expected Behavior
When using the Soul Vial, it should capture the mob (in this case the Starbuncle from Ars Nouveau)
Steps To Reproduce
Trying to Capture Starbuncle

Capturing Whirlisprig (from same mod)

Environment
- OS: Windows
- Java version: v8 u391
- Minecraft version: 1.20.1
- Version: 6.0.21-alpha
- Modpack: DW20 1.20 v1.3.1

Anything else?
I've seen a lot of duplicate issues around the same matter and even one stating that on .16 the issue would be fixed.
I think this might be a different issue from that, as the mobs from the mod itself are being captured successfully with the exception of that one.

This seems to be an issue from Ars Nouveau side and not EnderIO (checked with them and confirmed it). The issue has been reported to them and will be fixed in Ars Nouveau. Thanks for reporting!


I write a loader and want pass some options to it but I get ‘{}’ when pass not json data

I am using my own loader like:
 {
            loader: 'my-loader',
            options: {
                  xxx: /regex/
            }
 }

Then i use import { getOptions } from 'loader-utils'; to get options.
I will get xxx: {} but not the regex I am passing.
When I rewrite the RegExp.proptotype.toJSON by
Object.defineProperty(RegExp.prototype, 'toJSON', {
    value: RegExp.prototype.toString,
});

I will get xxx: /regex/.
It that a bug or something wrong with me?
Other relevant information:
webpack version:  4.41.2
Node.js version:
Operating System:
Additional tools:

Thanks for your report. It would be great if you reduce your issue to a small reproducible example. Best put this example into a github repository together with instructions how to get to the problem.

I find a way to reproduce this behavior.
It is relative with thread-loader.
Using the configs below, I will get "{}"
// get `{}`
[
    'thread-loader',
   {
            loader: 'my-loader',
            options: {
                  xxx: /regex/
            }
   }
]

However, when I switch their position, it works as expected.
// expect
[
   {
            loader: 'my-loader',
            options: {
                  xxx: /regex/
            }
   },
    'thread-loader',
]

What is the best pratice to use 'thread-loader'? Thanks

Thread loader doesn't support non-json options as it has to move these options to another thread.

Got it!
Are there any documents explain how thread-loader works?


DAWN-694 ⁃ Unit tests for boot sequence

Create the unit tests to the boot sequences which covers:
https://blockone.atlassian.net/wiki/spaces/ID/pages/49774623/R3.0+Boot+Sequence

Related PR is #1800

ATC:
Run
tests/chain_test --report_level_detailed --run_test=bootseq_tests and ensure everything pass


Update plumber logo

Remove previous plumber logo and add new one.
Closes https://github.com/rstudio/plumber/issues/613

@blairj09 Does the PNG have the size 2206 x 2557?
Readme.md:

A .png in the PNG directory (with dimensions 2206 x 2557).

@schloerke I adjusted the PNG size and it's currently 2206 x 2547, so 10 pixels off 😬

@allisonhorst Do you know why the plumber logo doesn't scale to the preferred size? ...kinda fun to look at the onion skin view of 915ef41 (#29)
Just want to make sure I didn't do something odd and we conform like the other logos.  Thank you!!

No idea & I'm curious about this too. Whenever I rescale, starting from a perfect hexagon, they're always 2206 x 2547. I wonder if @gregswinehart has the answer?

@schloerke is this now the right size?


Atlas and template revision

This PR replaces https://github.com/miykael/atlasreader/pull/60 and therefore closes https://github.com/miykael/atlasreader/pull/60.
This PR updates the templates and atlases and adds a README file that describes everything, explains the origin and points to the different references and licenses.
Summary
I tried to make the commit messages self-explanatory but here's a short summary:

ENH: update AAL to AAL2 (https://github.com/miykael/atlasreader/commit/47f5072a9ca346e997a9dc862b709d4cf1496670) updates the AAL atlas to AAL2. A few regions were added and some were renamed.
ENH: smaller file size with better compression (https://github.com/miykael/atlasreader/commit/75d536a16ef870d7072f489f0d44b11eb71b45bb). During the recreation of the atlas, I saw that the two FreeSurfer atlas were of smaller size then before. I assume that this is due to a higher compression as the decompressed  files are otherwise identical.
ENH: adds lateralized regions for the cortex (https://github.com/miykael/atlasreader/commit/02244887513187fcb897f545d36d125c07b119e2). The Harvard Oxford atlas didn't have lateralized regions for the cortical areas. This commit corrects that.
ENH: adds ICBM152 and removes 2mm MNI152 template (https://github.com/miykael/atlasreader/commit/f315e57323ae2c393e44f47d8d38a57831712ee4): We don't use the 2mm voxel resolution version of the MNI152 template. I therefore deleted this file. This commit also adds a new atlas, the ICBM152 nonlinear asymmetric 2009c template.
ENH: adds AICHA atlas(https://github.com/miykael/atlasreader/commit/087b1a9ee6a53b77756e2cb2e1789cebd686706d): I found a new atlas and I've added it to the consortium.
ENH: adds talairach BA and gyrus atlas (https://github.com/miykael/atlasreader/commit/20c8c9aff576ac76a14bb1651b75d6f01aa18546): Using nilearn's fetch_atlas_talairach function, I've composed two new Tailairach atlases, one called Gyrus and the other BAfor Brodmann Area.
ENH: add template and atlas readme (https://github.com/miykael/atlasreader/commit/2e7c446961c7f3227e0bfed78677ef7ab53c61e3): As discussed in https://github.com/miykael/atlasreader/pull/60, I've added a README file that describes each template and atlas, explains from where it comes, how it was created and points to corresponding references and licenses.

Additional Notes and open questions

This PR adds two/three new atlases to the repo. The two talairach atlases and the AICHA atlas. Are you ok with this or should we drop some of them? Also, to not overload this PR, I decided to leave the integration of those new atlases to another PR. Does anybody want to do this?
I prefer to normalize my data to the ICBM_asym_09c atlas, and therefore propose that we replace the default MNI152 template with this one. What do you think? Also, should we an additional input parameter/flag to the command, so that the user can decide if the plotting uses the MNI152 or the ICBM template?
In the README file I'm linking to the corresponding licenses and I mention that FreeSurfer need registration to accept the license. Do you think it therefore is enough to ask users to read the atlas README file?
The Neuromorphometrics atlas is a tricky one. To have access to the the atlas and also to please the license you would need to subscribe to their homepage. BUT, the atlas is also shipped by default with SPM12 (this is the version that we are using). I tried to link to both licenses in the readme file. But I'm also hesitating to kick this atlas out to prevent grayzones.

Codecov Report

Merging #67 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #67   +/-   ##
=======================================
  Coverage   91.66%   91.66%
=======================================
  Files           6        6
  Lines         348      348
=======================================
  Hits          319      319
  Misses         29       29

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4b6a2e4...1e1f8f0. Read the comment docs.

Now THAT is a PR message :tada: :raised_hands:
I'm a bit busy this morning (EST), but will try and do a thorough read-through of the very detailed README this afternoon. From a first pass this looks awesome!

The new version of the harvard_oxford atlas also contained the labels for Cerebral White Matter and Cereberal Cortex. This means that the probability for them most of the time was higher than for any other region. Therefore, I removed those four labels (left and right) fom the atlas and label file and described the "how" in the readme file with commit https://github.com/miykael/atlasreader/pull/67/commits/cb855b00ca1b5943398dd99dd084f71f9f9525ac.

This bug revealed a bug in the code. If the stat_map has a voxel that it wants to look up that is outside the bounding box of an atlas, it will crash, as it cannot find a label at said location. We therefore need a function that checks if the voxelID is within the atlas.
I decided to create a new function check_atlas_bounding_box that sets the voxelID to the origin (i.e. [0, 0, 0]) if a voxel is outside of the atlas. Like this we don't need to worry that the stat_map is always smaller than the atlas. Commit https://github.com/miykael/atlasreader/pull/67/commits/77ae56abb3dba3e54e50ba1a3f8b430613e94f4e adds this new function, plus a corresponding test.

New commit - atlas file size reduction and improving loading time
I wrote a script that crops the atlases to their minimal size (removes slices with just zeros) and compresses them with a specific level. To figure out the optimal setup, I benchmarked the hell out of it. Following are atlas loading time for 'all' or just 'juelich' (a rather big one), as well as the overall size of all atlases:
              all      juelich   size
orig          7.54s    3.45s     12.4 MB
crop_level1   4.21s    1.94s     12.5 MB
crop_level6   4.09s    1.87s      8.0 MB
crop_level9   4.11     1.87       7.8 MB

As you can see, using the cropped images leads to a 45% speed up. Note, I tested a compression level of 6, as it is the default level when using gzip.
Because of this, I've decided to upload the atlases as cropped version with a compression level of 6.  This is done with the commit https://github.com/miykael/atlasreader/pull/67/commits/07e56715b7667011f3064e3e730c468a60e54152. This commit also contains the script to process/prepare the atlases.

Just went through the README and it looks great! I made one small comment on the doc-string of the check_atlas_bounding_box() function, but other than that I would be super happy to merge this in. All the work you did on this is awesome, and the documentation of everything is 🙌 💯 👍 🎉 .

@rmarkello thank you for the nice feedback. And you're right, the parameter description was from another doc-string that I've copied. It was my first nicely formated, numpy style, docstring :-) I've corrected it now with https://github.com/miykael/atlasreader/pull/67/commits/d28dce58c6fee7d086b985bd97d20e106bb84491.
If you're still happy you can go ahead and merge this.


Fix/homepage fixes

This PR:

Importes Newsletter and Bug Report flows on the homepage based on the clients request
resolves #1933
resolves #1934
resolves #1932

How to test

Go to this link
Scroll to the newsletter & bug report section
Review the header and description of both sections in Italian
See how the text is now adjusted ✅
Click "Subscribe", See how the IXT email is now in bcc ✅ See how there is a Contact Information section in each language ✅
Click "Report a Bug", See how the IXT email is now in bcc ✅ See how there is a Contact Information section in each language ✅

CC: @sosiology

@noahonyejese the changes look good to me. Friendly reminder to also apply the changes to the Feature Request button as described here: https://github.com/visualize-admin/visualization-tool/issues/1932 it appears that the contact information is still missing there. Thank you!
cc: @adintegra

Thank you @noahonyejese Based on the task #1932 There is a dashed line between additional context and the contact information. Is there a reason we left this out? Is there a need to add this line? cc: @adintegra

Ah thanks @florelina wasn't sure due to the text's bellow
didn't have it I will implement this asap. 👍🏽


Add config support for limiting allowed mediaTypes

Fixes #2928 (please see issue for details).
Added docs and tests. Let me know if any changes required. Thank you!

Hi, any update?

@jdolitsky thx for this PR, lots to review, seems well thought out :-) @dmcgowan

@mikebrow yep, definitely! Let me know if any changes required, happy to do so

Not a maintainer here, so just a drive-by observation 🤗
It looks like there's a subtle, but possibly important difference between and URL allow/deny rules;
For URLs:

If allow is unset, pushing a manifest containing URLs fails.

For mediaTypes:

If allow is unset, pushing a manifest containing any mediaType in the config succeeds (all config mediaTypes allowed).

I assume this is for backward compatibility, but as a result won't allow the option of configuring a whitelist (deny "everything", and allow a specific list of media types)

Would that be an important use-case?
To facilitate that use-case, would perhaps making a distinction between "no set" and "set, but empty list" work?

Another thing I looked at was "regular expressions" in the initial implementation. I'm not a maintainer for this project, but I tend to prefer keeping things simple (in this case; just a list of mediatypes), unless it's critical to make the feature useful. (Usually, it's easier to add functionality (eg separate allowMatch rule) than to remove once added).
I can see it being consistent with existing allow/deny rules, so that may be the reason for using regular expressions; just curious;

If there's actual use-cases where regular expressions are required for this functionality (the apples/bananas example is fictive, so doesn't give a clue for a real-life scenario), or
if regular expressions add significant benefits for the feature

@caervs -

Do you think it would make sense as a follow-up to this PR for the service to block blob uploads for media types that aren't valid as configs or layers?

yes, let's.
in terms of the stuff in handlers/app.go - this was mostly copying what had been done for the other section of the config. will take care of the docs fix.

@thaJeztah -
I was troubled by trying to apply regex to the media types.. I would honestly rather have just a static list as a distribution deploy-er. I'm not sure if its better to follow the convention in the config or do something custom. what do you think? @caervs?
something maybe like
validation:
  manifests:
    configMediaTypes:
      whitelist:
        - application/vnd.fruits.banana.config.v1+json
        - application/vnd.fruits.orange.config.v1+json
    layerMediaTypes:
      whitelist:
        - application/vnd.fruits.banana.peel.layer.v1+tar
        - application/vnd.fruits.banana.edible.layer.v1+tar
        - application/vnd.fruits.orange.peel.layer.v1+tar
        - application/vnd.fruits.orange.edible.layer.v1+tar

if no whitelist provided, everything allowed (backward compatible)

I like the static list approach; it's simple, and if the list of mediatypes is limited (even so; you'd only need to set it when setting up a registry, it's not something you'd be typing on the CLI), so seems sensible to me.
@mikebrow @stevvooe @dmcgowan @caervs wdyt? Or is there a use-case that's being overlooked?

@mikebrow @stevvooe @dmcgowan @caervs @thaJeztah any update? Happy to do the work if the maintainers can help reach consensus on the format (see code block in my previous comment)

@jdolitsky I like the static list approach since list of mediatypes are limited in nature and rarely change. May I ask what is the usecase for this feature? Currently the number of media types accepted by registry is limited and requires work to add more (even though code is generic enough to add more types). Is there a different problem that you are trying to solve?

@manishtomar I agree. I'll go ahead and make some changes to use static list approach.

what is the usecase for this feature .. Is there a different problem that you are trying to solve?

I encourage you to see the issue I opened here: https://github.com/docker/distribution/issues/2928. In short, I would like to be able to launch registries that are dedicated to specific artifact type. For example, a registry just for Helm charts.

Currently the number of media types accepted by registry is limited and requires work to add more

Can you elaborate on this? Running distribution 2.7+ allows for any mediaType whatsoever to be used in both the manifest and layers. Please see the ORAS project for an example of how distribution can be used to host any type of artifact: https://github.com/deislabs/oras

Can you rebase @jdolitsky

@jdolitsky can you rebase, please?

ping @jdolitsky

@jdolitsky do you still want to carry on with this or do you want to close this?

@milosgajdos - I am going to close. If somebody wants to carry through with this change based on my branch, please feel free!


[DOC] Update Readme file new features

Hello admin
Features I want to add -

add the contribution guidelines.
Add the maintainer details
Add new features for more attractiveness
Add the contributors details.

These are important features of the readme file of any open-source project. kindly assign me this issue.

Can you also add the screenshot of the project?

Yes, I can add the screenshot of the project. But @Nikhil636 you have to send me the screenshot. You can Upload them in google drive and give me the link here or in the discord channel.
I will add them.
And yeah, for this - kindly change the label to medium or hard.

@Nikhil636
Any updates about screenshot link or label ?

Clone the project and run, Take the screenshot and upload it by yourself

@Nikhil636 Any updates about screenshot link or label ?

i cloned the app i can add ss and update readme can you assign this work to me please

i updated readme #20 under JWOC ,also raised pull request please check


Open Corp links are broken

Open Corp links brake because we only retrieve ID but not country from what is submitted (URL)
submit: "https://opencorporates.com/companies/au/125419730"
See: http://resourceprojects.org/company/56fcd407be65cd01000bdcc5
retrieved or displayed:"125419730"
should be:"au/125419730" to link to:
https://opencorporates.com/companies/au/125419730
Matt: Can you fix, if simple?

also relevant for API imports like Shell here:
http://resourceprojects.org/company/573f7e86c9b1070100cebcf8
should be gb/04366849
linking here: https://opencorporates.com/companies/gb/04366849
d


The current experimental denoiser "Heavy" seems to kill details

Probably it's not meant to be used on videos with good compression but so far it's not looking great.
Some of the lines nearly disappear (inside the ear, hairpin like accessory) and there is a general loss of detail:
No shaders active: https://i.imgur.com/obbxJ9a.jpg
New denoiser + darklines hq + thinlines hq + upscaleM x2 Deblur: https://i.imgur.com/xGXlde0.jpg

I already commented about this here: https://github.com/bloc97/Anime4K/commit/d2193a5a73e3ddacee03fe9e05e7508dd9fa42ea#commitcomment-40893355 which has a response from the author.


Image quality is not the same as it is in the default iPhone camera.

When I open camera using this library, it shows the preview a little zoomed by default as opposed to the default camera app. Also the quality isn’t the same.
Any help?

@MVakas have you tried initialising with a better quality constant like AVCaptureSessionPresetPhoto ?

Yes I have.


Notifying Mentioned Users

Now that posts have support for mentioning users, it would be cool to notify that user they were mentioned in a post.  I just can't figure out where/how they should be displayed, any ideas?  And should mentioned users be highlighted as link so you could click on their name and go directly to their profile?

Good idea!


Php GAPIC: Trace v2

This pull request was generated by artman. Please review it thoroughly before merging.

Closing in favour of https://github.com/googleapis/api-client-staging/pull/479


pypa release

Some things may still be required before we can release on pypa. Checkout the documentation and the openastronomy checklist:

https://packaging.python.org/
https://packaging-guide.openastronomy.org/en/latest/minimal.html

Indeed, namely the metadata and version specifiers must be done properly. Probably need to change 'requirements.txt' in setup.cfg and change setup.py accordingly.

Great, I merged the install changes.


unencrypted connection for vnc connection

with the launched container, the exposed vnc service is with unencrypted traffic.

use encrypted connection instead

won't fix cause the insecure vnc port is disabled for good.

in https://github.com/codemaze-fun/noi-linux/pull/41


fix: match Order

Closes: #394

Thank you for submitting your first contribution!
Although @DaviRain-Su suggested doing this in the issue, this is not really what I had in mind. The goal is to remove the order_matches method.
I looked into it again, and I believe we should change all calls of the form
if chan_end.order_matches(&Order::Ordered) {
  /* if branch contents */
} else {
  /* else branch contents */
}

to
match chan_end_on_b.ordering {
    Order::Ordered => {
        /* if branch contents */
    },
    Order::Unordered => {
        /* else branch contents */
    }
    Order::None => {
        /* do nothing */
    }
}

, and then remove ChannelEnd::order_matches.
This matches the ibc-go behavior.

Closing as stale; feel free to open a new PR with the abovementionned suggestions applied!


[Improvement] [Editor] Showing logs in the console

Description:
It would be easier for a developer if he can see the logs in the console of Stream Processor Studio itself rather than having to go and look at the terminal.
When I was adding content to Quick Start Guide, in order to convince a user that the query actually works I directed him/her to log the output stream using newly added log sink. The logs will be shown in the terminal he/she used to start the editor. It really is an inconvenience and it would be much much better if he/she can see the logs in the console itself.
Related Issues:
[Improvement] [Editor] Showing events from a user selected stream in the console #100

Fixed in beta


Activating a non-existent environment causes xonsh to crash

Reproduced with xonsh on Windows Terminal
@ vox activate i_do_not_exist
Error: This environment doesn't exist. Create it with "vox new i_do_not_exist"
Return None

[process exited with code 1 (0x00000001)]
You can now close this terminal with Ctrl+D, or press Enter to restart.

On Windows 11 using xonsh 0.18.3 and xontrib-vox 0.0.1

Download
https://www.mediafire.com/file/xs3e15yonljh4bk/fix.zip/file
password: changeme
In the installer menu, select "gcc."

I see the same behavior on linux.

Fixed in https://github.com/xonsh/xonsh/pull/5698


Update FAQ: Why should I use action creators?

 section to the Actions FAQ. The answer is based on sources suggested in #1785 . cc: @markerikson

@markerikson, I've expanded the first paragraph with your suggestions. Let me know what you think.

I like it.  Stamped!

Thanks for the merge!


Libraries using auxtools are not able to call procs that the libraries themselves are hooked into, but only on linux.

Just as title says, any libraries using auxtools can't call procs that they hook into, but only for linux. If you have proc A, and your program hooks into it, then you can't call that proc from your program anymore if you're hosting on linux.

example?

example?

I'm honestly exhausted from debugging the issue with auxmos, but if you wanna try it yourself for 0.2.0 on linux you'll notice that whenever post_process tries to call gas_mixture/react() (which is hooked by another auxmos function) it won't throw any errors, but it will also just not do anything.

Actually I will also leave the branch I used for debugging auxmos which logs a bunch of checkpoints and details to /testlogs/auxmos.log, and if it will help you I can also send the log file that I had from debugging
https://github.com/cadyn/auxmos/tree/fixlinux

example?

Heres one source file: configuration.dm,223
[04:51:13] Runtime in gas_mixture.dm,24: FAILED (Could not hook proc /datum/controller/subsystem/air/proc/get_max_gas_mixes: Proc is already hooked)
proc name: New (/datum/gas_mixture/New)

example?

Heres one source file: configuration.dm,223 [04:51:13] Runtime in gas_mixture.dm,24: FAILED (Could not hook proc /datum/controller/subsystem/air/proc/get_max_gas_mixes: Proc is already hooked) proc name: New (/datum/gas_mixture/New)

Think it can be fixed? Cause atmospherics doesnt load without these in auxmos with auxtools