Generating components necessary for OLM bundle Creation

Goal
Be able to offer the AWS Controllers for Kubernetes via any Operator Lifecycle Manager-enabled cluster.
References

Operator Lifecycle Manager ("OLM"): https://github.com/operator-framework/operator-lifecycle-manager

Bundle Generation Docs: https://sdk.operatorframework.io/docs/cli/operator-sdk_generate_bundle/

Design
Summary
Being able to bundle a given controller just requires a few additional files be generated for each service. With these files, the operator-sdk generate bundle command can be called, with certain flags, generating an appropriate bundle/ directory and bundle.Dockerfile. This can then be offered via a catalog in OLM-enabled clusters.
High-level implementation
The code generation documentation describes the phased approach used by the aws-controllers-k8s project today. With this in mind, the proposed implementation would be an additional (and optional) phase which generates all assets necessary to feed the operator-sdk generate bundle command. Automation can then execute operator-sdk generate bundle.
What needs to be generated
From the perspective a given service's config directory, which lives in the generated service directory (e.g. ack-controller-k8s/services/s3/config, here's the breakdown of what needs to be changed or added:
key:
	m : existing file that needs modification
	+ : net-new file
	T : file needs templating/generation
	c : file can be copied statically

---
m   ├── controller/
m   │   ├── kustomization.yaml
+ c │   └── patch-user-env.yaml
+   ├── manifests/
+   │   ├── bases/
+ T │   │   └── SERVICE.clusterserviceversion.yaml
+ T │   └── kustomization.yaml
+   ├── samples/
+ T │   ├── SERVICE-CRD.yaml
+ T │   └── kustomization.yaml
+   └── scorecard
+       ├── bases/
+ c     │   └── config.yaml
+ c     ├── kustomization.yaml
+       └── patches/
+ c         ├── basic.config.yaml
+ c         └── olm.config.yaml

Descriptions by parent directory
These directories are within the ack-controllers-k8s/services/SERVICENAME/config/ directory, so the directory controllers/ would effectively mean ack-controllers-k8s/services/SERVICENAME/config/controllers/`.

controllers/
patch-user-env.yaml
	This allow a cluster administrator to prepare the controller namespace with credentials and controller configuration in a fail-safe way. Today, OLM-managed controller deployments will be reconciled on change which causes issues with credential-management implementations that inject authentication bits to the controller directly. Including this in the deployment manifest that is provided to OLM to install allows cluster administrators to make changes to the configuration and secrets without impacting the deployment directly.
kustomization.yaml
This already exists, but would need to have patchesStrategicMerge populated if the above patch is to be included.

manifests/
This directory can be renamed, as operator-sdk generate bundle accepts a flag that allows the user to change the path definition for finding these files.
bases/SERVICE.clusterserviceversion.yaml
The ClusterServiceVersion ("CSV") is the key asset that needs generation. This is a Kubernetes custom resource that contains a bulk of the metadata necessary for OLM to "see" and be able to deploy the service controller.
This will need a template.
kustomization.yaml
The operator-sdk generate bundle command operates on a series of inputs (YAML manifests) that help render the final CSV. That final CSV is then bundled for a given release.
It takes, as input, a series of YAML manifests (e.g. the controller deployment yaml, the contents of the samples/ directory, etc). This kustomization is in place to facilitate calling kustomize build on a single target, which in turn calls downstream targets, resulting in all the necessary inputs for operator-sdk generate bundle to complete.

samples/
SERVICE-CRD.yaml
This is a sample custom resource yaml for the given service (e.g. for S3, this is a "Bucket" YAML manifest).  Projects built with Kubebuilder get this out of the box when create-ing APIs.
I don't believe the naming convention here to be important to operator-sdk, but is more for user visibility.
Services with multiple custom resources can have an individual file per-resource, or have all resources in a single file with YAML separators. Either appeared to work in testing.
kustomization.yaml
This is in place to ensure that calling kustomize build config/manifests properly includes this directory. Projects built with OperatorSDK get this by default. This file just needs to include the name of the SERVICE-CRD.yaml file listed under the resources section.

scorecard/
This parent directory ships basic tests with every operator built using OperatorSDK.
At the time of writing, the AWS Controllers for Kubernetes don't have any Scorecard tests to include so this directory can effectively be a "copy/paste" from what OperatorSDK scaffolds for new projects
Should ACK want to include Scorecard tests in the future, additional patches can be configured. Documentation on generating custom tests is found here.
bases/config.yaml
This is a blank Scorecard configuration. The patches in this kustomize directory patch in test specifications.
patches/basic.config.yaml
This is the included basic test suite which makes sure that all custom resources have a spec block. See definition.
patches/olm.config.yaml
This is the included OLM-specific test which ensures that the resulting assets have niceties for usage within OLM. See definition.
kustomization.yaml
Similar to samples/kustomization.yaml. this ensures that this directory is included when calling kustomize build config/manifests, as well as sets up the bases/patches relationship present in this directory.

Hi folks, just want to touch base here and make sure the implementation details manifested here are sound. If so, I'll do some testing and work towards a PR!

Hi folks, just want to touch base here and make sure the implementation details manifested here are sound. If so, I'll do some testing and work towards a PR!

@komish yes, the steps look proper. We're just in the process of splitting the monorepo into multiple source repositories, one for each service controller (please see #666). Hoping to get that merged today. You would want to make most of your changes to the github.com/aws-controllers-k8s/code-generator repository.

Understood, thanks @jaypipes ! I'll look to start getting spun up probably early next week.


0.8.5 release

I was wondering if there will be a new release since phpstan/phpstan is now at 0.8.5.

Hi,
let me assure you that for the PHAR version, 0.8.5 wouldn't bring anything new when compared to 0.8.4, it solved a specific issue when installed phpstan/phpstan via Composer.
But of course I'd publish PHPStan PHAR 0.8.5, but I encountered a specific build problem. I aim to solve this for 0.9 release which is coming soon. Once 0.9 is out, I will return to publishing PHARs as well.
Thank you for understanding.

Hi @ondrejmirtes, thanks for the quick reply. Good to know there will be a 0.9 release. I was only asking because there was a small fix in the 0.8.5 version. But it can wait till 0.9. Looking forward to the new release :)

I was able to solve this (reference: https://github.com/Ocramius/PackageVersions/issues/48) so the 0.8.5 PHAR is published.


”Breznița-Ocol” is listed in the drop-down menu

Description:
"Breznița-Ocol" is listed in the drop-down menu alongside its corresponding county and commune, according to the law.
Precondition:
The website is up an running.

Step 1
Type "Breznița-Ocol" in the search bar.

Expected results
The location is listed in the drop-down menu as "Comuna Breznița-Ocol, MEHEDINȚI".


URL module?

Most of the cloud providers have a (http based) metadata api for nodes to fetch some information about themself.
It would be nice if converge would have a url module to fetch those resources and be able to parse the response (mostly JSON afaik) into usable variables.

This feature has been discussed and it's something we'd definitely like to have support for in some form.  Rather than a URL module we're likely to implement a more general solution for getting data from external data sources, including both cloud provider metadata as well as info from things like consul, etcd, vault, etc.
It doesn't look like we have an existing issue outlining this exact use-case though so I'm going to go ahead and tag it so we can take a look soon.


Use offset when unprojecting

Another bug in unproject ... when we use an asymmetric depth_range then we need to apply the position of the transformation matrix. Previously, we have ignored it.
@almarklein I also noticed some surprising behavior for matrix_make_orthogonal in how it handles the far and near plane. Could you have a look at the comment and see what makes sense for us?

Okay, I made a proposal for the docs in this PR based on our discussion (thanks for the clarification @almarklein @Korijn ). I've also updated the explicit unit test to use the example that @Korijn made earlier, which I think gives it more context than the generic one that I had there before.
I think I'm happy with this one and will address future points (if any) in upcoming PRs.

I made a proposal for the docs in this PR based on our discussion

I'm not sure if introducing another coordinate frame will avoid confusion. The args are distances, not points that need to live somewhere. Perhaps it suffices to add a small note to the near and far plane arg docs, that a positive distance is defined as being in the +z direction?

The args are distances, not points that need to live somewhere.

@almarklein But parameters can be negative, and both left and bottom typically are. Strictly speaking, distances can never be negative, so we are actually not specifying distances, but rather points along the respective coordinate axes.

I'm not sure if introducing another coordinate frame will avoid confusion.

I agree that it adds complexity to the docs, but I think it is mostly because we are now pointing out the complexity that is inherent in the function rather than us adding complexity artificially.
The problem I'm trying to solve is this: For debugging, I wanted to create an orthographic camera that has a frustum that matched a particular box in world space. What values do I need to plug into matrix_make_orthographic to get such a projection? I checked the docs and found no explicit mention of the reference frame for parameters. Next, I inspected places where it was used and it looked like the parameters are in local space, which combined with our conventions document (cameras view along negative Z) led me to the wrong conclusion about which numbers go into near and far. This lead me down the rabbit hole that ultimately ended up in this discussion.
What would have helped me avoid all this is some kind of note on how each value relates to the frustum's position in local space; hence me adding this note.

What would have helped me avoid all this is some kind of note on how each value relates to the frustum's position in local space; hence me adding this note.

Ok, fair enough 😄


chore: Update error returned by cloudprovider.Create()

Fixes #N/A
Description
Update error returned by cloudprovider.Create() to use upstream CreateError.
How was this change tested?
make test
Does this change impact docs?

[ ] Yes, PR includes docs updates
[ ] Yes, issue opened: #
[x] No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>4
Details

2 of 3 (66.67%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 82.805%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

pkg/cloudprovider/cloudprovider.go
2
3
66.67%

Totals

Change from base Build<PHONE_NUMBER>6:
0.0%

Covered Lines:
5673

Relevant Lines:
6851

💛  - Coveralls


增量预训练时候，用大一些的模型（Yi-34B）在8*80G服务器上全量训练，无论-block_size 设置多少都报显存不足

Describe the bug
run_pt脚本如下：
CUDA_VISIBLE_DEVICES=2,3 torchrun --nproc_per_node 2 pretraining.py
--model_type auto
--model_name_or_path ../Yi-34B-Chat
--train_file_dir ./data/pretrain
--validation_file_dir ./data/pretrain
--per_device_train_batch_size 2
--per_device_eval_batch_size 2
--do_train
--do_eval
--use_peft False
--seed 42
--max_train_samples 10000
--max_eval_samples 10
--num_train_epochs 0.5
--learning_rate 2e-4
--warmup_ratio 0.05
--weight_decay 0.01
--logging_strategy steps
--logging_steps 10
--eval_steps 100
--evaluation_strategy steps
--save_steps 500
--save_strategy steps
--save_total_limit 13
--gradient_accumulation_steps 1
--preprocessing_num_workers 10
--block_size 16
--group_by_length True
--output_dir outputs-pt-Yi-v3
--overwrite_output_dir
--ddp_timeout 30000
--logging_first_step True
--target_modules all
--torch_dtype bfloat16
--bf16
--device_map auto
--report_to tensorboard
--ddp_find_unused_parameters False
--gradient_checkpointing True
--cache_dir ./cache
报错如下：
Grouping texts in chunks of 16 (num_proc=10): 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10000/10000 [00:00<00:00, 17065.83 examples/s]
Grouping texts in chunks of 16 (num_proc=10): 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10000/10000 [00:00<00:00, 15229.81 examples/s]
2024-01-23 14:25:20.855 | DEBUG    | __main__:main:592 - Num train_samples: 10000
2024-01-23 14:25:20.856 | DEBUG    | __main__:main:593 - Tokenized training example:
2024-01-23 14:25:20.857 | DEBUG    | __main__:main:594 - 曲匹地尔片的用法用量? 注意：同种药品可由于
2024-01-23 14:25:20.858 | DEBUG    | __main__:main:606 - Num eval_samples: 10
2024-01-23 14:25:20.858 | DEBUG    | __main__:main:607 - Tokenized eval example:
2024-01-23 14:25:20.858 | DEBUG    | __main__:main:608 - 曲匹地尔片的用法用量? 注意：同种药品可由于
2024-01-23 14:25:20.937 | DEBUG    | __main__:main:592 - Num train_samples: 10000
2024-01-23 14:25:20.937 | DEBUG    | __main__:main:593 - Tokenized training example:
2024-01-23 14:25:20.938 | DEBUG    | __main__:main:594 - 曲匹地尔片的用法用量? 注意：同种药品可由于
2024-01-23 14:25:20.939 | DEBUG    | __main__:main:606 - Num eval_samples: 10
2024-01-23 14:25:20.939 | DEBUG    | __main__:main:607 - Tokenized eval example:
2024-01-23 14:25:20.940 | DEBUG    | __main__:main:608 - 曲匹地尔片的用法用量? 注意：同种药品可由于
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15/15 [00:16<00:00,  1.12s/it]
2024-01-23 14:25:38.385 | INFO     | __main__:main:692 - Fine-tuning method: Full parameters training
trainable params:<PHONE_NUMBER>8 || all params:<PHONE_NUMBER>8 || trainable%: 100.0
2024-01-23 14:25:38.423 | INFO     | __main__:main:723 - *** Train ***
2024-01-23 14:25:38.849 | DEBUG    | __main__:main:724 - Train dataloader example: {'input_ids': tensor([[39616,  6619, 60445, 60790,   102,  2168, 50673,  6619,  2596, 59614,
         59599, 23805, 35190,    79,    85,    77],
        [ 1610, 59632, 59988, 60937,  7087, 59642,  7632, 60130, 30719,  6890,
           102, 25092, 59590,  7353,  9435, 43310]], device='cuda:1'), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]], device='cuda:1'), 'labels': tensor([[39616,  6619, 60445, 60790,   102,  2168, 50673,  6619,  2596, 59614,
         59599, 23805, 35190,    79,    85,    77],
        [ 1610, 59632, 59988, 60937,  7087, 59642,  7632, 60130, 30719,  6890,
           102, 25092, 59590,  7353,  9435, 43310]], device='cuda:1')}
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15/15 [00:18<00:00,  1.26s/it]
2024-01-23 14:25:40.251 | INFO     | __main__:main:692 - Fine-tuning method: Full parameters training
trainable params:<PHONE_NUMBER>8 || all params:<PHONE_NUMBER>8 || trainable%: 100.0
Detected kernel version 4.18.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
2024-01-23 14:25:40.293 | INFO     | __main__:main:723 - *** Train ***
2024-01-23 14:25:40.724 | DEBUG    | __main__:main:724 - Train dataloader example: {'input_ids': tensor([[ 7995,   101,  2480,  5060,  2986,  3603,   101, 59961, 61232,  1738,
          1138,  6440, 59635,  2665,  2893,   101],
        [59632, 30421, 60764, 60928,  3898, 59678, 59907, 60602, 60279, 10894,
         59664, 61598, 60167, 36358,   101,  1913]], device='cuda:0'), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]], device='cuda:0'), 'labels': tensor([[ 7995,   101,  2480,  5060,  2986,  3603,   101, 59961, 61232,  1738,
          1138,  6440, 59635,  2665,  2893,   101],
        [59632, 30421, 60764, 60928,  3898, 59678, 59907, 60602, 60279, 10894,
         59664, 61598, 60167, 36358,   101,  1913]], device='cuda:0')}
Traceback (most recent call last):
  File "pretraining.py", line 767, in <module>
    main()
  File "pretraining.py", line 728, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/transformers/trainer.py", line 1537, in train
Traceback (most recent call last):
  File "pretraining.py", line 767, in <module>
    return inner_training_loop(
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/transformers/trainer.py", line 1672, in _inner_training_loop
    main()
  File "pretraining.py", line 728, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/transformers/trainer.py", line 1537, in train
    model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/accelerate/accelerator.py", line 1227, in prepare
    return inner_training_loop(
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/transformers/trainer.py", line 1672, in _inner_training_loop
    result = tuple(
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/accelerate/accelerator.py", line 1228, in <genexpr>
    self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/accelerate/accelerator.py", line 1104, in _prepare_one
    model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/accelerate/accelerator.py", line 1227, in prepare
    return self.prepare_model(obj, device_placement=device_placement)
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/accelerate/accelerator.py", line 1355, in prepare_model
    result = tuple(
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/accelerate/accelerator.py", line 1228, in <genexpr>
    model = torch.nn.parallel.DistributedDataParallel(
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 809, in __init__
    self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/accelerate/accelerator.py", line 1104, in _prepare_one
    self._ddp_init_helper(
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 1098, in _ddp_init_helper
    return self.prepare_model(obj, device_placement=device_placement)
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/accelerate/accelerator.py", line 1355, in prepare_model
    self.reducer = dist.Reducer(
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 64.05 GiB. GPU 0 has a total capacty of 79.33 GiB of which 14.34 GiB is free. Including non-PyTorch memory, this process has 64.98 GiB memory in use. Of the allocated memory 64.17 GiB is allocated by PyTorch, and 323.50 KiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
    model = torch.nn.parallel.DistributedDataParallel(
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 809, in __init__
    self._ddp_init_helper(
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 1098, in _ddp_init_helper
    self.reducer = dist.Reducer(
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 64.05 GiB. GPU 1 has a total capacty of 79.33 GiB of which 14.34 GiB is free. Including non-PyTorch memory, this process has 64.98 GiB memory in use. Of the allocated memory 64.17 GiB is allocated by PyTorch, and 323.50 KiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2024-01-23 14:25:45,100] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 3264173) of binary: /home/centos/anaconda3/envs/cpt/bin/python
Traceback (most recent call last):
  File "/home/centos/anaconda3/envs/cpt/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/torch/distributed/run.py", line 806, in main
    run(args)
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/torch/distributed/run.py", line 797, in run
    elastic_launch(
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
pretraining.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2024-01-23_14:25:45
  host      : host188
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 3264174)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-01-23_14:25:45
  host      : host188
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 3264173)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

另外请问一下： --block_size 的作用是什么呢，通过batch_size可以调解显存使用，为啥还需要这个参数呢？

CUDA_VISIBLE_DEVICES=0,1,2,3 python pretraining.py

CUDA_VISIBLE_DEVICES=0,1,2,3 python pretraining.py

已经设置了，依然报的同样的错误；
Traceback (most recent call last):
  File "pretraining.py", line 767, in <module>
    main()
  File "pretraining.py", line 728, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/transformers/trainer.py", line 1537, in train
    return inner_training_loop(
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/transformers/trainer.py", line 1672, in _inner_training_loop
    model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/accelerate/accelerator.py", line 1227, in prepare
    result = tuple(
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/accelerate/accelerator.py", line 1228, in <genexpr>
    self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/accelerate/accelerator.py", line 1104, in _prepare_one
    return self.prepare_model(obj, device_placement=device_placement)
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/accelerate/accelerator.py", line 1355, in prepare_model
    model = torch.nn.parallel.DistributedDataParallel(
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 809, in __init__
    self._ddp_init_helper(
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 1098, in _ddp_init_helper
    self.reducer = dist.Reducer(
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 64.05 GiB. GPU 1 has a total capacty of 79.33 GiB of which 11.05 GiB is free. Process 2993049 has 3.10 GiB memory in use. Including non-PyTorch memory, this process has 65.17 GiB memory in use. Of the allocated memory 64.17 GiB is allocated by PyTorch, and 323.50 KiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Traceback (most recent call last):
  File "pretraining.py", line 767, in <module>
    main()
  File "pretraining.py", line 728, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/transformers/trainer.py", line 1537, in train
    return inner_training_loop(
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/transformers/trainer.py", line 1672, in _inner_training_loop
    model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/accelerate/accelerator.py", line 1227, in prepare
    result = tuple(
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/accelerate/accelerator.py", line 1228, in <genexpr>
    self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/accelerate/accelerator.py", line 1104, in _prepare_one
    return self.prepare_model(obj, device_placement=device_placement)
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/accelerate/accelerator.py", line 1355, in prepare_model
    model = torch.nn.parallel.DistributedDataParallel(
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 809, in __init__
    self._ddp_init_helper(
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 1098, in _ddp_init_helper
    self.reducer = dist.Reducer(
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 64.05 GiB. GPU 2 has a total capacty of 79.33 GiB of which 14.15 GiB is free. Including non-PyTorch memory, this process has 65.17 GiB memory in use. Of the allocated memory 64.17 GiB is allocated by PyTorch, and 323.50 KiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Traceback (most recent call last):
  File "pretraining.py", line 767, in <module>
    main()
  File "pretraining.py", line 728, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/transformers/trainer.py", line 1537, in train
    return inner_training_loop(
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/transformers/trainer.py", line 1672, in _inner_training_loop
    model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/accelerate/accelerator.py", line 1227, in prepare
    result = tuple(
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/accelerate/accelerator.py", line 1228, in <genexpr>
    self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/accelerate/accelerator.py", line 1104, in _prepare_one
Traceback (most recent call last):
  File "pretraining.py", line 767, in <module>
    return self.prepare_model(obj, device_placement=device_placement)
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/accelerate/accelerator.py", line 1355, in prepare_model
    model = torch.nn.parallel.DistributedDataParallel(
main()  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 809, in __init__

  File "pretraining.py", line 728, in main
    self._ddp_init_helper(
train_result = trainer.train(resume_from_checkpoint=checkpoint)  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 1098, in _ddp_init_helper

  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/transformers/trainer.py", line 1537, in train
    self.reducer = dist.Reducer(
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 64.05 GiB. GPU 3 has a total capacty of 79.33 GiB of which 14.24 GiB is free. Including non-PyTorch memory, this process has 65.07 GiB memory in use. Of the allocated memory 64.17 GiB is allocated by PyTorch, and 323.50 KiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
    return inner_training_loop(
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/transformers/trainer.py", line 1672, in _inner_training_loop
    model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/accelerate/accelerator.py", line 1227, in prepare
    result = tuple(
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/accelerate/accelerator.py", line 1228, in <genexpr>
    self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/accelerate/accelerator.py", line 1104, in _prepare_one
    return self.prepare_model(obj, device_placement=device_placement)
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/accelerate/accelerator.py", line 1355, in prepare_model
    model = torch.nn.parallel.DistributedDataParallel(
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 809, in __init__
    self._ddp_init_helper(
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 1098, in _ddp_init_helper
    self.reducer = dist.Reducer(
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 64.05 GiB. GPU 0 has a total capacty of 79.33 GiB of which 14.24 GiB is free. Including non-PyTorch memory, this process has 65.07 GiB memory in use. Of the allocated memory 64.17 GiB is allocated by PyTorch, and 323.50 KiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2024-01-23 15:22:18,696] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 3272628) of binary: /home/centos/anaconda3/envs/cpt/bin/python
Traceback (most recent call last):
  File "/home/centos/anaconda3/envs/cpt/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/torch/distributed/run.py", line 806, in main
    run(args)
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/torch/distributed/run.py", line 797, in run
    elastic_launch(
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/centos/anaconda3/envs/cpt/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
pretraining.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2024-01-23_15:22:18
  host      : host188
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 3272629)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
  time      : 2024-01-23_15:22:18
  host      : host188
  rank      : 2 (local_rank: 2)
  exitcode  : 1 (pid: 3272630)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
  time      : 2024-01-23_15:22:18
  host      : host188
  rank      : 3 (local_rank: 3)
  exitcode  : 1 (pid: 3272631)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-01-23_15:22:18
  host      : host188
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 3272628)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================


Handle cases of bad parent/child modelling

Cases where a resource (i.e. subnets) is modelled as both a child resource and as a property of the parent resource. This effectively tricks what-if into thinking the resource exists/could exist in both places because the VNET will return subnet info in it's GET response AND we can GET the resource directly as a child resource.

Customer reported examples:

#60
#46
#128?

https://github.com/Azure/arm-template-whatif/issues/140


chore: more explicit support link to discourse

Changed the description for the discourse link to a little more forceful wording. I also suggest to create a discourse label that has some description with a link to the discourse to  brush off people to the discourse.

This might make it easier to "unceremoniously" close issues by adding a label and moving the conversation to the discourse.

When I look at https://github.com/gohugoio/hugo/issues/new/choose my eyes don't make it past the first two entries. Massaging the text might make a difference, but probably not much of a difference.
Although you can change the sort order of the issue templates by changing the file name (title remains the same), the entries created by the config.yml file always float to the bottom; that's unfortunate.
Additional thoughts:

The H in GitHub should be uppercase.
Uppercase text doesn't get my attention; it's just annoying.
I wonder if the word Support makes people think of a formal support organization (paid, multiple tiers, phone trees, etc.).
The word "Issues" is confusing; the user is already on the Issues tab. Maybe something like Problems, Questions, and Troubleshooting?
I'd leave out the last two sentences.


[CARBONDATA-1614][Streaming] Show file format for segment

Show file format for segment, it is useful in case of streaming table

[X] Any interfaces changed?
No

[X] Any backward compatibility impacted?
No

[X] Document update required?
Yes, show segment output need to be changed

[X] Testing done
Testcase added

[X] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
NA

Build Failed  with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/1130/

SDV Build Success , Please check CI http://<IP_ADDRESS>:8080/job/ApacheSDVTests/1722/

Build Failed  with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/1145/

SDV Build Fail , Please check CI http://<IP_ADDRESS>:8080/job/ApacheSDVTests/1731/

SDV Build Fail , Please check CI http://<IP_ADDRESS>:8080/job/ApacheSDVTests/1735/

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/1158/

Build Failed  with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/1246/

retest this please

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/1249/

LGTM


test functions in lexemes.py

Could you please provide any description for the issue. Thanks!

Hey, take a look at Pg 12 and Pg 20 in the specification
https://docs.google.com/document/d/1yp_96UXcwNO49cktnHtT61iNmTO0RgrSQukuNYqACpM/
Take a look at:
https://github.com/WLCG-Lightweight-Sites/simple_grid_yaml_compiler/tree/master/compiler
It defines two functions

get_repo_list
parse_for_variable_hierarchies

get_repo_list parses the site_level_config_file line by line and extracts some_url from 'repository_url: some_url' wherever it appears.
We need to write a test to validate different scenarios (types of url's) that can be successfully extracted by this function. The tests reside in :https://github.com/WLCG-Lightweight-Sites/simple_grid_yaml_compiler/blob/master/tests/tests_lexemes.py
Feel free to delete the existing test case as it is outdated.
You'd have to install pytest etc. in your virtual environment and include them in the requirements.txt file when you make a PR.
The second function, parse_for_variable_hierarchies, process the from keyword (mentioned in the specification). It looks through the site_level_config_file recursively for instances where the from keyword appears. It generally appears in the following format
var2: &var2 value_2
var1: &var1
 __from__: *var2

Take a look at: https://github.com/WLCG-Lightweight-Sites/wlcg_lightweight_site_ce_cream/blob/master/default-data.yaml
The function should reduce the output to
var1: value2

We gotta write a unit test to check this functionality in the same test_lexemes.py file mentioned above. Let me know if more info is needed :) .

The code block
if keyword in data:
    data = data[keyword]
    return parse_for_variable_hierarchies(data, keyword)

Isn't this redundant? Because the first if statement doesn't allow any datatype other than CommentedSeq and CommentedMap and there is specific code blocks that handles either of them. This code block at line 23, assumes the data's datatype to be a dict which isn't in case of CommentedSeq.

The from keyword, by definition, has to be the only child of a
Dict/CommentedMap. That's because the assignment operation is one to one
mapping rather than a one to many mapping. Therefore, if a CommentedSeq
contains a from keyword, the case should either not be processed or it
should throw an exception. In the other case i.e. if the user specified a
from keyword in a CommentedSeq, the data = data[keyword] line will
throw an exception, except for the case that the from referred to a
variable that represented a number and the ConnectedSeq had an entry
corresponding to that index.
Regarding the redundancy, I believe it would exist if we parse through the
data more than once during the execution of the function. However, when we
check for the keyword in 'if keyword in data:', there is either an
exception thrown as described above or the next recursive iteration starts.
If the keyword is not present in data, we start the next recursive
iteration for all of the child objects. (that's what the being handled by
the for loops that follow later). Sure, you could move the if statement
inside the loop for CommentedMap and check if the child of the CommentedMap
has a key == keyword for every child. Assuming, python's implementation for
searching a key in a dict is efficient already, it must be safe to stay
away from adding a search step for every child object in the loops towards
the end of the function. I could be wrong here as I have not read about how
python core has implemented the search. Note that we are not dealing with
data big enough to generate the need to investigate optimization all of the
recursive aspects. But if you see a significant advantage in an alternate
approach, I'd be super interested to hear. :) Could you also give an
example for the same, if possible :)
On Sat, Mar 30, 2019 at 8:26 AM Vighnesh SK<EMAIL_ADDRESS>wrote:

The code block
https://github.com/WLCG-Lightweight-Sites/simple_grid_yaml_compiler/blob/5bdab27f9bc806fbfc9e1795bc375458e3728f85/compiler/lexemes.py#L23
if keyword in data:
data = data[keyword]
return parse_for_variable_hierarchies(data, keyword)
Isn't this redundant? Because the first if statement doesn't allow any
datatype other than CommentedSeq and CommentedMap and there is specific
code blocks that handles either of them. This code block at line 23,
assumes the data's datatype to be a dict which isn't in case of
CommentedSeq.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/WLCG-Lightweight-Sites/simple_grid_yaml_compiler/issues/25#issuecomment-478216314,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFofQhW0VOgAARMbkyMTWnnwO5SKkAErks5vbxG1gaJpZM4brVQp
.

--
Regards
Mayank Sharma

Thanks for the explanation, I had trouble figuring this out because I wasn't aware of the improbable edge case of of CommentedSeq that has the keyword to match in it.

The from keyword, by definition, has to be the only child of a
Dict/CommentedMap. That's because the assignment operation is one to one
mapping rather than a one to many mapping. Therefore, if a CommentedSeq
contains a from keyword, the case should either not be processed or it
should throw an exception. In the other case i.e. if the user specified a
from keyword in a CommentedSeq, the data = data[keyword] line will
throw an exception, except for the case that the from referred to a
variable that represented a number and the ConnectedSeq had an entry
corresponding to that index.

This surely confirms the test condition I assumed in #34


ULID form belongs_to

Builds on #145 . Same as what that PR did for :binary_id , this PR makes Ecto.ULID work, in terms of loading belongs_to form select options.

Hi,
Thanks for your contribution!
Is this still required and works on the latest version?
Gavin.

@ghenry yes it's still needed, yes it works on the latest version


Skill creation error

Hi, after getting the Arn for the lambda function at aws I receive the error. {"error":"Skill creation failed"} in https://kodiconnect.kislan.sk/alexa-skill/create, my device is displayed as connected in the site and In the Kodi application itself. Any support will be appreciated...

Can you please get back to the tutorial page, and try the new flow? I've had to update it, since Amazon made a breaking change, and it stopped working

Hi Tomas, I've checked the new tutorial and redo the steps, but now I receive an time out error  "{"error":"Timed out"} " in https://kodiconnect.kislan.sk/alexa-skill/update after clicking "Link Lambda" with my arn function

Testing and working. Can now create the skill.

Even if you try it multiple times?

I got the same error:
{
"error": "Skill update timeout"
}
and yes I tried multiple times.

Now after trying even more different things I managed to get through. The key for me was to select the ARN us-east-1 even though I live in the center of europe because I use alexa with English(US). on the us-east-1 i actually saw the alexa smart home entry, unlike in the others where I only saw alexa iot kit. (does that matter?). Now I just failed to find the skill in my alexa skills :(


Telemetry message should not be displayed if GAUGE_TELEMETRY_ENABLED

From @sguptatw on August 1, 2018 8:43
Expected behavior
If the env var GAUGE_TELEMETRY_ENABLED is explicitly set, this means the user has explicitly decided about telemetry. Hence the message is not to be shown when the env value is set.
Steps to replicate

Delete the gauge configuration folder
Set the GAUGE_TELEMETRY_ENABLED value to false
Install gauge
gauge version

The telemetry message is displayed

Version
Gauge version: 1.0.1.nightly-2018-07-31
Commit Hash: 4ae7c9a

Related to https://github.com/getgauge/gauge/issues/1121
Copied from original issue: getgauge/gauge-csharp#146

Which OS?

Had a typo in env var. Working as expected.


Better ansible.cfg

There's plenty of ways to make ansible faster, I recommend to look into https://github.com/AdopteUnOps/rancher-ansible/blob/master/ansible.cfg to have ideas to improve our current configuration.

@framiere we've started to make changes regarding some of these enhancements.  In particular, we just pushed a new release which contains pipelining, which should speed things up considerably.

@framiere going to close this out since I haven't heard back from you in some time.  If you have further suggestions please feel free to open a new issue.


Remove CODE_OF_CONDUCT.md

Can one of the admins verify this patch? (see README)

I think this file would best be removed by @gavinking since he was the one who added it.

Nooooo!! I loved that file since the moment I found it!


Setting language via http.request

Hi,
I think your project is great and I like the Docker solution very much. however, I just can't get it to change the language in the header at the moment.
Would you have a tip for me how I could change this?
I already tried with Header.Add and writer.CreateFormField("languages") without success.
Thanks a lot
`file, _ := os.Open(datei)
defer file.Close()
body := new(bytes.Buffer)
writer := multipart.NewWriter(body)
part2, err := writer.CreateFormField("languages")
part, _ := writer.CreateFormFile("file", filepath.Base(file.Name()))

if err != nil {
	fmt.Printf("Error creatformfield: %s", err)
}

part2.Write([]byte("deu"))
io.Copy(part, file)
writer.Close()

r, _ := http.NewRequest("POST", "http://<IP_ADDRESS>:9080/file", body)
//r.Header.Add("languages", "deu")
//r.Header.Set("languages", "deu")
r.Header.Add("Content-Type", writer.FormDataContentType())

I figured it out and it was simpler than I thought. I leave the solution here in case someone also has the problem :)
`q := r.URL.Query()
q.Add("languages", "deu")
// assign encoded query string to http request
r.URL.RawQuery = q.Encode()`


Improve tag definition

The current code requires that tags must be defined as:
[Dict("key" => "foo", "value" => "bar"), Dict("key" => "missy", "value" => "gala")]

but what about making all the logic behind to allow the user define the tags as:
# as an array of pairs
["foo" => "bar", "missy" => "gala"]

# as a key-value singular dict
Dict("foo" => "bar", "missy" => "gala")

A solution was implemented in #59 .


Missed detection of resource nodes on small planets

Originally mentioned by GhostReaper on Discord, replicated by me.
Seed: 56549442
System: Vouskiaho (starter)
Planet: I-I
Radius: 100 (indicated by RuntimeUnityEditor)
Severity: Significant, but can be worked around
Sometimes, resource nodes will not be detected by miners, so that even if they are in the mining range they are not actually "reached" by the miner. In an extreme case, this means a miner cannot be built, citing "needs resources" even though they are present.

Rotating or repositioning the miner may cause it to update and reach those otherwise unreachable nodes. Snapping-to-grid (rotation OR placement) does not inherently result in a better result.

The problem in this case gets worse the further you move from the poles. The first instance of it I captured was with resource nodes on the 11th latitude band (screenshot: where the shift-to-accelerate tip starts). Note the error box is unrelated, it's from the F12 debugging tools.

Even at the equator of this planet, the problem was present. It is unaffected by the presence or lack of foundations.

The problem seems consistently replicable once you hit the affected latitudes. It happened with 0 of 4 polar veins tested, and with 3 of 3 with latitudes starting at the 11th band.

My guess on that is that because of the curve of the planet... the miner don't detect the nodes because the base of the ressource node is a bit lower than the detection line

I like that theory but I think the first two screenshots debunk it. Screenshot two shows the left two nodes being missed, so apparently the right two nodes are at the right height and the left ones are not. But then when we move further left of those two nodes (screenshot 1) they're still not visible.

It's not the curvature as far as I can tell, I've moved that collider around and up and down and still can't target the veins that arent targetable (until I move the miner over some threshold, that then changes which nodes are targetable)

Fixed


Adjust to YQL shutdown

According to @ydn (Yahoo Developer Network) on Twitter:

On Jan. 3, 2019, YQL service at http://query.yahooapis.com  will be retired. YQL based services that use http://query.yahooapis.com , including users of http://datatables.org , will no longer operate. Yahoo Weather API users see the tweet below for info about continuing your service.

This is exactly the endpoint that we use for fetching the XML content of RSS feeds in podcast.js, so we will have to remove that code and:

find an alternative service that can fetch arbitrary XML for us
figure out how to enable CORS in GitHub (or a custom domain)
host an app, such as an App Engine app, which can issue arbitrary requests
leave it to customers to run their own versions of the app themselves only (which we already have a script for today)

According to a Stack Overflow question for the same shutdown, there are at least 2 potential alternatives for YQL, as described in the top two answers, namely:

https://api.rss2json.com (which also includes RSS to JSON conversion, as per the name), and
https://cloudquery.t9t.io/ (also open source)

For example, we can get the headings of an RSS feed using rss2json.com's sample code:

Alternatively, using CloudQuery's URL pattern with * as the selector, we can get the entire RSS feed as an XML string, which we can then parse ourselves using an existing xml2json function:
https://cloudquery.t9t.io/query?url=https%3A%2F%2Fnews.ycombinator.com/rss&selectors=*

Unfortunately, this URL appears to be heavily rate-limited, so it may not be easily usable for our purpose.

Looks like cors.io is down; https://cors.io/?https://news.ycombinator.com/rss returns 404.
However, https://api.rss2json.com/v1/api.json?rss_url=https://news.ycombinator.com/rss returns what we need, with access-control-allow-origin: * header that will make this easy. However, rather than returning us the raw XML, it returns a custom JSON response, so we'll need to adjust the code to match this format.
Additionally, https://cloudquery.t9t.io/query?url=https://news.ycombinator.com/rss&selectors=* seems to work and returns the raw XML, though within a custom JSON response, so we just need to extract the data from the right element, so that may be an easier and faster change to make.


Implement interface for preview and download the blurred file

A page to view the file that has the PII which is blurred. An option for the user to download the modified document.

Can I work on this issue?

Can I work on this issue?

Sure, assigned!

Hi , I want to work on this issue. Can you assign it to me?

Hi , I want to work on this issue. Can you assign it to me?

Sure


x/net/http: PROTOCOL_ERROR with HTTP2

What version of Go are you using (go version)?

$ go version
go version go1.11.2 linux/amd64

Does this issue reproduce with the latest release?
Yes, which I believe is the one I am running, hence 1.11.2
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/simo/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/simo/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build210197689=/tmp/go-build -gno-record-gcc-switches"

What did you do?
I am downloading tar files from the docker hub, uncompress them using the standard gzip, and passing the result to another software using STDIN.
If is necessary I will try to reproduce this using something simpler like sha256 sum in linux.
What did you expect to see?
I would not expect a PROTOCOL_ERROR
What did you see instead?

Hi @siscia - can you post an independent code sample that we can use to reproduce this issue ?

/cc @bradfitz

@fraenkel, so you see anything wrong here?
@siscia, which HTTP/2 server is cvmfs_server using?

So I am trying to find all the places we just report a stream error without any cause. There aren't many.
I did notice when we receive a WindowUpdateFrame and the increment is 0.
The other is parseHeadersFrame where we go negative.
Adding a cause to all places that return an error would at least identify where its coming from.

@fraenkel I thought this was some mystery server sending us a PROTOCOL_ERROR because it didn't like our Transport's HTTP/2 behavior.

I see them against the docker registries, in particular I can see them against the standard registries hosted by docker hub itself.
I believe they are hosted on AWS.

@bradfitz Sorry I missed the RST_STREAM at the top. I find it odd that all we see are just WINDOW_UPDATE. The fact that "all" or many streams hit this at the same time could mean that we exceeded the connection window?  I am guessing only because we are missing information.
What might be useful is to compare a successful run vs an unsuccessful run and see where the messages differ. It might just be in the exchange of WINDOW_UPDATEs but its just a guess.

Do you guys need any more info from my side?

It may be same as #31534 ?

It may it may not, I worked around the issue and I can't recal the problem!
Feel free to close though!

How did your workaround, if I may ask @siscia ?

I don't remember honestly, two options:

Just download the file/layers using directly docker instead of an HTTP calls
Swallow  the error and just re-trying

I don't remember which option, but nothing technically interesting I am afraid :(

I am seeing something similar to this. On the client side I see stream error: stream ID XXXXXX; PROTOCOL_ERROR and on the server I see stream error: stream ID XXXXXX; NO_ERROR when I try to read the request body with bodyBytes, err := ioutil.ReadAll(r.Body).
I've been able to get around this by using httpClient.Transport.RoundTrip(req) rather than httpClient.Do(req). This seems to reduce occurrences of the issue to basically zero but it also breaks all the timeouts I have set on the client and transport.
FYI I'm using a non tls http2 client:
	httpClient := &http.Client{
		Transport: &http2.Transport{
			AllowHTTP: true,
			DialTLS: func(netw, addr string, cfg *tls.Config) (net.Conn, error) {
				return dialer.Dial(netw, addr)
			},
		},
		Timeout: time.Duration(time.Second * 5),
	}

I'm unable to provide http2 logs because I only see the issue at scale.

2020/08/20 02:26:29 stream error: stream ID 1; PROTOCOL_ERROR
2020/08/20 02:26:29 stream error: stream ID 3; PROTOCOL_ERROR
	client := &http.Client{}
	req, err := http.NewRequest("GET", fmt.Sprintf("https://www.google.com/search?q=renata&source=lnms&tbm=isch"), nil)
	if err != nil {
		log.Fatal(err)
	}
	req.Header.Set("User-Agent", "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:79.0) Gecko/20100101 Firefox/79.0")
	req.Header.Set("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8")
	req.Header.Set("Accept-Language", "en-US,en;q=0.5")
	req.Header.Set("Referer", "https://www.google.com/")
	req.Header.Set("DNT", "1")
	req.Header.Set("Connection", "keep-alive")
	req.Header.Set("Upgrade-Insecure-Requests", "1")
	req.Header.Set("TE", "Trailers")
	req.Header.Set("Cookie", "CGIC=Ikp0ZXh0L2h0bWwsYXBwbGljYXRpb24veGh0bWwreG1sLGFwcGxpY2F0aW9uL3htbDtxPTAuOSxpbWFnZS93ZWJwLCovKjtxPTAuOA; NID=204=QcWiRuWRCGREPOPxM1sA_JgkiIYxtiuZc9xzhvz-fUaVdskJqgjTUsLW6lepRkZ8ieKykY7JZ70cfP8DjArCV-QulLi0oFtzg5r2XwBZRz0jQAS-b_2lVefc9dgcd-DPOkygGclY1YfC__PLq_AWDqrEPjIH_BasiZJ3rbmULkM; 1P_JAR=2020-8-19-19; OTZ=5592678_28_28__28_")
	res, err := client.Do(req)
	if err != nil {
		log.Fatal(err)
	}

	b, err := ioutil.ReadAll(res.Body)
	if err != nil {
		return result, err
	}

	fmt.Println(string(b))

@batara666 this error is coming because HTTP2 does not use transfer encoding (TE).
package main

import (
	"fmt"
	"io/ioutil"
	"log"
	"net/http"
)

func main() {
	client := &http.Client{}
	req, err := http.NewRequest("GET", fmt.Sprintf("https://www.google.com/search?q=renata&source=lnms&tbm=isch"), nil)
	if err != nil {
		log.Fatal(err)
	}
	req.Header.Set("User-Agent", "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:79.0) Gecko/20100101 Firefox/79.0")
	req.Header.Set("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8")
	req.Header.Set("Accept-Language", "en-US,en;q=0.5")
	req.Header.Set("Referer", "https://www.google.com/")
	req.Header.Set("DNT", "1")
	req.Header.Set("Connection", "keep-alive")
	req.Header.Set("Upgrade-Insecure-Requests", "1")
	// req.Header.Set("TE", "Trailers")
	req.Header.Set("Cookie", "CGIC=Ikp0ZXh0L2h0bWwsYXBwbGljYXRpb24veGh0bWwreG1sLGFwcGxpY2F0aW9uL3htbDtxPTAuOSxpbWFnZS93ZWJwLCovKjtxPTAuOA; NID=204=QcWiRuWRCGREPOPxM1sA_JgkiIYxtiuZc9xzhvz-fUaVdskJqgjTUsLW6lepRkZ8ieKykY7JZ70cfP8DjArCV-QulLi0oFtzg5r2XwBZRz0jQAS-b_2lVefc9dgcd-DPOkygGclY1YfC__PLq_AWDqrEPjIH_BasiZJ3rbmULkM; 1P_JAR=2020-8-19-19; OTZ=5592678_28_28__28_")
	res, err := client.Do(req)
	if err != nil {
		log.Fatal(err)
	}

	b, err := ioutil.ReadAll(res.Body)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(string(b))
}

If you uncomment the TE header the remote server will close the stream.
I do not think this is a bug in Go.

Currently just a "me too", but I'm running into a problem very similar to @stan-sack where sometimes file uploads are terminated with stream error: stream ID XXXXXX; NO_ERROR. This happens pretty reliably with the following setup:
.NET Framework HttpClient (HTTP/1.1) performing a multipard/form-data upload to a GCP load balancer over HTTPS with a H2C Cloud Run backend. This is with go 1.20.

upstream max stream duration reached leads me to believe some GCP component is resetting the stream? Maybe Envoy, somehow? This may not be go related, but rather GCP related.
@stan-sack, was your issue happening in GCP?

I was calling non-TLS HTTP/2 (h2c) between two internal services via envoy client load balancing. It was in AWS and there was no actual load balancer.

Seems suspicious though. In my case, envoy (inside GCP load balancer) is also calling non-TLS HTTP/2 (h2c)

This was a long time ago. I dont work at the company where I had this problem anymore.
If I was still trying to solve I'd start playing with TCP keepalive and connection pooling in both go and envoy. I didnt have very much experience with envoy at that time but in hindsight it seems like it could be the source of the problem.


Failed to setup, will retry (forever)

Hey there, trying to set up this integration for the first time.
Integration v0.2.0
OPNsense 24.7.3_1
HASS 2024.9.1 (container installation)
Configuration worked at first, but after this the integration seems to be stuck in an initialization loop as it alternates between these two screens:

and then

I configured extended debugging for the integration and see the following log lines.
2024-09-09 13:52:04.272 DEBUG (SyncWorker_3) [custom_components.opnsense] execution time: OPNSenseData._get_carp_status 60.127
2024-09-09 13:52:14.694 DEBUG (SyncWorker_4) [custom_components.opnsense] execution time: OPNSenseData._get_system_info 60.142
2024-09-09 13:52:14.788 DEBUG (SyncWorker_4) [custom_components.opnsense] execution time: OPNSenseData._get_host_firmware_version 0.093
2024-09-09 13:52:17.760 DEBUG (MainThread) [custom_components.opnsense] Finished fetching fw1 OPNsense state data in 29.001 seconds (success: False)
2024-09-09 13:52:29.891 DEBUG (SyncWorker_5) [custom_components.opnsense] execution time: OPNSenseData._get_firmware_update_info 60.217
2024-09-09 13:52:41.999 DEBUG (SyncWorker_1) [custom_components.opnsense] execution time: OPNSenseData._get_config 60.146
2024-09-09 13:52:42.083 DEBUG (SyncWorker_1) [custom_components.opnsense] execution time: OPNSenseData._get_interfaces 0.084

There are even more logs, containing tons of metrics (gateways, interfaces, filesystems and so on). Tell me if I should also include some of these, but I don't see anything fishy there and it's really a lot of data.
What can I try next?
Regards, Uli

Make sure the url provided is correct, make sure the user for which you created the API key/secret pair has admin privileges.
Please show the config page before submitting it so we can double check things.
From the debug log, the integration has totally failed to communicate with opnsense, might be the url or the key/secret used.

Hi, of course I double-checked the URL and the user.

(and yes, this is a real SSL cert)
some metrics indeed appear in the log, as an example the following mbuf_info and cpu:
2024-09-09 16:51:54.176 DEBUG (SyncWorker_34) [custom_components.opnsense.pyopnsense] [get_telemetry_mbuf] mbuf_info: {'mbuf-statistics': {'mbuf-current': 16728, 'mbuf-cache': 3087, 'mbuf-total': 19815, 'cluster-current': 12374, 'cluster-cache': 3120, 'cluster-total': 15494, 'cluster-max': 378665, 'packet-count': 32, 'packet-free': 1492, 'jumbo-count': 7, 'jumbo-cache': 1263, 'jumbo-total': 1270, 'jumbo-max': 189332, 'jumbo-page-size': 4, 'jumbo9-count': 0, 'jumbo9-cache': 0, 'jumbo9-total': 0, 'jumbo9-max': 56098, 'jumbo16-count': 0, 'jumbo16-cache': 0, 'jumbo16-total': 0, 'jumbo16-limit': 31555, 'bytes-in-use': 28972, 'bytes-in-cache': 12063, 'bytes-total': 41036, 'mbuf-failures': 0, 'cluster-failures': 0, 'packet-failures': 0, 'mbuf-sleeps': 0, 'cluster-sleeps': 0, 'packet-sleeps': 0, 'jumbop-sleeps': 0, 'jumbo9-sleeps': 0, 'jumbo16-sleeps': 0, 'jumbop-failures': 0, 'jumbo9-failures': 0, 'jumbo16-failures': 0, 'sendfile-syscalls': 0, 'sendfile-no-io': 0, 'sendfile-io-count': 0, 'sendfile-pages-sent': 0, 'sendfile-pages-valid': 0, 'sendfile-pages-bogus': 0, 'sendfile-requested-readahead': 0, 'sendfile-readahead': 0, 'sendfile-busy-encounters': 0, 'sfbufs-alloc-failed': 0, 'sfbufs-alloc-wait': 0}}
2024-09-09 16:58:22.301 DEBUG (SyncWorker_58) [custom_components.opnsense.pyopnsense] [get_telemetry_cpu] cpu: {'count': 4}
2024-09-09 16:59:11.344 DEBUG (SyncWorker_8) [custom_components.opnsense.pyopnsense] [get_telemetry_cpu] cpu_info: ['QEMU Virtual CPU version 2.5+ (4 cores, 4 threads)']

Also "interface_info" and "interface" are there. Others fail. So there is fundamental communication between my HASS and OPNsense instances but not all of the queries are successful.
Maybe a privilege problem, my user is in the "admin" group. As I undestand, it should inherit all privileges from there and so I did not configure any more privileges - do I need to?

You didn't post the full debug log, so I couldn't check if there was valid information, I only saw the lines in the OP.
If you post the full one, we can have more info in order to troubleshoot correctly.
The symptoms point toward a privilege issue, but if the user is in the admins group it inherites all privileges. So the only problem could be the key/secret. You can try to regenerate another one.
Let's see the full debug log if you don't solve with the key/secret.

OK, I threw away all the configuration and started from scratch. My intention was to create nice log output for debugging. For some strange reason this time everything worked. No idea why :-)
I'll close this issue - thank you for your help!


Is Ruby 1.9.3.rc1 production-ready?

Or, to put it another way, are you using 1.9.3 in production at 37signals?
I ask because I'm on 1.9.2-p290 and Rails 3.0.10, and while my app runs snappily its startup is glacial.
Thanks for this great software, and rbenv too!

Yes, we're using 1.9.3-rc1. It's running great and is much faster than 1.9.2. The slow startup issues are fixed.
Check out the REE-style GC tuning environment variables, too.


Add E2E tests for Redhat latest minor release upgrades on vsphere

Issue #, if available:
Description of changes:
This adds coverage for latest minor release upgrades on VSphere with Redhat
    cluster.go:896: Skipping VM cleanup
--- PASS: TestVSphereKubernetes126To127RedhatUpgradeFromLatestMinorRelease (2075.17s)
PASS

Testing (if applicable):
Ran e2e
Documentation added/planned (if applicable):
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/test eks-anywhere-presubmit

/cherrypick release-0.17


Skia Roll failing on ASAN stack-use-after-scope in DisplayList unit tests

Sample logs: https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8790775366896723393/+/u/test:_Host_Tests_for_host_debug_unopt/stdout?format=raw
Starting in https://github.com/flutter/engine/pull/39212, but not failing in the attempt at https://github.com/flutter/engine/pull/39204, so it's due to one of these commits:
2023-01-27 [bsalomon<EMAIL_ADDRESS>Reland "Seed coords earlier in shader traversal for SkRasterPipeline."
2023-01-27 [skia-autoroll<EMAIL_ADDRESS>Roll vulkan-deps from d723200bd4b5 to ead30fe6b410 (8 revisions)
2023-01-27 [bsalomon<EMAIL_ADDRESS>Revert "Seed coords earlier in shader traversal for SkRasterPipeline."
2023-01-27 [bsalomon<EMAIL_ADDRESS>Seed coords earlier in shader traversal for SkRasterPipeline.

cc @bsalomon

The fix has been merged in. Closing.


Can’t submit an invoice sent by a vendor

When submitting an invoice, the field “Who is being paid for this expense” only shows me, the different organizations I’m part of, but it doesn’t let me create a new organization for the invoice that I received.
We got a €40 invoice from the lawyer and we cannot add it for them in the collective (and it’s very cumbersome to ask them to do it themselves). I should be able to submit the expense on their behalf.

We can't pay to third parties without their involvement. I have suggested many times we create a feature for this (called "submit on behalf" for lack of a better name), which should allow the submitter to fill out all the info and just send the actual payee an invitation link to confirm.
See #3278

We should be able to.
Look, here I have to submit an invoice from Deliveroo and Open Collective thinks that I'm the one that will receive the money. That's not ok.
Any accounting software would let you add a vendor without their consent.
That said, I agree that it would be a good idea to require an email address when adding a vendor that could confirm the creation of an open collective account. But this shouldn't block the ability for a collective to record a vendor.

Any accounting software would let you add a vendor without their consent.

This is not true - we constantly add any of our hosts as vendors to multiple companies / foundations and we always need to validate email. I don't think this is even legal under GDPR regulations.
This is a feature we also want to see done - but we need to do it properly.

@xdamman the workaround for now is you can create an organization profile for the vendor yourself and select it when you submit the expense under "pay to". You can either add someone from there as the admin and remove yourself, or just manage that profile for them. Then at least our records show the income going to the correct payee, and if we have to get a tax form or whatever we will contact you as the admin of that profile and if you can't represent the vendor to fill out the tax form or whatever, then you'd need to get them involved. But you can see there are obvious drawbacks to having random people managing profile for companies they don't actually represent.

This will be solved by the invite expense feature which is being built.

Since it refactors multiple crucial components in the expense flow, I shipped this under a feature flag enabled on a collective basis for testing.
@xdamman, I enabled it for Brussels Together, let me know if you want to test this in any other collective.

Implemented and enabled by default.


Bumping up the bolt version to latest commit tag for compiling Power …

…and Z
Signed-off-by: Srini Brahmaroutu<EMAIL_ADDRESS>sorry to duplicate the effort from @hqhq #16898 but now that the support for Power and Z are also merged this PR is going to help setup Power and Z CI runs.

also please see @tianon boltdb/bolt#438

Aren't we waiting for a tag for bolt? Once we got that, or got answer from bolt, I can rebase #16898 , it'll surely including Power and Z.

@hqhq Sure that will work for me.

@brahmaroutu so are we inn favor of @hqhq 's then, can we close this and wait forr a tag

Sure, closing this.


NestedCloudKitCodable.CKCodableError.ErrorType.recordNotFound

Hello,
I'm having a problem retrieving the data. I have two main classes, where one have an array of the other. When this array is empty, all the data are saved and retrieved correctly. But when the array is not empty, I get the following error:
NestedCloudKitCodable.CKCodableError.ErrorType.recordNotFound
I'm not sure what may be the cause of this problem. Any help is appreciated.

Yes I just encoded my objects as data.. But it would be nice if this problem can be fixed

Hi @HasanKassem I've recently released a new version (1.0.3), could you please update and see if the error persists?
If you could provide the class structures, it would help me a lot in fixing this issue.

It's working now, thank you. But I'm having another issue, when trying different zone using CKRecordEncoder(zoneID: ) I'm getting ZoneId provided doesn't match target zone.

Okay so the problem was in
private func produceReference(for value: CKEncodable) throws -> CKRecord.Reference {         let recordID = CKRecord.ID(recordName: value.cloudKitIdentifier)         return CKRecord.Reference(recordID: recordID, action: .deleteSelf)     }
I changed it to
private func produceReference(for value: CKEncodable) throws -> CKRecord.Reference {         let recordID = CKRecord.ID(recordName: value.cloudKitIdentifier, zoneID: zoneID ?? .default)         return CKRecord.Reference(recordID: recordID, action: .deleteSelf)     }

Thanks for the PR!

@HasanKassem I published 1.0.4 which includes yours fix. Thanks again.


Generate secrets for Iroha peers in k8s and add persisted storage.

Signed-off-by: bagf

Description of the Change

Currently k8s-iroha Ansible playbook uses a Python script to save each peer key pair to files/conf. The command kubectl create configmap should be called afterwards to save the: peer key pairs, genesis block and config to Kubernetes.
This PR will generate secrets in k8s-peer-keys.yaml alongside the statefulset in k8s-iroha.yaml. The statefulset will use a init container (busybox) to copy create peer keys for each pod. I see this as a temporary implementation, once PodPresets become available there wont be any need to copy the keys before the Iroha container starts.
Additionally persistent volumes will be requested for both the Postgres and Iroha block store volumes. emptyDir was used previously which would forget data between restarts. If this temporary store feature is required I can implement a config option to use emptyDir for storage in the statefulset.
Benefits

Each Iroha peer gets it's own Kubernetes Secret, each pod can only read it's allocated Secret.
Resulting Configmap is smaller as it doesn't need to store any key pair files.
Persistent storage now configurable for both Postgres and Iroha's block store.

Possible Drawbacks

Temporary storage for statefulset no longer used. As mentioned above I can add it if needed
Slightly more complicated pod initialization
Postgres starts in the same pod and might not be ready when irohad tries to connect (sleep was used previously, might use it again that if there isn't a better option)

Usage Examples or Tests [optional]

Alternate Designs [optional]

PodPresets should work well when the cluster supports them. This solution was tested on 1.9 and 1.10 without any issues / modifications to the cluster configuration.
Alternative secrets implementation using PodPresets: https://github.com/CastleOne/iroha/blob/k8s-ansible-secrets-pod-preset/deploy/ansible/roles/iroha-k8s/templates/k8s-peer-keys.yaml.j2#L13-L28

LGTM
Wait for @bakhtin

@bagf happy to see this first-time contribution from you! Can we meet your team in our public rocket.chat in order to know more about your project? Please join the group https://chat.hyperledger.org/channel/iroha


[REF] Make session count names clearer in query response

Closes #271

Changes proposed in this pull request:

Response model for a single matching session response updated
Query template updated
Default SPARQL query regenerated

Note:

the query parameters themselves have not been renamed (b/c min_num_{phenotypic,imaging}_sessions still makes more sense in a query)
we currently (still) do not have a field in a single session query response that indicates the total (not matching) num. of sessions available for the subject. however, this can be deduced from the number of session objects present for a subject in the full response

Checklist

[ ] PR has an interpretable title with a prefix ([ENH], [FIX], [REF], [TST], [CI], [MNT], [INF], [MODEL], [DOC]) (see https://neurobagel.org/contributing/pull_requests for more info)
[ ] PR has a label for the release changelog or skip-release (to be applied by maintainers only)
[ ] PR links to GitHub issue with mention Closes #XXXX
[ ] Tests pass
[ ] Checks pass

For new features:

[ ] Tests have been added

For bug fixes:

[ ] There is at least one test that would fail under the original bug conditions.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 2 (100.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 99.785%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
929

Relevant Lines:
931

💛  - Coveralls

:rocket: PR was released in v0.2.0 :rocket:


check_indexes: run without blocking during a vacuum_full or cluster

Right now, if you do a vacuum_full or a cluster, and then you query check_indexes, check_indexes is blocked.

Complete.

Can we add how much Index fragmented/bloated as well

@binumarar that's a separate feature request, unrelated to this issue. Please don't add unrelated comments to Github issues. Thanks for understanding.

Thank you for the update.
On Tue, 29 Oct, 2024, 5:02 pm SmartPostgres, @.***>
wrote:

@binumarar https://github.com/binumarar that's a separate feature
request, unrelated to this issue. Please don't add unrelated comments to
Github issues. Thanks for understanding.
—
Reply to this email directly, view it on GitHub
https://github.com/SmartPostgres/Box-of-Tricks/issues/32#issuecomment-2443961601,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BMQDPIASVPJD63GMCLI6PXTZ55W6BAVCNFSM6AAAAABQKI7WWOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBTHE3DCNRQGE
.
You are receiving this because you were mentioned.Message ID:
@.***>


inject go modules directly for easier debugging.

Debugging with go modules are quite difficult, so with that pull request nakama' startup sequence moved to package based system. It shouldn't effect regular flow of nakama instance by default, but it will allow injecting go modules directly to core instance.
If you want to test this, clone this source branch. lets say path is "/Users/mertcelen/new-nakama"
example code should like like this
package main

import (
	"context"
	"database/sql"
	nakamaRuntime "github.com/heroiclabs/nakama-common/runtime"
	nakamaInstance "github.com/heroiclabs/nakama/v3/cmd/instance"
	nakamaServer "github.com/heroiclabs/nakama/v3/server"
)

func main() {
	mainInstance := nakamaInstance.New()
	mainInstance.AddGoModule(Test)
	mainInstance.Start()
}

func Test(ctx context.Context, logger nakamaRuntime.Logger, db *sql.DB, module *nakamaServer.RuntimeGoNakamaModule, initializer *nakamaServer.RuntimeGoInitializer) error {
	return nil
}

and you should add this line to go.mod file below import to be able to test the new code
replace github.com/heroiclabs/nakama/v3 => /Users/mertcelen/new-nakama

final should like this
module nakamaDebugger

go 1.22

require github.com/heroiclabs/nakama/v3 v3.22.0

replace github.com/heroiclabs/nakama/v3 => /Users/mertcelen/new-nakama
require (
    ...

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

We recently revamped our Debugging with Delve documentation, that is our official recommendation for debugging Go runtime code or its integration with Nakama's internals.
Re-flowing the integration as you suggest might be possible, but it should not be necessary. 🙇


Can not interrupt a multi part download with Ctrl-C

My interact log
./biligrab.py -a 1999286 -s 3 -d wget
INFO:root:No part number set, download all the parts.
INFO:root:You are downloading all the parts in this video...
INFO:root:Fetching api to read video info...
INFO:root:cid is 3109121
INFO:root:partname is 喜
INFO:root:Title is 【哔哩哔哩2015拜年祭】
INFO:root:Total pages is 4
INFO:root:Your target download is av1999286, part 1~4, from source 3
INFO:root:Downloading part 1 ...
INFO:root:Fetching api to read video info...

First Ctrl-C, not stopping, but immediately skip to part 2
^CWARNING:root:Cannot connect to API server!
If you think this is wrong, please open an issue at
https://github.com/cnbeining/Biligrab/issues with *ALL* the screen output,
as well as your IP address and basic system info.
You can get these data via "-l".
ERROR: Biligrab failed: local variable 'data' referenced before assignment
       If you think this should not happen, please dump your log using "-l", and open a issue ar https://github.com/cnbeining/Biligrab/issues .
       Make sure you delete all the sensive data before you post it publicly.
INFO:root:Downloading part 2 ...
INFO:root:Fetching api to read video info...
INFO:root:cid is 3107647
INFO:root:partname is 羊
INFO:root:Title is 【哔哩哔哩2015拜年祭】
INFO:root:Total pages is 4
INFO:root:Fetching XML...
^CINFO:root:The XML file, 羊.xml should be ready...enjoy!
INFO:root:Finding video location...
INFO:root:Trying to get download URL...

Second Ctrl-C, not stopping, but throw out an error as below, and skipped to part 3
^CERROR:root:Cannot connect to HTML5 API!
ERROR: Biligrab failed: 'utf8' codec can't decode byte 0x8b in position 1: invalid start byte
       If you think this should not happen, please dump your log using "-l", and open a issue ar https://github.com/cnbeining/Biligrab/issues .
       Make sure you delete all the sensive data before you post it publicly.
INFO:root:Downloading part 3 ...
INFO:root:Fetching api to read video info...

Third Ctrl-C, still not stopping, threw an error.
^CWARNING:root:Cannot connect to API server!
If you think this is wrong, please open an issue at
https://github.com/cnbeining/Biligrab/issues with *ALL* the screen output,
as well as your IP address and basic system info.
You can get these data via "-l".
ERROR: Biligrab failed: local variable 'data' referenced before assignment
       If you think this should not happen, please dump your log using "-l", and open a issue ar https://github.com/cnbeining/Biligrab/issues .
       Make sure you delete all the sensive data before you post it publicly.
INFO:root:Downloading part 4 ...
INFO:root:Fetching api to read video info...

Fourth Ctrl-C, stopped after printing out garbage.
^CWARNING:root:Cannot connect to API server!
If you think this is wrong, please open an issue at
https://github.com/cnbeining/Biligrab/issues with *ALL* the screen output,
as well as your IP address and basic system info.
You can get these data via "-l".
ERROR: Biligrab failed: local variable 'data' referenced before assignment
       If you think this should not happen, please dump your log using "-l", and open a issue ar https://github.com/cnbeining/Biligrab/issues .
       Make sure you delete all the sensive data before you post it publicly.

Problem
This could a pain if someone wants to batch download 100 parts.
Conclusion
After pressing Ctrl-C, Biligrab randomly skip to another part or print out random garbage.
Solution
A known solution is to replace all except: (bare except) to except Exception:

Not sure whether I've done this right, give it a shot.

Close for now.


Update Existing.md

Поправил опечатки

Codecov Report

Merging #53 into master will increase coverage by 65.19%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           master      #53       +/-   ##
===========================================
+ Coverage   17.63%   82.83%   +65.19%
===========================================
  Files          45       45
  Lines         635      635
===========================================
+ Hits          112      526      +414
+ Misses        523      109      -414

Impacted Files
Coverage Δ

CoreNetKit/Utils/Logging/Log.swift
100% <0%> (+33.33%)
:arrow_up:

...sponseProcessingLayer/ResponseDataParserNode.swift
63.33% <0%> (+63.33%)
:arrow_up:

...uestProcessingLayer/Support/Method+Alamofire.swift
65% <0%> (+65%)
:arrow_up:

...eNetKit/CacheNode/IfServerFailsFromCacheNode.swift
73.33% <0%> (+73.33%)
:arrow_up:

...yers/RequestProcessingLayer/ReqestSenderNode.swift
76.47% <0%> (+76.47%)
:arrow_up:

...ocessingLayer/ResponseHttpErrorProcessorNode.swift
80.95% <0%> (+80.95%)
:arrow_up:

...Layers/TrasportLayer/TechnicaErrorMapperNode.swift
85.71% <0%> (+85.71%)
:arrow_up:

...esponseProcessingLayer/ResponseProcessorNode.swift
86.84% <0%> (+86.84%)
:arrow_up:

CoreNetKit/Utils/Mapping/CodableRawMapper.swift
90.9% <0%> (+90.9%)
:arrow_up:

...tKit/Layers/DTOProcessingLayer/DTOMapperNode.swift
90.9% <0%> (+90.9%)
:arrow_up:

... and 21 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ef5e565...6f10d93. Read the comment docs.

Вот это у меня мощный ПР, так Code coverage поднял 😎


Let's get sticky

Can you add support for the y modifier?

So this is a won't fix???

No, it's fixed at up on the site right now. Change 988f5ca added support for /some expression/y for sticky and /some expression/u for unicode support.
Sorry for the confusion, I'll try to add better notes in the future. To be honest, I was confused about why I closed this for a moment.


ON-14945: Add kustomization to configure source locations

Adds a cluster/ directory with an onload-sources.conf file which specificies the locations of tarballs for the Dockerfiles to pull and build from. This defaults to using github as the source, but this change will allow for easy configuration if github is inaccesible.
Also adds kustomization replacements to update the buildArgs for the onload and sfc builds. Existing command line invocations should be unaffected by this change.
Testing done
Works in cluster, also tested with a local http server which provided the onload tarball

A limitation to this approach is how we update the image tags for all of the built images.
I had thought about adding extra definitions in onload-sources.conf for the version of each source we are using (eg ONLOAD_SOURCE, SFPTPD_SOURCE, KUBERNETES_ONLOAD_SOURCE, etc.), but this type of kustomization will replace the entire field of the definition.
A potential solution is to define the full image names in onload-sources.conf with registry, image and tag all as one value.

I'd be tempted to use ./base (or ./locale) instead of ./cluster due to the directory not defining a cluster.

./base sounds good!

@pcolledg-xilinx I have added a section to README, could you give it a check?


Add input delay

Make sure the init routine only runs once and add input delay.
input_latency_frames is a private which means you can get rid of it using poly or by reloading the save.

Certainly an improvement


Authentication(Jwt)

Can I setup authentication via jwt token, here when an user logs in to our application we provide a user with a token which can be used for later verification.
I will also create a logout route for logging out the the user where the token initially provided will be destroyed.
Wishing to work on the backend part of this.
Participant under gssoc’22.

Sure @ayush25102001


Mirror PT labels as GH issue labels

This enables filtering stories by labels, as well as saving screen real-estate by removing labels from story description and comments.
Out of scope:

handling PT label renamed to avoid orphan GH labels

The associated risks:

GH label pollution

thanks @ArthurHlt for the improvements brought to this issue. I'm wondering whether this change might have introduced regression on the state mapping as delivered issues are now in the github open status, whereas we'd expect a closed state.

Pending suspected limitation to address: a removed label from a story remained on the GB issue mirror.

Addressed.
Validated that removed labels from stories are properly mirrored in the github issue. See https://github.com/cf-trackermirror-bot/buildpacks-tracker-mirror/issues/195#event-765129798


ng2-smart-table data how to use a pipe?

if I hava a date in my data like this.

what I should do to translate the date to yyyy-MM-dd HH:mm:ss ?

https://github.com/akveo/ng2-smart-table/issues/52

#52

any answer else ?

if I hava a date in my data like this.

what I should do to translate the date to yyyy-MM-dd HH:mm:ss ?

any answer else ?

@xiaomoinfo i have achieved this by the following way:
columns: {
id: {
title: Id,
type: 'number',
sort: false
},
Date: {
title: Date,
type: 'date',
valuePrepareFunction: (date) => {
if (date) {
return new DatePipe('he-IL' or 'en-GB' ['any lang code you want]).transform(date, 'dd/MM/yyyy hh:mm');
}
return null;
},
sort: false
}

i try to use pipe but inside my pipe there's async data :
my pipe :
import { Pipe, PipeTransform } from "@angular/core";
import { VehiculeService } from "app/pages/fleetManagement/services/vehicule.service";

@Pipe({
  name: "convertVehicule"
})
export class ConvertVehiculePipe implements PipeTransform {
  constructor(private vehicleService: VehiculeService) {}

  transform(value) {
    this.vehicleService.getVehicules().subscribe(
      (data: any = []) => {
        for (let j = 0; j < data.length; j++) {
          if (value === data[j].id) {
            return data[j].registration_number;
          }
        }
      },
      error => {
        console.log(error);
      }
    );

    return value;
  }
}

And this is my function :

 valuePrepareFunction: cell => {
            let newValue = this.convertVehiclePipe.transform(cell);
            return newValue;
          }

I did this but no value is rendered, i thinked to use async like {{  data | async }} because the data is asynchronus , but where

i try to use pipe but inside my pipe there's async data :
my pipe :
import { Pipe, PipeTransform } from "@angular/core";
import { VehiculeService } from "app/pages/fleetManagement/services/vehicule.service";

@Pipe({
  name: "convertVehicule"
})
export class ConvertVehiculePipe implements PipeTransform {
  constructor(private vehicleService: VehiculeService) {}

  transform(value) {
    this.vehicleService.getVehicules().subscribe(
      (data: any = []) => {
        for (let j = 0; j < data.length; j++) {
          if (value === data[j].id) {
            return data[j].registration_number;
          }
        }
      },
      error => {
        console.log(error);
      }
    );

    return value;
  }
}

And this is my function :

 valuePrepareFunction: cell => {
            let newValue = this.convertVehiclePipe.transform(cell);
            return newValue;
          }

I did this but no value is rendered, i thinked to use async like {{  data | async }} because the data is asynchronus , but where


Error 0x0001: One or more arguments are not correct

I got this error and couldn't find it in the FAQ.
Here's the entire log:
--- ModelDocumenter 2.0 for.Net (v4.0.30319)
Successfully loaded plugin CommandLineArgumentsParser, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=2990a79b704d8378 from assembly CommandLineArgumentsParser.dll
Successfully loaded plugin Dax.Metadata, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=null from assembly Dax.Metadata.dll
Successfully loaded plugin Dax.Model.Extractor, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=null from assembly Dax.Model.Extractor.dll
Successfully loaded plugin Dax.ViewVpaExport, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=null from assembly Dax.ViewVpaExport.dll
Successfully loaded plugin Dax.Vpax, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=null from assembly Dax.Vpax.dll
Successfully loaded plugin Microsoft.AnalysisServices.AdomdClient, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=89845dcd8080cc91 from assembly Microsoft.AnalysisServices.AdomdClient.dll
Successfully loaded plugin Microsoft.AnalysisServices.Core, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=89845dcd8080cc91 from assembly Microsoft.AnalysisServices.Core.dll
Successfully loaded plugin Microsoft.AnalysisServices, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=89845dcd8080cc91 from assembly Microsoft.AnalysisServices.dll
Successfully loaded plugin Microsoft.AnalysisServices.SPClient.Interfaces, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=89845dcd8080cc91 from assembly Microsoft.AnalysisServices.SPClient.Interfaces.dll
Successfully loaded plugin Microsoft.AnalysisServices.Tabular, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=89845dcd8080cc91 from assembly Microsoft.AnalysisServices.Tabular.dll
Successfully loaded plugin Microsoft.AnalysisServices.Tabular.Json, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=89845dcd8080cc91 from assembly Microsoft.AnalysisServices.Tabular.Json.dll
Successfully loaded plugin netstandard, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51 from assembly netstandard.dll
Successfully loaded plugin Newtonsoft.Json, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed from assembly Newtonsoft.Json.dll
Successfully loaded plugin System.Diagnostics.Tracing, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a from assembly System.Diagnostics.Tracing.dll
Successfully loaded plugin System.IO.Compression, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b77a5c561934e089 from assembly System.IO.Compression.dll
Successfully loaded plugin System.IO.Compression.FileSystem, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b77a5c561934e089 from assembly System.IO.Compression.FileSystem.dll
Successfully loaded plugin System.Net.Http, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a from assembly System.Net.Http.dll
Argument s(server) is not marked as optional and was not found on the command line.
Usage:
-s, --server... This parameter specifies the server name.
    -d, --database... This parameter specifies the database name.

    -f, --filename[optional]... This parameter specifies the export filename.

    -p, --pbitemplate... This parameter specifies the location of the power bi template file.

Error 0x0001: One or more arguments are not correct.

Hi @zfrank67
Sorry to hear you're running into an error. Can you provide me a bit more details on what you're trying to do when you run into this error?

Did you run the tool from the external tools ribbon in Power BI Desktop?
Do you have the data model imported or on DirectQuery in the model, or you use a live connection to an existing dataset?
Which version of Power BI Desktop are you running?

Thanks!
--Marc

Oh my gosh. Marc please ignore me and forget this ever happened. I can't read basic instructions and was running the tool directly from it's .exe vs through PowerBI.

From: Marc Lelijveld @.>
Sent: Friday, September 2, 2022 5:47 AM
To: marclelijveld/External-Tools-Model-Documentation @.>
Cc: zfrank67 @.>; Mention @.>
Subject: Re: [marclelijveld/External-Tools-Model-Documentation] Error 0x0001: One or more arguments are not correct (Issue #46)
Hi @zfrank67https://github.com/zfrank67
Sorry to hear you're running into an error. Can you provide me a bit more details on what you're trying to do when you run into this error?

Did you run the tool from the external tools ribbon in Power BI Desktop?
Do you have the data model imported or on DirectQuery in the model, or you use a live connection to an existing dataset?
Which version of Power BI Desktop are you running?

Thanks!
--Marc
—
Reply to this email directly, view it on GitHubhttps://github.com/marclelijveld/External-Tools-Model-Documentation/issues/46#issuecomment-1235351032, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFQ6QIUUMAWMTDOIQQ75IELV4HLNTANCNFSM6AAAAAAQBRF274.
You are receiving this because you were mentioned.Message ID: @.***>

Glad you figured it out @zfrank67! I'll close this one.


concat example stuck at

Hi
the concat examples gets deployed effectively (https://github.com/projectriff/riff/tree/master/samples/python/concat) but when you use the riff publish line listed in the readme the function is not able to execute as the containers creation gets stuck at:
NAME                      READY     STATUS             RESTARTS   AGE
concat-77db889cd4-s8zhr   0/2       ImagePullBackOff   12         54m
Is there something that got moved?
thank you very much.
F.

I make this more clear (hopefully). It seems that in some examples the riff docker push fails while in other examples works, If I manually push the docker image just after issuing the riff create command then the image pull succeeds:

The sample has been moved to https://github.com/projectriff/python3-function-invoker/tree/master/samples/concat.
I cannot reproduce this after cleaning out my local docker repository.  If you are running minikube, you need to eval $(minikube docker-env) .   Feel free to reopen if you think this is still an issue.


Every example in Tutorial mis-references CSS

Is it me or does every example in the tutorial contain:
<script src="/release/ui-grid-unstable.css"></script>
Shouldn't this be <link rel="stylesheet" href="/release/ui-grid-unstable.css" /> ?
For example, line 10 in http://ui-grid.info/docs/#/tutorial/102_sorting . But every example has it the same way.
Or am I missing something.

Closing on the basis that it doesn't break anything, and it's an upstream problem.


ApiTest. Sharing link

https://owncloud.dev/libre-graph-api/#/drives.permissions/CreateLink

[x] Create a sharing link @SagarGi

[x] with all roles and all existing permissions, https://github.com/owncloud/ocis/pull/7982
[x] custom name, https://github.com/owncloud/ocis/pull/8036
[x] expiry date (https://matrix.to/#/!QxIasTYvslDCVqoPzC:matrix.org/$G5MC_dC6YmOGr0SK44E5563NMW-8MCTGCo-fhlsk6Ko?via=matrix.org&via=openproject.org&via=element.io) https://github.com/owncloud/ocis/pull/8036
[x] without password  (ocis environment may needed) https://github.com/owncloud/ocis/pull/8040
[x] type internal  https://github.com/owncloud/ocis/pull/8210

[x] update expiration date after creating share https://github.com/owncloud/ocis/pull/8071
[x] update roles after creating share https://github.com/owncloud/ocis/pull/8071
[x] set password after user creates link https://github.com/owncloud/ocis/pull/8079
[x] set banned password while creating share link https://github.com/owncloud/ocis/pull/8194

Closing this issue since all the tests have been added for sharing Ng.