[PyTorch 2.0] Bug with torch.compile() and self.log()

Bug description
When using brand new torch.compile from PyTorch 2.0 with Lightning 2.0, I encounter a weird error with my LightningModule...
The is running when no call to self.log(...) is made. When I try to log my loss value, the code crashes.
How to reproduce the bug
import lightning as L
import torch
import torch._dynamo
import torchvision
from timm import create_model
from torch import nn
from torch.utils.data import DataLoader
from torchvision import transforms

class CIFAR10DataModule(L.LightningDataModule):
    def __init__(self, batch_size=32) -> None:
        super().__init__()
        self.transform = transforms.Compose(
            [transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))]
        )

        self.batch_size = batch_size

        self.train_set = torchvision.datasets.CIFAR10(
            root='~/data', train=True, download=True, transform=self.transform
        )
        self.test_set = torchvision.datasets.CIFAR10(
            root='~/data', train=False, download=True, transform=self.transform
        )

    def train_dataloader(self):
        return DataLoader(self.train_set, batch_size=self.batch_size, shuffle=True, num_workers=4)

    def val_dataloader(self):
        return DataLoader(self.train_set, batch_size=self.batch_size, shuffle=False, num_workers=4)

class LitModel(L.LightningModule):
    def __init__(self):
        super().__init__()
        self.model = create_model('resnet18', num_classes=10)
        self.criterion = nn.CrossEntropyLoss()

    def forward(self, x):
        return self.model(x)

    def common_step(self, x, y, stage):
        logits = self(x)
        loss = self.criterion(logits, y)
        self.log(f'{stage}_loss', loss, on_epoch=True, prog_bar=True, logger=True)
        return loss

    def training_step(self, batch, batch_idx):
        x, y = batch
        loss = self.common_step(x, y, 'train')
        return loss

    def validation_step(self, batch, batch_idx):
        x, y = batch
        loss = self.common_step(x, y, 'val')
        return {'val_loss': loss}

    def configure_optimizers(self):
        return torch.optim.Adam(self.model.parameters(), lr=0.001)

def main():
    datamodule = CIFAR10DataModule()
    model = LitModel()
    compiled_model = torch.compile(model)
    trainer = L.Trainer(max_epochs=3, devices='auto')
    trainer.fit(compiled_model, datamodule=datamodule)

if __name__ == '__main__':
    main()

Error messages and logs
Files already downloaded and verified
Files already downloaded and verified
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
You are using a CUDA device ('NVIDIA RTX A5000 Laptop GPU') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]

  | Name      | Type             | Params
-----------------------------------------------
0 | model     | ResNet           | 11.2 M
1 | criterion | CrossEntropyLoss | 0
-----------------------------------------------
11.2 M    Trainable params
0         Non-trainable params
11.2 M    Total params
44.727    Total estimated model params size (MB)
Sanity Checking DataLoader 0:   0%|                                                                                | 0/2 [00:00<?, ?it/s]/alex/lib/python3.8/site-packages/torch/_inductor/compile_fx.py:90: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
  warnings.warn(
Traceback (most recent call last):
  File "test_3.py", line 72, in <module>
    main()
  File "test_3.py", line 68, in main
    trainer.fit(compiled_model, datamodule=datamodule)
  File "/alex/lib/python3.8/site-packages/lightning/pytorch/trainer/trainer.py", line 520, in fit
    call._call_and_handle_interrupt(
  File "/alex/lib/python3.8/site-packages/lightning/pytorch/trainer/call.py", line 44, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/alex/lib/python3.8/site-packages/lightning/pytorch/trainer/trainer.py", line 559, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/alex/lib/python3.8/site-packages/lightning/pytorch/trainer/trainer.py", line 935, in _run
    results = self._run_stage()
  File "/alex/lib/python3.8/site-packages/lightning/pytorch/trainer/trainer.py", line 976, in _run_stage
    self._run_sanity_check()
  File "/alex/lib/python3.8/site-packages/lightning/pytorch/trainer/trainer.py", line 1005, in _run_sanity_check
    val_loop.run()
  File "/alex/lib/python3.8/site-packages/lightning/pytorch/loops/utilities.py", line 174, in _decorator
    return loop_run(self, *args, **kwargs)
  File "/alex/lib/python3.8/site-packages/lightning/pytorch/loops/evaluation_loop.py", line 115, in run
    self._evaluation_step(batch, batch_idx, dataloader_idx)
  File "/alex/lib/python3.8/site-packages/lightning/pytorch/loops/evaluation_loop.py", line 375, in _evaluation_step
    output = call._call_strategy_hook(trainer, hook_name, *step_kwargs.values())
  File "/alex/lib/python3.8/site-packages/lightning/pytorch/trainer/call.py", line 288, in _call_strategy_hook
    output = fn(*args, **kwargs)
  File "/alex/lib/python3.8/site-packages/lightning/pytorch/strategies/strategy.py", line 378, in validation_step
    return self.model.validation_step(*args, **kwargs)
  File "/alex/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 209, in _fn
    return fn(*args, **kwargs)
  File "test_3.py", line 56, in validation_step
    loss = self.common_step(x, y, 'val')
  File "test_3.py", line 44, in common_step
    logits = self(x)
  File "test_3.py", line 46, in <graph break in common_step>
    self.log(f'{stage}_loss', loss, on_epoch=True, prog_bar=True, logger=True)
  File "/alex/lib/python3.8/site-packages/lightning/pytorch/core/module.py", line 403, in log
    apply_to_collection(value, dict, self.__check_not_nested, name)
  File "/alex/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 337, in catch_errors
    return callback(frame, cache_size, hooks)
  File "/alex/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 404, in _convert_frame
    result = inner_convert(frame, cache_size, hooks)
  File "/alex/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 104, in _fn
    return fn(*args, **kwargs)
  File "/alex/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 262, in _convert_frame_assert
    return _compile(
  File "/alex/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper
    r = func(*args, **kwargs)
  File "/alex/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 324, in _compile
    out_code = transform_code_object(code, transform)
  File "/alex/lib/python3.8/site-packages/torch/_dynamo/bytecode_transformation.py", line 445, in transform_code_object
    transformations(instructions, code_options)
  File "/alex/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 311, in transform
    tracer.run()
  File "/alex/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1726, in run
    super().run()
  File "/alex/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 576, in run
    and self.step()
  File "/alex/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 540, in step
    getattr(self, inst.opname)(inst)
  File "/alex/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 378, in wrapper
    resume_call_insts = self.create_call_resume_at(self.next_instruction)
  File "/alex/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1755, in create_call_resume_at
    new_code: types.CodeType = ContinueExecutionCache.lookup(
  File "/alex/lib/python3.8/site-packages/torch/_dynamo/resume_execution.py", line 154, in lookup
    cls.cache[code][key] = cls.generate(code, lineno, *key)
  File "/alex/lib/python3.8/site-packages/torch/_dynamo/resume_execution.py", line 233, in generate
    new_code = transform_code_object(code, update)
  File "/alex/lib/python3.8/site-packages/torch/_dynamo/bytecode_transformation.py", line 446, in transform_code_object
    return clean_and_assemble_instructions(instructions, keys, code_options)[1]
  File "/alex/lib/python3.8/site-packages/torch/_dynamo/bytecode_transformation.py", line 470, in clean_and_assemble_instructions
    code_options["co_stacksize"] = stacksize_analysis(instructions)
  File "/alex/lib/python3.8/site-packages/torch/_dynamo/bytecode_analysis.py", line 184, in stacksize_analysis
    assert fixed_point.value, "failed to reach fixed point"
AssertionError: failed to reach fixed point

from user code:
   File "/alex/lib/python3.8/site-packages/lightning_utilities/core/apply_func.py", line 94, in apply_to_collection
    if is_dataclass_instance(data):

Set torch._dynamo.config.verbose=True for more information

You can suppress this exception and fall back to eager by setting:
    torch._dynamo.config.suppress_errors = True

Environment

Current environment
#- Lightning Component (e.g. Trainer, LightningModule, LightningApp, LightningWork, LightningFlow):
#- PyTorch Lightning Version (e.g., 1.5.0): 2.0.0
#- PyTorch Version (e.g., 2.0): 2.0.0
#- Python version (e.g., 3.9): 3.8.13
#- OS (e.g., Linux): Linux
#- CUDA/cuDNN version:  11.7
#- How you installed Lightning(`conda`, `pip`, source): pip

More info
The error appears when the following line is set:
        self.log(f'{stage}_loss', loss, on_epoch=True, prog_bar=True, logger=True)

Without that line, the code is working as expected. Any idea?
cc @carmocca @tchaton

I think torch.compile supposed to be applied to nn.Module instead of L.LightningModule
try change:
def main():
    datamodule = CIFAR10DataModule()
    model = LitModel()
    compiled_model = torch.compile(model)
    trainer = L.Trainer(max_epochs=3, devices='auto')
    trainer.fit(compiled_model, datamodule=datamodule)

to:
def main():
    datamodule = CIFAR10DataModule()
    model = LitModel()
    model.model = torch.compile(model.model)
    trainer = L.Trainer(max_epochs=3, devices='auto')
    trainer.fit(compiled_model, datamodule=datamodule)

or just put the torch.compile inside the LightningModule init method:
class LitModel(L.LightningModule):
    def __init__(self):
        super().__init__()
        self.model = torch.compile(create_model('resnet18', num_classes=10))
        self.criterion = nn.CrossEntropyLoss()

@BenedictusAryo Thanks for your answer. Indeed, using torch.compile directly on the nn.Module will work, but as a user, I expect it to work also on a LightningModule (since, under the hood, it is a subclass of nn.Module.
Moreover, it is said in the changelog:

Lightning 2.0 is best friends with PyTorch 2.0. You can torch.compile your LightningModules now!
import torch
import lightning as L

model = LitModel()
# This will compile forward and {training,validation,test,predict}_step
compiled_model = torch.compile(model)

trainer = L.Trainer()
trainer.fit(compiled_model)

So we should expect torch.compile working with our LightningModules too. :)
Thanks for the workaround in the meantime.

I'm not able to reproduce. Can you share the output of pip list?

I was able to trim the issue down and opened a bug report in https://github.com/pytorch/pytorch/issues/97811
I believe this is only an issue on Python 3.8

Thanks @carmocca for your help. Indeed, I should have made it clearer that the 3.8 version of python was used. I haven't tested other versions.
Thanks for your investigation. 😊

I think torch.compile supposed to be applied to nn.Module instead of L.LightningModule try change:
def main():
    datamodule = CIFAR10DataModule()
    model = LitModel()
    compiled_model = torch.compile(model)
    trainer = L.Trainer(max_epochs=3, devices='auto')
    trainer.fit(compiled_model, datamodule=datamodule)

to:
def main():
    datamodule = CIFAR10DataModule()
    model = LitModel()
    model.model = torch.compile(model.model)
    trainer = L.Trainer(max_epochs=3, devices='auto')
    trainer.fit(compiled_model, datamodule=datamodule)

or just put the torch.compile inside the LightningModule init method:
class LitModel(L.LightningModule):
    def __init__(self):
        super().__init__()
        self.model = torch.compile(create_model('resnet18', num_classes=10))
        self.criterion = nn.CrossEntropyLoss()

Just an FYI; I think if you only compile the nn.Module within a LightningModule, lightning's load_from_checkpoint no longer works because the keys in the state_dict have been renamed: i.e. from torch_model.decoder.embed.weight to torch_model._orig_mod.decoder.embed.weight, causing missing and unexpected key errors when trying to load from checkpoint.

On Lightning 2.0.6 or nightly with Pytorch nightly (2023-08-03), I now have a different error:
Traceback (most recent call last):
  File "/home/plekhac/crocoman-toy/tmp2.py", line 72, in <module>
    main()
  File "/home/plekhac/crocoman-toy/tmp2.py", line 68, in main
    trainer.fit(compiled_model, datamodule=datamodule)
  File "/nfs/tools/3d/caiman/conda/envs/caiman-tmp/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 540, in fit
    call._call_and_handle_interrupt(
  File "/nfs/tools/3d/caiman/conda/envs/caiman-tmp/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 43, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/nfs/tools/3d/caiman/conda/envs/caiman-tmp/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 579, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/nfs/tools/3d/caiman/conda/envs/caiman-tmp/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 985, in _run
    results = self._run_stage()
  File "/nfs/tools/3d/caiman/conda/envs/caiman-tmp/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1026, in _run_stage
    self._run_sanity_check()
  File "/nfs/tools/3d/caiman/conda/envs/caiman-tmp/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1055, in _run_sanity_check
    val_loop.run()
  File "/nfs/tools/3d/caiman/conda/envs/caiman-tmp/lib/python3.10/site-packages/lightning/pytorch/loops/utilities.py", line 177, in _decorator
    return loop_run(self, *args, **kwargs)
  File "/nfs/tools/3d/caiman/conda/envs/caiman-tmp/lib/python3.10/site-packages/lightning/pytorch/loops/evaluation_loop.py", line 147, in run
    self._evaluation_step(batch, batch_idx, dataloader_idx)
  File "/nfs/tools/3d/caiman/conda/envs/caiman-tmp/lib/python3.10/site-packages/lightning/pytorch/loops/evaluation_loop.py", line 401, in _evaluation_step
    output = call._call_strategy_hook(trainer, hook_name, *step_kwargs.values())
  File "/nfs/tools/3d/caiman/conda/envs/caiman-tmp/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 307, in _call_strategy_hook
    output = fn(*args, **kwargs)
  File "/nfs/tools/3d/caiman/conda/envs/caiman-tmp/lib/python3.10/site-packages/lightning/pytorch/strategies/strategy.py", line 388, in validation_step
    return self.lightning_module.validation_step(*args, **kwargs)
  File "/nfs/tools/3d/caiman/conda/envs/caiman-tmp/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 321, in _fn
    return fn(*args, **kwargs)
  File "/home/plekhac/crocoman-toy/tmp2.py", line 56, in validation_step
    loss = self.common_step(x, y, 'val')
  File "/home/plekhac/crocoman-toy/tmp2.py", line 46, in common_step
    self.log(f'{stage}_loss', loss, on_epoch=True, prog_bar=True, logger=True)
  File "/nfs/tools/3d/caiman/conda/envs/caiman-tmp/lib/python3.10/site-packages/lightning/pytorch/core/module.py", line 398, in log
    apply_to_collection(value, dict, self.__check_not_nested, name)
  File "/nfs/tools/3d/caiman/conda/envs/caiman-tmp/lib/python3.10/site-packages/lightning/pytorch/core/module.py", line 399, in <resume in log>
    apply_to_collection(
  File "/nfs/tools/3d/caiman/conda/envs/caiman-tmp/lib/python3.10/site-packages/lightning/pytorch/core/module.py", line 445, in <resume in log>
    results.reset(metrics=False, fx=self._current_fx_name)
  File "/nfs/tools/3d/caiman/conda/envs/caiman-tmp/lib/python3.10/site-packages/lightning/pytorch/core/module.py", line 486, in <resume in log>
    results.log(
  File "/nfs/tools/3d/caiman/conda/envs/caiman-tmp/lib/python3.10/site-packages/lightning/pytorch/trainer/connectors/logger_connector/result.py", line 387, in log
    meta.sync = _Sync(_should=sync_dist, fn=sync_dist_fn, _group=sync_dist_group, rank_zero_only=rank_zero_only)
  File "/nfs/tools/3d/caiman/conda/envs/caiman-tmp/lib/python3.10/site-packages/lightning/pytorch/trainer/connectors/logger_connector/result.py", line 395, in <resume in log>
    raise MisconfigurationException(
  File "/nfs/tools/3d/caiman/conda/envs/caiman-tmp/lib/python3.10/site-packages/lightning/pytorch/trainer/connectors/logger_connector/result.py", line 395, in <resume in log>
    raise MisconfigurationException(
lightning.fabric.utilities.exceptions.MisconfigurationException: You called `self.log(val_loss, ...)` twice in `validation_step` with different arguments. This is not allowed

This is the same for both my own model and the snippet above.

Closing, this was fixed in PyTorch


Add LocalizationResourceManager Documentation (Updated to 1.1.0)

Fixes #83
Docs for Toolkit PR #216
What changes to the docs does this PR provide?
Added new page
PR Checklist
Please check if your PR fulfills the following requirements:

[x] For new pages, used the provided template.
[x] For new features, added an entry in the Table of Contents.
[x] Ran against a spell and grammar checker.
[x] Contains NO breaking changes.

Other information

Docs Build status updates of commit 98cdc7f:
:warning: Validation status: warnings

File
Status
Preview URL
Details

docs/helpers/localizationresourcemanager.md
:warning:Warning
View
Details

docs/TOC.yml
:white_check_mark:Succeeded
View

docs/extensions/translateextension.md
:white_check_mark:Succeeded
View

docs/helpers/localizationresourcemanager.md

Line 11, Column 224: [Warning-file-not-found] Invalid file link: 'localizedstring.md'.
Line 70, Column 3: [Warning-file-not-found] Invalid file link: 'localizedstring.md'.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit 441d816:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

:x:Error

Details

[Error-RuningBuildFailed] Some unexpected errors happened when running build, please open a ticket in https://SiteHelp and include the error report for our team to troubleshoot

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit 0247ac4:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/TOC.yml
:white_check_mark:Succeeded
View

docs/extensions/translateextension.md
:white_check_mark:Succeeded
View

docs/helpers/localizationresourcemanager.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit 36e8969:
:warning: Validation status: warnings

File
Status
Preview URL
Details

docs/helpers/localizationresourcemanager.md
:warning:Warning
View
Details

docs/TOC.yml
:white_check_mark:Succeeded
View

docs/extensions/translateextension.md
:white_check_mark:Succeeded
View

docs/helpers/localizationresourcemanager.md

Line 11, Column 224: [Warning-file-not-found] Invalid file link: 'localizedstring.md'.
Line 70, Column 3: [Warning-file-not-found] Invalid file link: 'localizedstring.md'.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit ed6f1fa:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

:x:Error

Details

[Error-PullRequestMergeConflict] Cannot sync git repo to specified state. Please make sure the pull request is mergeable.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit 728e552:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/TOC.yml
:white_check_mark:Succeeded
View

docs/extensions/translateextension.md
:white_check_mark:Succeeded
View

docs/helpers/localizationresourcemanager.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


Fix comment regarding multiple nood pools

Comment was not clear as we are creating a node pool with 2 nodes and the comment talks about "single-node". I think it should be "single node pool" instead.

@jbpaux : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@zr-msft
Can you review this PR?
IMPORTANT: When this content is ready to merge, you must add #sign-off in a comment or the approval may get overlooked.
#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

Thank you for the feedback.
LGTM
#sign-off

#assign @MGoedtel


Model for removing instruments?

Hi tsurumeso, is there any chance you can make a model for removing instruments (instruments-remover)? So we can train such a model with a code like we can train with your current vocal remover model.
The other models/codes I found require much GPU power and your vocal remover model is the only one which my GPU can handle very well.
Thanks.

but it already seperates the vocals and insturments...


System settings for staff users

I created a Staff User, but it does not have access to "System settings" options although I selected the permissions for Knowledge base management. I do not know if it is my problem.
Thanks in advance.

Thank you for the bug report Andrés, and my apologies for the inconvenience. The problem is caused by presenting the "System settings" menu option based on user.is_superuser, not on user.is_staff. The work-around is for staff to access the Django admin manually.
To provide the diegetic navigation path to all potential admin permissions wielders we'll have to  change more than just the one permission check, however. Since we can no longer assume a superuser's blanket auth'z is in effect, we'll need to evaluate perms for all the links the intermediate view would potentially display. At least we'll have the opportunity to fill one of the gaps in our coverage! 😉
Since the temporary work-around is relatively easy, this will have to wait until some more costly issues ahead of it in our queue are finished off first. I expect it to make the next release, but still have to advise that staff users enter the /admin (or custom AdminSite.site_url) path into their browser by hand... Sorry we can't deliver an immediate turnaround on this one, but thanks for your patience as we work our way through this minor-version-worthy change set! 😊

Permissions need an overhaul so might need to look at this alongside #239 for the next release. Sorry about the inconvenience, but please continue to use the workaround of going directly to the /admin path if you are a plain staff user (not superuser).

Maybe it is better to document this in the documentation instead to do some changes?

@Mte90 I suspect this might have actually been fixed already, I think I remember updating code to check for staff and not superusers at one point. We'll need to doublecheck.


WIP part 2 : refactoring read_mds

starting the conversation here

Codecov Report

Merging #89 into master will increase coverage by 0.11%.
The diff coverage is 92.44%.

@@            Coverage Diff             @@
##           master      #89      +/-   ##
==========================================
+ Coverage   91.77%   91.88%   +0.11%
==========================================
  Files           4        4
  Lines         644      789     +145
  Branches      143      178      +35
==========================================
+ Hits          591      725     +134
- Misses         33       39       +6
- Partials       20       25       +5

Impacted Files
Coverage Δ

xmitgcm/mds_store.py
91.47% <100%> (ø)
:arrow_up:

xmitgcm/utils.py
91.84% <92.39%> (+0.36%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e339d9b...f3ba4fa. Read the comment docs.

I am a bit confused that this PR contains many of the same changes that were in #87. That makes it very hard to review.
Could you try rebasing on upstream/master to see if this goes away?

Done! I also found a bug in my new read_mds (corrected in the newest branch) but I'd like to put the fix here so it's cleaner.

I'm pretty sure this would work:

Delete your fork on github
Fresh fork xgcm/master
clone your fork locally
git remote add upstream xgcm...
git checkout -b refactor-mds
copy in the files from your OTHER local orphaned copy of xgcm
git diff to check it looks ok
git commit; git push origin refactor-mds

Now, if you want to skip the deleting fork part, you can start from the git checkout -b step as long as whatever branch you are starting from is completely up to date with the upstream/master. You should be able to do git rebase upstream/master on your master branch and not get any conflicts or weird errors. It should fast-forward.

If you have replaced this with #93, could you please close this? I want to start reviewing but I want to make sure I'm reviewing the right thing.


personal functions

add personal dir to functions that is ignored by git.  this is needed for using personal analysis functions to get them on path during scc batch. ds/functions/personal

add docs on this too

done adding it to gitignore on dev

'''
ui_command =  matlab -nodisplay -nosplash -r
m_command =  sim_job1
cwd =  /projectnb/crc-nak/erob/batchdirs/study_HH_varyI_cluster_loop_comp_par
HOSTNAME =  scc-pe8.scc.bu.edu
JOB_NAME =  3408775
JOB_NAME =  study_HH_varyI_cluster_loop_comp_par_sim_job1
                        < M A T L A B (R) >
              Copyright 1984-2014 The MathWorks, Inc.
                R2014a (<IP_ADDRESS>2) 64-bit (glnxa64)
                         February 11, 2014

To get started, type one of these: helpwin, helpdesk, or demo.
For product information, visit www.mathworks.com.
SimIDs =
 1

Starting parallel pool (parpool) using the 'local_3408775' profile ... >> finished
'''
'''{Error using parallel.Cluster/parpool (line 89)
Failed to start a parallel pool. (For information in addition to the causing
error, validate the profile 'local_3408775' in the Cluster Profile Manager.)
Error in sim_job1 (line 40)
parpool(c,2);
Caused by:
Error using parallel.internal.apishared.ConnMgrBuilder.buildForCJS (line
113)
MatlabPoolPeerInstance{fUuid=997abb97-6af1-4c3d-81b9-82ba5127d213,
fGroupUuid=e4835f72-40bf-496a-80cd-a60c3af916ac, fLabIndex=-1,
fNumberOfLabs=-1} could not bind a ServerSocketChannel on scc-pe8 to port
27372; it failed with a JVM Exception: Invalid argument
}
'''

'''
ui_command =  matlab -nodisplay -nosplash -r
m_command =  sim_job2
cwd =  /projectnb/crc-nak/erob/batchdirs/study_HH_varyI_cluster_loop_par
HOSTNAME =  scc-bb6.scc.bu.edu
JOB_NAME =  3408766
JOB_NAME =  study_HH_varyI_cluster_loop_par_sim_job2
                        < M A T L A B (R) >
              Copyright 1984-2014 The MathWorks, Inc.
                R2014a (<IP_ADDRESS>2) 64-bit (glnxa64)
                         February 11, 2014

To get started, type one of these: helpwin, helpdesk, or demo.
For product information, visit www.mathworks.com.
SimIDs =
 2

Starting parallel pool (parpool) using the 'local_3408766' profile ...

finished
'''
{Error using parallel.Cluster/parpool (line 89)
Failed to start a parallel pool. (For information in addition to the causing
error, validate the profile 'local_3408766' in the Cluster Profile Manager.)

Error in sim_job2 (line 40)
parpool(c,2);
Caused by:
Error using parallel.internal.pool.InteractiveClient/start (line 330)
Failed to start pool.
Error using save
Unable to write to MAT-file
/usr3/graduate/erob/.matlab/local_cluster_jobs/R2014a/Job15/Task1.in.mat
    File may be corrupt.

}

'''
'''

{Error using parallel.Cluster/parpool (line 89)
Failed to start a parallel pool. (For information in addition to the causing
error, validate the profile 'local_3408786' in the Cluster Profile Manager.)

Error in sim_job2 (line 40)
  parpool(c,2);

Caused by:
    Error using parallel.internal.pool.InteractiveClient/start (line 330)
    Failed to start pool.
        Error using parallel.Cluster/createCommunicatingJob (line 82)
        Unable to write to MAT-file
        /usr3/graduate/erob/.matlab/local_cluster_jobs/R2014a/Job19.in.mat

        File may be corrupt.
}


make registry.go more idiomatic

Completly untested, so use with caution.
I left two TODOs in there:

I'm not sure if the defer on the node process does something. Does is wait for the node process to stop when the main program exits? Not sure.
Is ffjson with the map faster instead of using the standard lib with a struct? You could drop a dependency this way. Or use ffjson with a struct and generate the mapping file.

I used defer to make Wait call asynchronous, so we can run both node server and go proxy processes.
I think it's slightly faster, I don't want to drop this dependency. I have no experience in go, so maybe you'd like to fix it with struct?

defer means the function is run after the containing function exits. "go" runs the function concurrently. cmd.Start already runs the process concurrently. Wait just waits for it to finish (https://golang.org/pkg/os/exec/#Cmd.Wait). So you could drop this part or use go and some logic to restart node if it fails.

How much faster is it and do you need it to be faster? I personally would use the std lib as much as possible unless you really need to speed. To have that argument there should be some benchmark to show the speed difference.

I added a few more commits on top with dropping ffjson for encoding/json.
I don't have Postgres installed so I didn't run tests. If you want to take the commit which drops ffjson you need to update the Godeps file as well (never used that tool so I didn't do that either).

I did some benchmarking.  The big json speedup is moving from a map to a struct.  The json->ffjson step doesn't make much difference.  I recommend sticking with the standard library for now.

I've deployed the code and there isn't really a difference:
https://www.dropbox.com/s/mu2c19vi0s0adbw/Screenshot 2016-04-08 17.07.09.png?dl=0
The code is cleaner though, so thank you :)


Load pre-trained AlexNet ONNX official model

Note: Providing complete information in the most concise form is the best way to get help. This issue template serves as the checklist for essential information to most of the technical issues and bug reports. For non-technical issues and feature requests, feel free to present the information in what you believe is the best form.
For Q & A and discussion, please start a discussion thread at https://discuss.mxnet.io
Description
Using mxnet ONNX module to load pre-trained AlexNet model obtained from ONNX official website, but got an error message.
Environment info (Required)
What to do:
1. Download the diagnosis script from https://raw.githubusercontent.com/apache/incubator-mxnet/master/tools/diagnose.py
2. Run the script using `python diagnose.py` and paste its output here.
----------Python Info----------
Version      : 3.5.2
Compiler     : GCC 5.4.0 20160609
Build        : ('default', 'Jul 10 2019 11:58:48')
Arch         : ('64bit', 'ELF')
------------Pip Info-----------
Version      : 19.2.2
Directory    : /usr/local/lib/python3.5/dist-packages/pip
----------MXNet Info-----------
Version      : 1.4.0
Directory    : /home/leicongl/.local/lib/python3.5/site-packages/mxnet
Commit Hash   : a03d59ed867ba334d78d61246a1090cd1868f5da
Library      : ['/home/leicongl/.local/lib/python3.5/site-packages/mxnet/libmxnet.so']
Build features:
No runtime build feature info available
----------System Info----------
Platform     : Linux-4.15.0-58-generic-x86_64-with-Ubuntu-16.04-xenial
system       : Linux
node         : llc-dev
release      : 4.15.0-58-generic
version      : #64~16.04.1-Ubuntu SMP Wed Aug 7 14:10:35 UTC 2019
----------Hardware Info----------
machine      : x86_64
processor    : x86_64
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                16
On-line CPU(s) list:   0-15
Thread(s) per core:    2
Core(s) per socket:    8
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 158
Model name:            Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz
Stepping:              12
CPU MHz:               2418.735
CPU max MHz:           5000.0000
CPU min MHz:           800.0000
BogoMIPS:              7200.00
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              16384K
NUMA node0 CPU(s):     0-15
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d arch_capabilities
----------Network Test----------
Setting timeout: 10
Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0019 sec, LOAD: 1.8096 sec.
Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 0.0166 sec, LOAD: 2.0959 sec.
Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0157 sec, LOAD: 1.5621 sec.
Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.0067 sec, LOAD: 2.2819 sec.
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0017 sec, LOAD: 5.2355 sec.
Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0138 sec, LOAD: 1.8591 sec.
----------Environment----------
KMP_DUPLICATE_LIB_OK="True"

Package used (Python/R/Scala/Julia):
(I'm using ...)
For Scala user, please provide:

Java version: (java -version)
Maven version: (mvn -version)
Scala runtime if applicable: (scala -version)

For R user, please provide R sessionInfo():
Build info (Required if built from source)
Compiler (gcc/clang/mingw/visual studio):
MXNet commit hash:
(Paste the output of git rev-parse HEAD here.)
Build config:
(Paste the content of config.mk, or the build command.)
Error Message:
(Paste the complete error message, including stack trace.)
Minimum reproducible example
(If you are using your own code, please provide a short script that reproduces the error. Otherwise, please provide link to the existing example.)
Steps to reproduce
(Paste the commands you ran that produced the error.)

Link to the model, https://github.com/onnx/models/blob/master/vision/classification/alexnet/README.md

mxnet code used to load

import mxnet.contrib.onnx as onnx_mxnet
import numpy as np
import mxnet as mx
from PIL import Image
from collections import namedtuple

alexnet_path = "/path/to/file"
alexnet_tensor = np.random.random([10, 3, 224, 224])
sym, arg_params, aux_params = onnx_mxnet.import_model(alexnet_path)
alexnet_tensor = mx.nd.array(alexnet_tensor)
mod = mx.mod.Module(symbol=sym, data_names=['actual_input_1'], label_names=None)
mod.bind(for_training=False, data_shapes=[('actual_input_1', alexnet_tensor.shape)])
mod.set_params(arg_params=arg_params, aux_params=aux_params, allow_missing=True, allow_extra=True)
Batch = namedtuple('Batch', ['data'])
mod.forward(Batch([alexnet_tensor]))
mx_out = mod.get_outputs()[0][0][0]

error message

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/xxx/.local/lib/python3.5/site-packages/mxnet/contrib/onnx/onnx2mx/import_model.py", line 59, in import_model
    sym, arg_params, aux_params = graph.from_onnx(model_proto.graph)
  File "/home/xxx/.local/lib/python3.5/site-packages/mxnet/contrib/onnx/onnx2mx/import_onnx.py", line 115, in from_onnx
    mxnet_sym = self._convert_operator(node_name, op_name, onnx_attr, inputs)
  File "/home/xxx/.local/lib/python3.5/site-packages/mxnet/contrib/onnx/onnx2mx/import_onnx.py", line 61, in _convert_operator
    op_name, new_attrs, inputs = convert_map[op_name](attrs, inputs, self)
  File "/home/xxx/.local/lib/python3.5/site-packages/mxnet/contrib/onnx/onnx2mx/_op_translations.py", line 434, in reshape
    reshape_shape = list(proto_obj._params[inputs[1].name].asnumpy())
KeyError: 'concat1'

Model graph structure

What have you tried to solve it?

@vandanavk please take a look

@zachgk assign [@oorqueda ]

@oorqueda Please comment if you want me to assign this issue to you (I can only assign it to users who have commented on an issue)

I got the same error "KeyError: 'concat1'", how can you solve it

@szha  @Zha0q1 Has this issue been solved?


Large ruleset not highlighted in Atom

v3.4.0
Atom 1.33.0
https://github.com/Yara-Rules/rules/blob/master/Packers/peid.yar (2.62 MB) is not highlighted when opened in atom, even though YARA is automatically detected.

Works in VS Code with https://github.com/infosec-intern/textmate-yara

Probably due to https://github.com/atom/atom/issues/16916
Converting to tree-sitter would fix


Upgrade all dependencies

It should also fix the annotation warning

The set-output command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

No changeset bot comment, probably need to add this repo to the github app.


Scala Optimizers can cause seg faults

Description
In Scala, many of the optimizers cause a seg fault when used in conjunction with a FeedForward. Looked into this and it's because they are recreating native resources during execution which are getting disposed of by the jvm. This happens in the 1.4.0 release.
Minimum reproducible example
The following Scala code will reproduce this for the Adam optimizer.
object TestAdamBug {
  // Simple MLP network
  def mlpNetwork(): Symbol = {
    val input = Symbol.Variable("data")
    val label = Symbol.Variable("label")
    val fc1 = Symbol.FullyConnected(name = "fc1")()(Map("data" -> input, "num_hidden" -> 128))
    val act1 = Symbol.Activation(name = "relu")()(Map("data" -> fc1, "act_type" -> "relu"))
    val fc2 = Symbol.FullyConnected(name = "fc2")()(Map("data" -> act1, "num_hidden" -> 1))
    val loss = Symbol.LinearRegressionOutput(name="loss")()(Map("data" -> fc2, "label" -> label))
    loss
  }

  def getNDArrayIter(): NDArrayIter = {
    val f = NDArray.zeros(100, 20, 20)
    val l = NDArray.zeros(100, 1)
    val data = Array(f)
    val labels = Array(l)
    val batchSize = 10
    val iter = new NDArrayIter(data, labels, batchSize)
    iter
  }

  def main(args: Array[String]): Unit = {

    val net = mlpNetwork()
    val iter = getNDArrayIter()
    val optimizer = new Adam(0.001f, 0.9f, 0.999f, 1e-8f, 1 - 1e-8f, 0f, 10f, null);
    val init = new Normal(0.01f);

    val model = FeedForward.newBuilder(net)
      .setContext(Array(Context.cpu()))
      .setInitializer(init)
      .setNumEpoch(100000)
      .setOptimizer(optimizer)
      .setTrainData(iter)
      .setEvalData(iter)
      .build();
  }
}

@mxnet-label-bot add [Scala, bug]

@mxnet-label-bot add [Scala, Bug]


Installation process onto shared hosting

I have seen that you need to build the code especially if you want to brand and add your own filters etc but I am using a shared hosting account(UKhost4U). Is there another way to do this as I cant see how this can be done easily.....I was hoping for a download and upload to hosting via FTP process.
Can anyone help? Thanks

Hello, looks like this is what you are looking for.  This is located in our wiki.

Hi, The only problem is that is for built version of the career portal (therefore without my custom added extra filter fields and extra job field added).
As you mentioned in #165 I need to clone the site and then build but I cannot build (or at least I dont know how to on shared hosting). Unless there is another way to build it with my changes and then upload

In the link that I provided there, it also includes the steps for building.  This build process generates files used for static hosting.  You can find the steps for building here.


Use mouseover instead of mousemove event

I just learned that mouseover bubbles too...

Fixed in 0.3.0


MINOR: migrate GenericAvroIntegrationTest to TTD

Continue the work of #242 , #245 , #246 , and #248 by establishing a pattern for GenericRecord Avro tests to use TopologyTestDriver.

Thanks, @bbejeck. I agree, and this is part of the reason I was thinking to push the serde changes upstream. Having to use different serdes in testing vs. production seems sketchy.

@bbejeck Ah. This PR is targeted for the -post branch. I can only propose that SR change for not-yet-released versions (obviously), so I think we need to consider merging this code as-is, and then pursue an upstream change for future versions.
Does that sound good?

Thanks for the review. Merging now...


ui: refactor get api params in forms

Description
Refactor code getting API params for APIs in UI forms.
Added a new util plugin in plugins.js
Fixes some dysfunctional forms such as AddUser.vue

Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[x] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[x] Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale

[ ] Major
[ ] Minor

Bug Severity

[ ] BLOCKER
[ ] Critical
[ ] Major
[ ] Minor
[ ] Trivial

Screenshots (if appropriate):
How Has This Been Tested?

Based on testing and feasibility either of this PR or #5063 can be merged.
@blueorangutan ui

@shwstppr a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

UI build: :heavy_check_mark:
Live QA URL: http://qa.cloudstack.cloud:8080/client/pr/5064 (SL-JID-193)

@blueorangutan ui

@shwstppr a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

UI build: :heavy_check_mark:
Live QA URL: http://qa.cloudstack.cloud:8080/client/pr/5064 (SL-JID-194)

@blueorangutan ui

@shwstppr a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

UI build: :heavy_check_mark:
Live QA URL: http://qa.cloudstack.cloud:8080/client/pr/5064 (SL-JID-195)

@blueorangutan ui

@shwstppr a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

@blueorangutan ui

@blueorangutan ui

@shwstppr a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

UI build: :heavy_check_mark:
Live QA URL: http://qa.cloudstack.cloud:8080/client/pr/5064 (SL-JID-263)

@shwstppr is this ready for review/testing/merge?

@rhtyd yes, this is ready


Table header first-child  is sticky by default

The table header first-child <th> tag is sticky and has left-0 by default. I think this should be optional (maybe a new CSS class) because it doesn't work well with responsive layouts when you don't want a sticky first column.
https://github.com/saadeghi/daisyui/blob/91ef44f1814fbea969af5f1fc50886a2fe611f8d/src/components/unstyled/table.css#L3-L

It will look better if the first column of each row be <th> (not <td>)
Like the examples here: https://daisyui.com/components/table/
But you're right, it would be better if being sticky was optional. I will work on it

How about adding sticky option for <th> and the first column of each row?
There are some cases where both could be used in a large data table.

How about adding sticky option for <th> and the first column of each row? There are some cases where both could be used in a large data table.

I have solved with little override in my CSS
.table th:first-child {
     position: relative !important;
}
.table.sticky th:first-child {
    position: sticky !important;
    position: -webkit-sticky !important;
    left: 0px;
    z-index: 11;
}

Now if I want sticky I add class sticky on table

Same issue here, in my case if I don't add an empty th and a number then it takes the first column and make it sticky when scrolling horizontally. This does not make sense. Please fix this issue, shouldn't be that hard to fix


[WIP] Improve the AWS EMR provisioning documentation

[x] Attaching policy to create roles
[ ] Formatting
[ ] Describe the steps more clearly.

Resolved by https://github.com/apache/systemds/pull/1222


Rename Azure Database Migration Service to Azure Database Migration Service to Azure Database Migration Service (classic)

Rename Azure Database Migration Service to Azure Database Migration Service to Azure Database Migration Service (classic)
ARM API Information (Control Plane)
MSFT employees can try out our new experience at OpenAPI Hub  - one location for using our validation tools and finding your workflow.

Azure 1st Party Service can try out the Shift Left experience to initiate API design review from ADO code repo.  If you are interested, may request engineering support by filling in with the form https://aka.ms/ShiftLeftSupportForm.
Changelog
Add a changelog entry for this PR by answering the following questions:

What's the purpose of the update?

[ ] new service onboarding
[ ] new API version
[ ] update existing version for new feature
[ ] update existing version to fix swagger quality issue in s360
[x] Other, please clarify - Rename Azure Database Migration Service to Azure Database Migration Service to Azure Database Migration Service (classic)

When are you targeting to deploy the new service/feature to public regions? Please provide the date or, if the date is not yet available, the month.
When do you expect to publish the swagger? Please provide date or, the the date is not yet available, the month.
By default, Azure SDKs of all languages (.NET/Python/Java/JavaScript for both management-plane SDK and data-plane SDK, Go for management-plane SDK only ) MUST be refreshed with/after swagger of new version is published. If you prefer NOT to refresh any specific SDK language upon swagger updates in the current PR, please leave details with justification here.

Contribution checklist (MS Employees Only):

[x] I commit to follow the Breaking Change Policy of "no breaking changes"
[x] I have reviewed the documentation for the workflow.
[x] Validation tools were run on swagger spec(s) and errors have all been fixed in this PR. How to fix?

If any further question about AME onboarding or validation tools, please view the FAQ.
ARM API Review Checklist

Applicability: :warning:
If your changes encompass only the following scenarios, you should SKIP this section, as these scenarios do not require ARM review.

Change to data plane APIs
Adding new properties
All removals

Otherwise your PR may be subject to ARM review requirements. Complete the following:

[ ] Check this box if any of the following apply to the PR so that the label "ARMReview" and "WaitForARMFeedback" will be added by bot to kick off ARM API Review.  Missing to check this box in the following scenario may result in delays to the ARM manifest review and deployment.

Adding a new service
Adding new API(s)
Adding a new API version
-[ ] To review changes efficiently, ensure you copy the existing version into the new directory structure for first commit and then push new changes, including version updates, in separate commits. You can use OpenAPIHub to initialize the PR for adding a new version. For more details refer to the wiki.

[x] Ensure you've reviewed following guidelines including ARM resource provider contract and REST guidelines. Estimated time (4 hours). This is required before you can request review from ARM API Review board.

[ ] If you are blocked on ARM review and want to get the PR merged with urgency, please get the ARM oncall for reviews (RP Manifest Approvers team under Azure Resource Manager service) from IcM and reach out to them.

Breaking Change Review Checklist
If you have any breaking changes as defined in the Breaking Change Policy, request approval from the Breaking Change Review Board.
Action: to initiate an evaluation of the breaking change, create a new intake using the template for breaking changes. Additional details on the process and office hours are on the Breaking Change Wiki.
NOTE: To update API(s) in public preview for over 1 year (refer to Retirement of Previews)
Please follow the link to find more details on PR review process.

As discussed offline, model validation errors will be fixed in another pr. And since this pr only update the document, SDK breaking change will be ignored.


Refactor Lexer

src/lexer.js is easily the weakest part of our codebase. It is built with a very imperative architecture with lots of state manipulation and is difficult to reason about and work on. I would like to make this work in a more functional manner, by creating a Stream object exposing several methods such as match(), consume(), consumeUntil(), and returnToken() which can be chained into a definition for a lexical structure. This would look something like:
function createTokenizer(ufmCode) {
    const stream = new Stream(ufmCode);
    return function () {
        return stream.match(/[0-9]/, // When a digit is matched
            () => stream.consume(/* first digit */).consumeUntil(/[^0-9]/ /* not a digit */)
                .returnToken(new Token(stream.consumedChars, constants.TYPE.NUMBER))
        ).match(/[a-zA-Z_]/, // When a letter or underscore is matched
            () => stream.consume(/* first letter */).consumeUntil(/[^a-zA-Z_]/ /* not a letter */)
                .returnToken(new Token(stream.consumeChars, constants.TYPE.IDENTIFIER))
        );
    };
}

const tokenizer = createTokenizer("1234test4321");
tokenizer(); // new Token("1234", constants,TYPE.NUMBER);
tokenizer(); // new Token("test", constants.TYPE.IDENTIFIER);
tokenizer(); // new Token("4321", constants,TYPE.NUMBER);

Implemented an interface similar to the one above. I'm not completely convinced that this is better than the old solution. The previous system was hard to follow, but it was clear what every line of code did.
This functional pattern is more novel and abstracted certainly. I like that the mechanics of the Stream are in a separate class which knows nothing of the Uniform language, while a subclass of it defines the lexical structure for Uniform itself. The logic of how Uniform is tokenized is clear while the implementation of it is handled by the Stream class.
Downside is that it certainly takes a little while to really get it. This is also very likely less performant as RegExs can't easily match halfway through a string in JavaScript, so the entire source code (possibly 1000s of characters) is trimmed one character at a time as it is lexxed, creating a new string each time. It didn't cause any issues for me, and we might find a way around it in the future, but it could result in performance issues.
Also I basically looked at a 300+ line function and decided to replace it with a 65 line statement.
Sawyer, I'll let you decide if that is actually better.

It appears to work, and I can follow it for now. My only issue with the really long function is the scalability. It could get out of hand in the future. We will worry about it if it gets to that point.


MNT: Re-render the feedstock [ci skip]

Hi! This is the friendly conda-forge-admin automated user.
I've re-rendered this feedstock with the latest version of conda-smithy (1.4.6) and noticed some changes.
If the changes look good, then please go ahead and merge this PR.
If you have any questions about the changes though, please feel free to ping the 'conda-forge/core' team (using the @ notation in a comment).
Remember, for any changes to the recipe you would normally need to increment the version or the build number of the package.
Since this is an infrastructural change, we don't actually need/want a new version to be uploaded to anaconda.org/conda-forge, so the version and build/number are left unchanged and the CI has been skipped.
Thanks!

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.


Enum Request - Warehouse Request (Table 5765) - Field 11 "Destination type"

Please implement the field 11 ("Destination type") in Table 5765 ("Warehouse Request") as Enum.

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. We will update this issue with information about availability.

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. We will update this issue with information about availability.


Gift Entry: Centralize State Management - Remaining Field Types and Widgets

WIs:
W-8185263
W-8209641
W-8209648
and
W-8246967
Critical Changes
Changes
Issues Closed
SFDO Elevate: Incomplete Payment record created if Data Import record not originally processed correctly
Community Ideas Delivered
New Metadata
Deleted Metadata

Great, Thanks Victor!
On Tue, Nov 17, 2020 at 8:28 AM Victor Oduyemi<EMAIL_ADDRESS>wrote:

@voduyemi commented on this pull request.
In src/classes/GE_GiftEntryController_TEST.cls
https://github.com/SalesforceFoundation/NPSP/pull/5987#discussion_r525299792
:

@@ -34,15 +34,15 @@

@group-content ../../ApexDocContent/GiftEntry.htm
@description Unit tests to cover methods in GE_GiftEntryController.
*/
-@isTest
+@IsTest

Updated!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/SalesforceFoundation/NPSP/pull/5987#discussion_r525299792,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ALCOJYPIEQMHSARLCJQH323SQKQA5ANCNFSM4TBDWD3Q
.

--
Ken Lewis
Senior Member of Technical Staff, Engineering | Salesforce.org
Mobile:<PHONE_NUMBER>

This work has my approval in its current state with the one follow up WI to wrap the parsing of the Additional Object JSON value in a try/catch.  Thanks for all of this work, everyone!  @vmartinov-sforg FYI this work might need a final review/approval from you since as the owner of the PR it looks like I can't approve it myself :)

@kenlewis-sfdo Work item created for the try/catch created. https://gus.lightning.force.com/lightning/r/ADM_Work__c/a07B0000008j1RxIAI/view


Current path is not displayed with Sierra

Hello, I've updated to Sierra recently and my zsh stopped to show my current path. Path is showing only when in a repo (relative).
iTerm2 3.0.5
Freshly updated oh-my-zsh
zsh 5.2.0
I'm using nicoulaj theme.

That's just how the nicoulaj theme works ¯\_(ツ)_/¯

Nope, before Sierra it worked well. Outside of any repo it showed path relative to home directory. And it still work properly on El Capitan.

That's not what I see on debian. Which zsh version did you use before?

#elcapitan
~❯ cd Downloads
~/Downloads❯ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.11.6
BuildVersion:	15G31

#sierra
❯ cd Downloads
❯ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.12
BuildVersion:	16A323

I used zsh installed in system:
# from El Capitan:
zsh --version
zsh 5.0.8 (x86_64-apple-darwin15.0)

It seems the VCS_INFO API changed on 5.2.0. Let me see how to proceed.

There is a bug in the VCS_INFO_nvcsformats where the msgs variable is made local. This commit fixes it, but the 5.2 release was made 6 days before: https://github.com/zsh-users/zsh/commit/4105f79a3a9b5a85c4ce167865e5ac661be160dc. I'll let the core team know, see if they push a revision.
In the meantime, you can delete that line yourself and it'll work again. To find the location of the file, run:
for d ($fpath); do
    [[ -f "$d/VCS_INFO_nvcsformats" ]] && print "$d/VCS_INFO_nvcsformats"
done

Thank you, all works now.

This is also happening to me using the pure theme after upgrading to macOS Sierra.

Have a look at this issue as well: https://github.com/robbyrussell/oh-my-zsh/issues/4745#issuecomment-258717241.

@alejandroiglesias That is because you are trying to change macOS's native copy of zsh, which it does not allow under the "rootless" system integrity protection. The solution is to install an updated copy of zsh from a different source (I use homebrew) and use that as your shell instead.
 Still, it's possible to change it from System Preference (details here), so it's not really that bad. My default shell is not /usr/local/bin/zsh and everything works great. 👍
A potentially safer solution would be to have bash as your default shell and have a bashrc that looks something like /usr/local/bin/zsh; exit, so zsh would run immediately and the terminal would still exit when it returns. But... eh.

@krs013 I'm already using the zsh from Homebrew. On my macOS user account preferences, I see /bin/zsh which is the default macOS one. But on terminal, I run zsh --version and I get the version from Homebrew. I'm a bit puzzled...

@alejandroiglesias That's not quite the same. zsh --version runs whatever zsh is on your path and returns the version, but that does not mean that the shell currently running is the same. If you want to check which shell is running, use echo $SHELL.

Upgrading zsh to 5.3.1 didn't fix the issue for me.
I fixed "refined" or "pure" using the following hack. If anyone has something more elegant, please provide it.
refined.zsh-theme
repo_information() {
inside_git_repo="$(git rev-parse --is-inside-work-tree 2>/dev/null)"
if [ "$inside_git_repo" ]; then
echo "%F{blue}${vcs_info_msg_0_%%/.} %F{8}$vcs_info_msg_1_git_dirty $vcs_info_msg_2_%f"
else
echo "%F{blue}%~%f"
fi
# echo "%F{blue}%~%F{blue}${vcs_info_msg_0_%%/.} %F{8}$vcs_info_msg_1_git_dirty $vcs_info_msg_2_%f"
}

I was also missing the current path while using the Pure theme. I fixed this by:

Changing my shell in the System Preferences > Users & Groups > advanced settings for my user account.
Installing Pure from npm: npm install --global pure-prompt.
Replacing in my .zshrc:

ZSH_THEME="pure"

with:
autoload -U promptinit; promptinit
prompt pure


RHBPMS-473: Set TCCL to workaround login failures with JSM enabled

jenkins please retest

jenkins retest this.

@porcelli I've amended your commit with a test to cover this change here: https://github.com/csadilek/uberfire/commit/b1d1e12922bd94677dff7c7a0b6643ae7afb3f41
Maybe you want to bring it in.

jenkins retest this


Feature: Ping knocker once they are let in

This PR attempts to implement issue #1031.

@joshtrivedi can you try to get YesBot to run and see if this is what you had in mind?


how to do a production deployment for a high IO situation

If I want to do a production deployment, what suggestions do you have?
The configurations I can find are all for single-instance deployment. I believe that when the size of stored data is large and the requests are high, I/O could become a bottleneck.
So do you have any recommended deployment methods similar to sharding?
I haven't read the code yet. If this is a very basic question, please don't hesitate to answer it. Thx.

For a large deployment, I would recommend:

Configure both clients and server to use zstandard compression. This reduces both network and disk traffic. Use fast storage, SSD-like or better. See how far this scales before optimising further.

Consider partitioning clients- run multiple instances with separate storage, if you hit disk IO bottlenecks and can't move to faster storage. You could for example configure clients that build different things to use separate bazel-remote instances, eg a cache server just for linux clients, another one for mac clients and so on. Or a cache server for trunk, another one for release branches etc. Or location based partitioning. The important requirement here is that clients should not switch between partitions during a single build.

Consider tiered servers, eg use one central bazel-remote server with a lot of storage, and have clients connect to multiple smaller bazel-remote instances which are configured to use the central bazel-remote server as a HTTP or gRPC backend proxy. As in (2), it is important that clients do not switch between cache servers during a single build.

Really appreciate your quick response.
So, for high I/O situations, bazel-remote needs to setup multiple instances on the server side and configure clients use different instances.  The concept of tiered servers is essentially the same idea.
I'm wondering do you have plans to implement sharding on the server side. This way, users would have a unified access point, and when expansion is needed, changes could be made only on the server side, making it more convenient to use.

https://github.com/buildbarn/bb-storage is written as both a routing and storage tool. https://github.com/buildbarn/bb-deployments/ has examples where it is used as a sharding frontend and you can choose to use Bazel-remote as the storage backend. There are deployments of Buildbarn reaching PB of storage.

Hey, I'm looking at setting up something similar for a fairly large environment at the moment.
I was wondering if there is any opinions on doing the setup described here?
Specifically:

deploying n shards of bazel-remote
pointing all of them to a shared blob s3-compliant storage
routing to a specific shard through nginx directives on the basis of the incoming request

I wonder how the above works in terms of @mostynb remark in:

The important requirement here is that clients should not switch between partitions during a single build.

I guess if we accept some downtime during scaling in or out then technically this is a feasible option?

I also wonder how that compares to what @moroten suggested with bb-storage above.  Would appreciate any input here.

I also wonder how that compares to what @moroten suggested with bb-storage above. Would appreciate any input here.

Instead of using nginx for routing, bb-storage frontend would be used. As bb-storage is protocol aware it can perform existence checking on GetActionResult against all the storages, depending on where the output blobs are expected to be located.

I'm not sure if it's appropriate to continue discussing bb-storage here, but I still want to ask more questions.
According to what you said, the bb-storage frontend will sequentially（or randomly） pass the GetActionResult request to the backend cache service after receiving the request, until it can retrieve the cache. Is that correct?
how about the UpdateActionResult,  will the traffic to the backend instances be balanced?

The sharding in bb-storage is done by digest, so bb-storage will only do read/write/find-missing calls to that single backend. Think of N nodes, then a blob with digest D is assigned to backend B=hash(D) mod N (but weighted if your shards have different sizes).
There are also configurations for splitting by instance name and mirroring, but this thread is mostly about sharding.

It's OK with me to discuss these kinds of setups here, but the bb-deployments repository might be a better place to look for examples. If there is a good example to point to, we can add a link in bazel-remote's README.md file.
I know of a few fairly large (but maybe not petabyte) cache server setups using a combination of bb-storage and bazel-remote, with bb-storage handling the sharding and disabling ActionResult validation in bazel-remote (because the bazel-remote instances don't know how to check if blobs exist on the other bazel-remote instances). IIRC bb-storage uses the prefix of SHA256 digests to define the sharding, and that gives some basic load-balancing between the shards (unless the client picks ActionCache keys that all share the same prefix).
I don't recommend using S3 blob storage, because I do not know if it is possible to implement LRU-like cache eviction in this setup, and other forms of cache eviction like basic TTL setups will cause build failures.
And as @moroten mentioned- you should use an REAPI-aware frontend, not something generic like nginx.

Okay, thanks a ton @mostynb.
I do have a follow up question on:

I don't recommend using S3 blob storage, because I do not know if it is possible to implement LRU-like cache eviction in this setup, and other forms of cache eviction like basic TTL setups will cause build failures.

I went through the code and I believe the current state is that bazel-remote with S3 would essentially copy the artifacts from S3 to a local storage, then it ensures that the local storage follows LRU and it does evict files from that storage -- however no eviction whatsoever is done from the S3 storage. Is this understanding correct?

For:

you should use an REAPI-aware frontend, not something generic like nginx.

Is there any existing config like this but with bazel-remote that any one of you (@mostynb , @moroten ) would be able to share?
I dug up the protos for the config so I can reverse engineer this but an example would be awesome.

I went through the code and I believe the current state is that bazel-remote with S3 would essentially copy the artifacts from S3 to a local storage, then it ensures that the local storage follows LRU and it does evict files from that storage -- however no eviction whatsoever is done from the S3 storage. Is this understanding correct?

Correct. Bazel-remote always uses a local disk cache, which it tries to keep disk usage under the --max_size setting using LRU eviction. It also optionally reads and writes blobs to proxy backends (s3, gcs, etc) but performs no cache eviction for the proxy backends. PRs to add this feature for individual proxy backend implementations would be welcome (please open an issue for discussion first).

Is there any existing config like this but with bazel-remote that any one of you (@mostynb , @moroten ) would be able to share?

I don't have one handy, but I think you should specify 'grpc' backends instead of 'local' as used in the example you linked to.

The backends are specified in common.libsonnet which in your case should be Bazel-remote.

Since there is no further discussion, I'll close this issue.
Thank you all for your replies!

Correct. Bazel-remote always uses a local disk cache, which it tries to keep disk usage under the --max_size setting using LRU eviction. It also optionally reads and writes blobs to proxy backends (s3, gcs, etc) but performs no cache eviction for the proxy backends. PRs to add this feature for individual proxy backend implementations would be welcome (please open an issue for discussion first).

I may have some cycles to burn on exploring LRU for the S3 backend sometime in the not to distant future.   I think the main issue is the need to scan the S3 bucket (which will be quite expensive for large volumes of objects).  am I otherwise missing something here?

I may have some cycles to burn on exploring LRU for the S3 backend sometime in the not to distant future. I think the main issue is the need to scan the S3 bucket (which will be quite expensive for large volumes of objects). am I otherwise missing something here?

Yes- I'm not sure if there's a good way to do that with S3.
Before they shut down, Zenly had a cache that had a TTL based solution for GCS, which might be interesting to copy into bazel-remote (behind a configuration option). IIUC whenever touching a blob on GCS it would also update some metadata that was used by some GCS-side "object lifecycle management". Maybe there's something similar for S3?
I have an archive of their repository here: https://github.com/mostynb/znly-bazel-cache

Interesting.  My use case is slightly different - I'm not using S3 itself, I'm using an "S3 compatible" solution on prem attached to Kubernetes.  I have to do some experiments from a performance standpoint to see if we can get away with just the S3 backing store and no disk cache (understanding that this is not something that works today)

Another point to consider is compression — it is not supported by the bb-storage front-end, but it is supported by bazel-remote. An interesting question arises: in this setup, will the bb-storage front-end correctly pass compressed requests to bazel-remote or not?

IIUC bb-storage will only make regular/uncompressed data requests, so any communication with bazel-remote would not be compressed. If bazel-remote was using the (default) compressed storage mode then bazel-remote would need to decompress the data from disk in every request before sending uncompressed data to back to bb-storage.


Research: Determine if Submission Process uses DB transactions

Investigate to see whether the Submission Process uses DB transactions. If not, create a new ticket to modify the submission process to make database transactions.

The submission process does indeed use a transaction when calling the copy_*_from_workspace_to_global procedures. In the SubmissionProcessService of easey-camd-services, the procedure calls are built in the buildTransaction method, then executed inside a single database transaction in the successCleanup method.
There are status updates made to other tables outside of the transaction (camdecmsaux.submission_queue, camdecmpsaux.em_submission_access,  camdecmpswks.monitor_plan, camdecmpswks.qa_supp_data, camdecmpswks.qa_cert_event, camdecmpswks.test_extension_exemption, camdecmpswks.emission_evaluation, & camdecmpswks.mats_bulk_file) so that the statuses can be set accordingly if the transaction fails.


CMS Post Publish Date widget can write invalid numbers of seconds, causing Jekyll build failure

In admin/config.yml, the Posts collection date field is configured as following:
- {label: "Publish Date", name: "date", widget: "datetime", format: "YYYY-MM-DD HH:MM:SS"}
Netlify CMS's datetime widget uses Moment.js formatting. Capital S in "HH:MM:SS" indicates "fractional seconds", allowing the widget to sometimes write times such as "03:06:84". The invalid number of seconds (84) causes Jekyll build failure.


Add pulse permission fixing, network mode setting & container name

Probably better to split these up in a few pull requests but was hoping to get initial thoughts on the various changes.  Happy to break these apart and also add updates to the readme if you're okay with the changes.
On the Pulse permissions I was hoping to allow something like what some of the linuxserver.io images allow where you can set the UID/GID to use as part of the runtime environment but that is beyond my docker skills at the moment.
I needed to set the network mode to host to get Zwift to find my WiFi enabled kicker:
https://support.wahoofitness.com/hc/en-us/articles/9211851310738-Using-KICKR-KICKR-BIKE-Wi-Fi

Thanks for the comments.  I'm going to close this pull request and rework into multiple smaller ones.


chore: removing dependencies on new:ing collections with Guava (refs #2111)

A small step removing some imports from Guava.
References
#2111
Submitter checklist

[x] The PR request is well described and justified, including the body and the references
[x] The PR title represents the desired changelog entry
[x] The repository's code style is followed (see the contributing guide)
[x] Test coverage that demonstrates that the change works as expected
[x] For new features, there's necessary documentation in this pull request or in a subsequent PR to wiremock.org

I think the failing Build and test / build (macos-latest, 17) (pull_request) here might be a because of a timing issue, a toggling test case maby?

Yes, that's almost certainly just a performance issue... we should probably bump the expectation up to 10 seconds or something to prevent it happening again. The problem it's checking for was making them take minutes, so that should be OK.

Created https://github.com/wiremock/wiremock/issues/2259 for the flake, merging:


More company logos

Add "Emoon" under the "Apps" heading
https://www.emoon.io/

Add "POAP" to "Apps" section
https://scan.poap.xyz/

Under the Apps section, add "EnCirca"
https://www.encirca.com/eth/

Under the Apps section, add "meVu"
https://mevu.bet/


Crash when open PanelSDK

You are using UIAlertView instead UIalertController inside the sdk. UIAlertView is deprecated from iOS13+ so your sdk is unusable. Could you fix it?
Terminating app due to uncaught exception 'NSObjectNotAvailableException', reason: 'UIAlertView is deprecated and unavailable for UIScene based applications, please use UIAlertController!' *** First throw call stack: (0x187eb0300 0x187bc4c1c 0x187d9fe68 0x18b5a2188 0x18b5a26e8 0x10286ec28 0x1028e8630 0x1028e7a34 0x102c7da8c 0x1027a5c74 0x10279f8d8 0x1027a9568 0x1027a95dc 0x102c2ec6c 0x103b69d10 0x103b6b18c 0x103b79d14 0x187e2b6b0 0x187e262c8 0x187e258f4 0x19223c604 0x18bff9358 0x1027a3824 0x187ca12dc) libc++abi.dylib: terminating with uncaught exception of type NSException

UIScene base application that is new yet not support UIAlertView so it crashes the app, you can remove the UIScene base application, you can use window base application.

Without UIScene all applications launched on iOS 13+ devices will have a black screen because is mandatory in iOS 13+ and Xcode 11. Moreover, starting June 30, Apple won't accept applications submitted with older versions than Xcode 11.
https://developer.apple.com/news/?id=03262020b
That change is mandatory to use your SDK

iPhone applications may not need to use UIScene.

So, Does your SDK support iPad OS? Can we develop universal apps (iPhone/iPad) with multi screen scene support?

In a future we will support iPad OS and  multi screen scene.

Thanks for your reply, could you please update this issue when that feature is ready?

Sure, thanks for your feedback.


Help to confirm understanding of forward_flashattn

Dear Authors and @yukang2017 ,
Thanks for the amazing work. I am trying to understand the following:
https://github.com/dvlab-research/LongLoRA/blob/2a33f37543038877c70e9a625a61dc72a71621d0/llama_attn_replace_sft.py#L24
Could you please help me to confirm if my following understanding is correct?
output_unpad = flash_attn_varlen_qkvpacked_func(
    # 2 from bsz*2*seq_len is the 2 half-heads for non-shift and shift groups.
    qkv=x_unpad[bsz*2*seq_len, num_heads//2, head_dim],
    cu_seqlens=[0, 2048, 4096, 6144, 8192, 9216, 11264, 13312, 15360, 16384],
    max_seqlen=group_size=2028,
    ...)

from https://github.com/dvlab-research/LongLoRA/blob/2a33f37543038877c70e9a625a61dc72a71621d0/llama_attn_replace_sft.py#L123
The result of the function call, output_unpad, is the attention weighted values. I was trying to look into the source code of flash_attn_varlen_qkvpacked_func to understand it, but I was blocked by the native code call flash_attn_cuda.varlen_fwd().
Could you please help confirm my following understanding. My guess is that flash_attn_varlen_qkvpacked_func is able to calculate attention weighted values for variant length sequences(local groups). The locations of each local group attention is given by cu_seqlens and local group attentions are calculated for [0, 2048], [2048, 4096], ..., [8192, 9216], ..., [15360, 16384] in this example. Note that some of the local groups(for example [8192, 9216], [15360, 16384]) may be less than max_seqlen(group_size), but flash_attn_varlen_qkvpacked_func can calculate for these variant lengths. Attentions along the sequence are calculated with causal attention mask and calculated separately for first half heads(for non-shifted groups) and second half heads(for shifted groups). Therefore, there is no information leak from shifted groups into non-shifted groups(There is potential information leak in forward_noflashattn, as mentioned in #90. I think information leak might be prevented if rolling back is done before attn_weights calculation, but it is not important, as flash attention is always used for real training). A latter far away token in a different local group is able to absorb the information from former token in another local group because of shifted group. And this ability is expected to be enlarged to cover longer distance over the layers. A former token will never be able to get information from a latter token because of causal attention mask. The attention weighted value is calculated separately for non-shifted and shifted local groups but the information will be considered together at FeedForward.
Thanks in advance for your time and help,
Cheng

Hi,
I think your understanding is correct. Thanks for your question.
Regards,
Yukang Chen

Thanks Yukang for your time and answer.


helm chart (and perhaps a terraform module)

I think it would be really awesome if django-cookiecutter would have an option helm_chart=y that would produce a secure helm chart (or rather helm chart values.yaml for a standard django-cookiecutter helm chart) to facilitate deployment to any cloud infrastructure out of the box.
That would take this project to a whole new level and once you have that one could easily add a terraform module to take it even a level higher.
Anybody having the same thoughts?
We have some custom helm charts we use for our django deployment that I am willing to make open source and we took some (a lot of) inspiration from https://gitlab.com/glitchtip/glitchtip-helm-chart

A lot of interesting insights!
https://gitlab.com/glitchtip/glitchtip-helm-chart/-/issues/5#note_451868819

@ayushin I'd love to help with this - however, I am a Kubernetes novice. I'll chip in where I can

Great, I'll make a start this week and we take it from there.

@ayushin It might take me a while to get up to speed - I don't want to make any impressions that I'd be some more of great productive member, but if I see where I can chip in, I will help

Hi! I work in an infrastructure (ops) team in a big governament agency in Brazil and we use K8S to support our applications. We developed our own Helm Chart to deploy internal Java applications. I have experience in developing Helm Charts and you can count on me to solve this issue.

Hi! I work in an infrastructure (ops) team in a big governament agency in Brazil and we use K8S to support our applications. We developed our own Helm Chart to deploy internal Java applications. I have experience in developing Helm Charts and you can count on me to solve this issue.

Also working on this now that I've gotten time to go back to Kubernetes.

Shooting for Promethius and Grafana with Custom Metrics for Celery horizontal scaling.
Going to keep Traefik as the Ingress

Also working on this now that I've gotten time to go back to Kubernetes.

Shooting for Promethius and Grafana with Custom Metrics for Celery horizontal scaling.
Going to keep Traefik as the Ingress

we already running a number of django projects internally with helm, i think i can make our charts open source.
the work would be to tailor them to the actual cookie-cutter configuration

we already running a number of django projects internally with helm, i think i can make our charts open source.
the work would be to tailor them to the actual cookie-cutter configuration

@ayushin expose them, even if no one picks it up (I don't know how to do that yet, still junior), it will help someone. Worst case scenario it helps someone out haha.

@ayushin expose them, even if no one picks it up (I don't know how to do that yet, still junior), it will help someone. Worst case scenario it helps someone out haha.

This would take the project to the next level

This would take the project to the next level

Hi, I am sorry I just realised that I published our internal helm charts in December but never posted the link.
https://github.com/apexlabs-ai/django-helm-charts

I've made some progress: https://github.com/grstein/cookiecutter-django
Now one can select yes to use_helm for cookiecutter and it will render a nginx helm chart (output of helm create). The problem is that helm uses Go Templating, which, in turn, uses the same double curly-brackets sintax. So, we need to use the {% raw %} and {% endraw %} to escape the variables within yaml files.
Next step is to port @ayushin's templates.

So, I guess there a number of things we should made configurable (some are already in the helm chart):

postgres deployment / external database
redis deployment / external database
celery worker/beat instance yes/no
extra env variables
extra env secrets (like AWS_... etc)
google cloud credentials

Also for the good order, our templates are based mainly on https://gitlab.com/glitchtip/glitchtip-helm-chart/

I'm updating the GlitchTip chart a little now. Eventually I'd like it not to be specific to GlichTip at all. My first priority is updating against a brand new helm create template. @ayushin that list is pretty close to what I'd like to improve myself, except maybe google cloud as that's vendor specific. Let me know any broad level improvements or ideas you come up with.

Also I think I'm going to get more consistent about template filename convention - service-purpose.
deployment-web
deployment-worker
hpa-web
hpa-worker
etc
If anyone strongly prefers the opposite, ie web-deployment and web-hpa let me know now and why.

Those are hard problems! :-) For some reason component-resource_type sounds more logical, I guess you are more interested to see the files grouped by component vs by the resource type.
Re Google Cloud - the cookiecutter-django allows you to configure google cloud or AWS out of the box, so an optional support for both makes a lot of sense for me, especially because you can't add it to the chart post-install - i.e. you need to add a mount point to the container etc
I am not sure if this project should be a part of cookiecutter - ideally we would make it independent as the chart itself won't be changing with each django project and it should be possible to refer to it's helm repository from any project using django-celery-redis-postgresql

component-resource_type follows the syntax in values.yaml better.

This is a good example too https://github.com/itswcg/django-helm

https://gitlab.com/glitchtip/glitchtip-helm-chart/-/merge_requests/1 has been updated for most of what I wanted done. I made directories for components.
One problem is that helm can't seem to correctly replace instead of update the deployments. Strangely, helm diff is very clear that the deploy contains changed immutable values and must destroy and create the deployments.  This is probably a helm bug. It can be worked around by destroying the deployments manually.

I finally removed every reference to GlitchTip now https://gitlab.com/burke-software/django-helm-chart/
This chart is being used to host 8 different applications including larger ones like Saleor now.

What's the status of this ? :)

Any update on this?

In my opinion is the end product of the project a container. There is nothing special to Django or this project that requires a specific solution. When some masters containers in general and is familiar with helm specification deploying on k8s is no big deal.
Since three years there has not been a PR so I think we need to close this as not plannend.

I agree with the above, it feels out of scope of the project. While I've used Terraform a few times, I don't have any real experience with helm charts, so I don't feel comfortable adding this to the project and maintaining this. We gave it some time, but realistically it won't happen unless someone steps up.
Thanks everyon who contributed, there might provide some useful links for folks who are interested.


Login with in pop window

Hi
i am trying to open the login page in separate popup window but its opening in same window so i want to know where should i put the display: 'popup' in this plugin.

This is a bug tracker, not a help forum. Use one of Cake's help forums.

but i think that's not the cake issue. i am using your plugin so that's why
ask from you did your library support the popup functionality ??
On Fri, Nov 30, 2018 at 7:57 PM ADmad<EMAIL_ADDRESS>wrote:

This is a bug tracker, not a help forum. Use one of Cake's help forums.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/ADmad/cakephp-social-auth/issues/55#issuecomment-443228709,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ANTPys-owjMSljX5t5UC9hTgQzbO7hNjks5u0UdtgaJpZM4Y7tbm
.

--
Regards
Ghufran Mehmood
Software Engineer
Mobile : +923459099879


chore(deps): update terraform-module

This PR contains the following updates:

Package
Type
Update
Change

terraform-ibm-modules/cos/ibm (source)
module
patch
7.5.1 -> 7.5.3

terraform-ibm-modules/kms-all-inclusive/ibm (source)
module
patch
4.8.4 -> 4.8.5

Release Notes

terraform-ibm-modules/terraform-ibm-cos (terraform-ibm-modules/cos/ibm)
v7.5.3
Compare Source
Bug Fixes

add fix for breaking change introduced to fscloud submodule in 7.5.2 (#564) (8fba3fc)

v7.5.2
Compare Source
Bug Fixes

added fix to fscloud profile submodule where value of create_cos_instance was being ignored (#563) (2d47517)

BREAKING CHANGE
Do not upgrade to this version if using the fscloud submodule as it contains a breaking change that will destroy COS resources. Please upgrade directly to v7.5.3

terraform-ibm-modules/terraform-ibm-kms-all-inclusive (terraform-ibm-modules/kms-all-inclusive/ibm)
v4.8.5
Compare Source
Bug Fixes

pull key protect module from hashicorp instead of github
gave key_protect_instance_name variable a default value and removed the associated validation logic (#430) (6ab4529)

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

/run pipeline

/run pipeline

:tada: This PR is included in version 2.12.2 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Improve large number readability

Proposal
Often I have queries like foo > 1000000.  This is hard to read and very easy to make a mistake between 100k, 1M, and 10M.  I would love to have some way formatting large numbers for increased clarity.
Option 1 - digit grouping - eg 1_000_000 or 1.000.000
Option 2 - engineering SI notation  - eg 1M or 900k or 2Ti

You can do foo > 10^6 or foo > 1000 * 1000

Absolutely there are some work around but one you move off easy powers of 10.  Let's say I wanted to alert on 90% of 2TiB,  2TiB != 2TB and there is a huge difference between the two.
My big concern is the readability and maintenance rather than strict compactness.  0.90 * 2TiB is clearer than 0.9 * 2 * 1_099_511_627_776 is clearer than 1.979 * 1000 * 1000 * 1000 * 1000 even though they're all equivalent.
A built in function like pow would also be pretty good and wouldn't require changing the parsing engine.  So something like 0.9 * parse_number("2Ti")

Indeed this can be done with 0.9*2*2^40. My concern with introducing a specific function for bytes would be where to draw the line: should we also parse centimeters, milliliters, tons, .. ? I think we can still push this to the user. Note that PromQL expression also support comments with # which could help reading queries.

yes, please
several AI tools even already suggest (allucinated suggestions :) to use M, G, Gi... as with https://docs.victoriametrics.com/metricsql/
K, Ki, M, Mi, G, Gi, T and Ti....

Grouping with _ already works (e.g. 1_000_000 is accepted as an input).
WRT prefixes, I'm torn. I would expect many, if not a majority of users, to read process_resident_memory_bytes > 1G as "more RSS than 1GiB", but that would be written as process_resident_memory_bytes > 1Gi. In contrast process_resident_memory_bytes > 10^9 vs. process_resident_memory_bytes > 2^30 or  process_resident_memory_bytes > 1024 * 1024 * 1024 leaves no doubt.

several AI tools even already suggest (allucinated suggestions :) to use M, G, Gi... as with https://docs.victoriametrics.com/metricsql/ K, Ki, M, Mi, G, Gi, T and Ti....

With all due respect, but this should never be a reason to introduce bad ideas into PromQL.

Hello from the bug scrub.
Discussed during the bug scrub and agreement that we should not add support for K, Ki, M, Mi, G, Gi, T and Ti style prefixes/suffixes.
Closing this issue as _ style grouping already works.


Add yew-validation crate

Description
This relates to PR #1344 and is the separation of the validation logic out of it into its own crate.
Checklist:

[ ] I have run ./ci/run_stable_checks.sh
[x] I have reviewed my own code
[x] I have added tests

Would be useful to add the tests for this to the CI.

Would be useful to add the tests for this to the CI.

Strange, could've sworn I had added those.. oh well, pushed!

@philip-peterson maybe we can have this live outside of Yew since it will be useful for other projects and is not yew specific?

I suppose we could, I suspect this crate will be deleted as soon as #1269 is solved though which I definitely think we can get done quickly. Reason I would prefer to keep this in Yew is that based on the PR comments I received it got taken in a direction I wouldn't want it to be as a standalone crate; I did create the public taginfo crate but it behaves contrary to the PR changes that were requested.

Alternatively we could solve #1269 first and then this crate won't be necessary.

Let's merge this for now and we can remove later if #1269 makes this obsolete


Query vLLM OpenAI /models endpoint to get model name and context window

Description
Similar to #755 , the OpenAI-compatible API server by vLLM exposes both the model name via id and the context length via max_model_len, which can be used to automatically setup the context length of the deployed LLM.
I'm not sure how to best check if the endpoint should be queried, e.g. by checking the API Base URL, or introducing a new apiType besides Azure, etc.
Checklist

[x ] The base branch of this PR is dev, rather than main it's preview as in your contributing guidelines
[ ] The relevant docs, if any, have been updated or created

References

vLLM PR adding max_model_len
#609 Looks like this is closely related to the "AUTODETECT" functionality. however, that only uses the model names, not additional parameters as far as I am aware of (ref. load.ts:244). So it may be better to adjust the listModels() interface to include model parameters in its response

sorry I messed something up, I only intended to include my one change in the OpenAI.ts file

@simon376 I think this might be a good enough reason to make a subclass of the OpenAI class (we do this with Deepseek, Groq, and others for example). Everything can be the same except that there doesn't need to be a URL check to find out whether it is an instance of vLLM

Hey @simon376 , any progress on this? Would love to get it shipped!

@sestinj  I've created a new vLLM subclass and add documentation for it. Sorry for the whitespace changes in OpenAI.ts, couldn't figure out how to get rid of them

Great, now worries on the whitespace this looks perfect

Hi @sestinj what version of the extension has this feature been integrated into?


Where unzip the dataset.

I use the next command to download the dataset.
python loader/download.py mini

That download the next zip files

When I run the command
./faster-rcnn.pytorch/run_train.sh

and get the next error
Loaded dataset `gta_det_train` for training
Set proposal method: gt
Preparing training data...
done
before filtering, there are 0 images...
after filtering, there are 0 images...
0 roidb entries
Loading pretrained weights from data/pretrained_model/resnet101_caffe.pth
Traceback (most recent call last):
  File "trainval_net.py", line 445, in <module>
    epoch, step))
NameError: name 'step' is not defined
root : 201_14_18895 Finish!

This happened too when I try to run the gen_dataset.py
PYTHONPATH=. python loader/gen_dataset.py gta val

and this give the error:
AssertionError: Not label files found in /workspace/3d-vehicle-tracking/3d-tracking/data/gta5_tracking/val/label

I suppose that this happened because the dataset has not been unzipped. But I don't know where to do this.
I unzip the files gta_3d_tracking_val_image_0001.zip and gta_3d_tracking_val_label_0001.zip in the directory 3d-tracking/data/gta5_tracking/val/{image,label} but that problem continue.
Thanks.

you should unzip all the files to the correct positions

Hi @TheLastImperial, have you solve the problem?
Please unzip all the zip files you downloaded to the designated location and you may found it working. Just following the readme GTA Dataset Download and extract our GTA dataset and put it under ./3d-tracking/data/gta5_tracking/{split}/{type}. Where split = {train, val, test} and type = {image, label}
Thank you @yanganlan0310 for explaining!


[fpga] Manage bootrom as Vivado-managed file

We currently include the bootrom vmem file without having it managed through Vivado. This needs a edalize update (https://github.com/olofk/edalize/pull/86), and https://github.com/imphil/opentitan/commit/75c644b7aec2888e33db4fdb0a8b9bc115d8be20. Let's wait for a couple more weeks to people pick up the edalize updates, and then move over.

Is there more to do here or can this be closed?

i was actually meaning to try this out, but i don't think whatever @imphil mentioned here has been done.

Well, we've definitely got the Edalize updates :-) See https://github.com/lowRISC/opentitan/pull/6426 for an attempt at using them.

@timothytrippel is this still relevant in the way we build things today?

No, this is actually more of a burden than a feature. fusesoc's dependency tracking is inferior to bazel's (it was simply meant for a much more limited scope), and it would be harmful for maintenance of software builds if the location of the software output were hard-coded in a core file.
I'd be in favor of closing and electing not to implement.


Logger timeout on apt-fast install

Operating System Version: Ubuntu 18.04.5
Deploying via (VirtualBox/VMWare/AWS/Azure/ESXi): ESXi
Vagrant Version (if applicable): 2.2.19

Please verify that you are building from an updated Master branch before filing an issue.
Description of the issue:

Running the apt-fast command manually on Logger shows that it is unable to resolve or connect to the apt-fast repository, though DNS resolves fine on the system. Bypassed this issue by manually changing the command to apt install and it worked to complete this step.

thank you. I was having the same issue with ansible logger hanging for a long time and apt worked