Fix Flash CLI: in case of converting functions to classes + JIT tracing tests

What does this PR do?

CI has been failing for Flash CI for the case when a function is converted to ClassFromFunction class. We were not checking if its type is subclass of CustomClassFunctionBase.
For JIT tracing, the model needs to be attached to the Trainer class, else it will give an error similar to: ImageClassifier is not attached to Trainer.
Baal integration fails on recent Lightning Trainer versions. on_predict_dataloader, on_train_dataloader, on_test_dataloader were deprecated, and removed in PL 1.7. I'm not very sure if we need them, removing that works?
For BaaL integration, probabilities need to have a check if it's not empty else torch.cat will fail.

This PR attempts to fix both of the issues. Note that we should document that the best way to JIT trace the model in Flash is to:
# 2. Build the task
model = ImageClassifier(backbone="resnet18", labels=datamodule.labels)

# 3. Create the trainer and finetune the model
trainer = flash.Trainer(max_epochs=3, gpus=torch.cuda.device_count())

# Attach trainer to the model object
model.trainer = trainer
trace = torch.jit.trace(model, torch.randn((3, 196, 196))

Issue: #1413
Before submitting

[ ] Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
[ ] Did you read the contributor guideline, Pull Request section?
[ ] Did you make sure your PR does only one thing, instead of bundling different changes together?
[ ] Did you make sure to update the documentation with your changes?
[ ] Did you write any new necessary tests? [not needed for typos/docs]
[ ] Did you verify new and existing tests pass locally with your changes?
[ ] If you made a notable change (that affects users), did you update the CHANGELOG?

PR review

[ ] Is this pull request ready for review? (if not, please submit in draft mode)

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.
Did you have fun?
Make sure you had fun coding 🙃

About the 5 failures left in the CI:

CI testing / pytest (ubuntu-20.04, 3.9, latest, stable, graph) (pull_request): This failure is flaky, and has been fixed in PL. Once there is another release of PL, we should be good.
CI testing / pytest (ubuntu-20.04, 3.9, latest, stable, audio) (pull_request): PR https://github.com/Lightning-AI/lightning-flash/pull/1409 is opened to fix it.
Lightning-AI.lightning-flash (Special): Horovod versioning error, but I also see /usr/bin/pip not found error. I plan to fix this in a separate PR, doesn't seem to be very urgent as the first step was to support PL 1.7 with Flash.

Thanks to @otaj for his help with a few failures related to PL 1.7, it was very helpful. ❤ Also @rohitgr7 for a fix! 🔥
cc: @ethanwharris @Borda @carmocca

Hi, @ethanwharris - The 5 failures are expected in the CI, only 1 of these is something that I haven't investigated yet. Fixes for others are either already merged in PL, or we have a PR opened in Flash. Could you please take a look again, and help merge if it all looks good? Thanks!


refactor log level and log file controls

Instead of --log_level DEBUG do this -vvv.

Maybe a option for machine-readble logs?


Request: Can add on two different discord server

As said in above, is it possible to have it on 2 or more server and only using 1 bot instead of creating a new one? I just have to add the bot in the other server? It's just a request. This is rare on other discord to fabric chat so this might be a good one reason to make more people use it.
Btw, still love the mod. 11/10

Unfortunately the current architecture of the mod doesn't support this, and it'd take major rewrite to do so. This feature will not be implemented.

Okay, thank you.


Here's what a failed push would look like

Tests will be the last bit for me to get working 😅 https://travis-ci.org/pamo/distributed/builds/443521835

@pamo can you fix the conflicts? I merged older pull requests first

Will do

🤔 the tests are still broken. Did you do a quick check to see if the app works?

@glauberramos np I should add a work in progress label or something so it doesn't happen again.

Sure, you can add an WIP - in front of the PR name. Another suggestion that I have is to create smaller PR's.


Unable to upgrade to v1.21.5+k3s1

Environmental Info:
K3s Version:  v1.21.5+k3s1
Node(s) CPU architecture, OS, and Version:
Linux  5.4.0-1044-raspi #48-Ubuntu SMP PREEMPT Thu Sep 9 15:24:01 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
Cluster Configuration:
1 master node, 1 node
Describe the bug:
Unable to upgrade from v1.21.4+k3s1 to v1.21.5+k3s1, k3s fails to start. Get the following output when trying to upgrade the master node:
$ curl -sfL https://get.k3s.io | sh -
[INFO]  Finding release for channel stable
[INFO]  Using v1.21.5+k3s1 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.21.5+k3s1/sha256sum-arm64.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.21.5+k3s1/k3s-arm64
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Skipping /usr/local/bin/kubectl symlink to k3s, already exists
[INFO]  Skipping /usr/local/bin/crictl symlink to k3s, already exists
[INFO]  Skipping /usr/local/bin/ctr symlink to k3s, already exists
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s
Job for k3s.service failed because the control process exited with error code.
See "systemctl status k3s.service" and "journalctl -xe" for details.

Output from systemctl status k3s:
Oct 05 19:06:39 rpi1 systemd[1]: Starting Lightweight Kubernetes...
Oct 05 19:06:39 rpi1 sh[2767]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Oct 05 19:06:39 rpi1 sh[2768]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
Oct 05 19:06:39 rpi1 k3s[2779]: time="2021-10-05T19:06:39Z" level=fatal msg="exec format error"
Oct 05 19:06:39 rpi1 systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
Oct 05 19:06:39 rpi1 systemd[1]: k3s.service: Failed with result 'exit-code'.

Steps To Reproduce:
curl -sfL https://get.k3s.io | sh -

Expected behavior:
To be able to upgrade
Actual behavior:
Unable to upgrade
Additional context / logs:
Couldn't find anyway around the issue, had to downgrade back to v1.21.4+k3s1 - which worked as expected.
Backporting

[ ] Needs backporting to older releases

Which model of Rasperry Pi is this on? The only thing I can think of that might have caused this is the bump of golang from 1.15.8 to 1.15.10, which mirrors upstream's change.

Hi @brandond, it's a Raspberry Pi 4 Model B Rev 1.2

Hmm. Our QA team tests each release on Raspberry Pi before it goes out so I'm a bit stumped. Can you reproduce this on demand?

I have the same issue on an amd64 machine running in Oracle Cloud, with same Ubuntu version. Ubuntu is installed in the minimal variation, and doesn't have network-manager installed at all (which I presume is the source of the nm-cloud-setup service).

The nm-cloud-setup bit is a red herring. The check in the systemd unit ensures that k3s won't start if it IS running. If it is not present or is disabled, startup will continue as normal. Look further along in your logs for whatever's going wrong.

The nm-cloud-setup bit is a red herring. The check in the systemd unit ensures that k3s won't start if it IS running. If it is not present or is disabled, startup will continue as normal. Look further along in your logs for whatever's going wrong.

Thanks. Looking further in the logs a line that was less obviously a problem turned out to be the actual problem :facepalm:.


Add types to composables/use-scroll.js

Description

Add type checking to use-scroll.js. This module has one untyped dependency that will be typed in #952. Make sure to add it to tsconfig.json's include list.
Blocked by #952
Additional context

Part of an ongoing effort to add type checking to parts of the project that can be type checked.
Implementation

[ ] 🙋 I would be interested in implementing this feature.

Closing this issue because use-scroll.js is going to be removed in #1705, and use-window-scroll.ts that is used instead is already using TypeScript.


Onsen go back issue

Hi, I am usin Onsen 2.9.1 with AngularJS 1, when I use the navigation and go to a page from the map, when I go back, the map is blank. Do you know what is the problem or how can I refresh it?
Regards,
J. Carlos

Please share your project files on GitHub

ping

ping

I can’t upload it, so you can close this question and I will investigate, thank you.
Enviado desde mi iPhone

El 13 feb 2018, a las 15:13, Masashi Katsumata<EMAIL_ADDRESS>escribió:
ping
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.


chore: fix typo on test description

Codecov Report

Merging #12 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff          @@
##           master    #12   +/-   ##
=====================================
  Coverage     100%   100%
=====================================
  Files           4      4
  Lines          62     62
  Branches       13     13
=====================================
  Hits           62     62

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1f92036...133d743. Read the comment docs.

:tada: This PR is included in version 1.1.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Markdoc module.exports pattern not working with Next.js 14.1.0

What happened?
When using markdoc it is common to have a file to export a single markdoc tag. For example markdoc/tags/step.markdoc.ts:

import { Tag, Node, Config } from '@markdoc/markdoc';

export const step = {
  render: 'Step',
  attributes: {
    title: {
      type: String
    },
   ...
  },

It also seems semantic to then have a markdoc/tags/index.ts file to re-export all of the tags from these different files with potentially fixed or different casing to enable kebab casing inside of the markdown files:

import { step } from './step.markdoc';

module.exports['step'] = step;

However, with Next.js 14.1.0 it seems this pattern is no longer working. I am unsure if this is a bug with Next.js or if this pattern should no longer be used:
Server Error
TypeError: Cannot set properties of undefined (setting 'step')
This error happened while generating the page. Any console logs will be displayed in the terminal window.
Source
markdoc/tags/index.ts (19:22) @ eval
  18 |
> 19 | module.exports['step'] = step;

If this is instead switched to using export instead of module.exports everything works.
export { step }

However this is not an ideal solutions for handling the renaming of exports to kebab case. For example
module.exports['mini-card'] = miniCard; // Valid js
export { ... }; // No nice way to export miniCard as mini-card

To reproduce
Shown above
Version
0.4.0
Additional context
No response

Thanks for reporting @ReeceHumphreys. Seems like this definitely could be on the next side of things, maybe in them changing a core dependency or something (e.g. Babel?).
If you have the chance, would greatly appreciate you looking into a solution for this in our @markdoc/next.js library—a good place to start is the loader, which calls this defaultObject function.

Hi @mfix-stripe , I am actually not using the @markdoc/next.js library. I am eventually just calling Markdoc.parse and friends which rules that the loader as the problem.
What is stranger about this issue is I have been unable to reproduce it with a simple Next.js starter either. It seems to be that the issue is that module.exports itself is undefined.

@ReeceHumphreys that seems like it could be a problem in Next itself?
Also for your particular use case above, instead of module.export['step'] you could do:
export const step = ...

Closing this issue since this is definitely more of a Next.js issue. However the solution given for step will not work for things like module.exports['mini-card'] since you cannot do:
export const mini-card = ...

This prevents you from defining custom markdoc tags containing hyphens. Our solution was just to use the syntax that @mfix-stripe suggested and to rename any tags that were using hyphens.

Thanks @ReeceHumphreys. Please follow up here if you resolve the Next.js issue—I'm curious myself.


Panel doesn't allign well when side bar is pinned to the right

See screen shot. The header of the editor aligns well with the header of the side bar however the header of the panel somehow 'stops' at the side bar without a nice visual ending.

The visual ending in the editor is actually the scroll bar. Notice the small space between the panel and the scroll bar in the editor, there the editor also does not have a nice ending - same as panel
Are you suggesting we should draw the border at the end of the panel header?

Forwarding to Bard so he gets the feedback and thinks about the solution

The only problem with doing that is we need to be ready to accommodate horizontal scrolling as well (which is why there is the gap).  I would suggest, however, that we draw it down anyway and make horizontal scrolling go as far as the vertical scroll bar.
Alternatively, we could do this (add stroke to the horizontal scrollbar area (not my favorite).

Another way we can make it happen is by having the vertical scrollbar take up all the space except on a hover event which makes room for a horizontal scroll bar.  I like this approach as well as the first approach I suggested, but what do you guys think?

@bgashler1 let's discuss this in our ux sync today


when executing model runs, suppress model output to avoid bogging down the instance.

https://github.com/IMMM-SFA/statemodify/issues/54
https://github.com/openjournals/joss-reviews/issues/6325


feature request: ignore duplicate files, only show unique

Hi, thank you for the plugin. I'm using transform_file_path in such a way that some files end up being duplicates of one another.
It would be nice to have an option like ignore_duplicates or show_unique_only to remedy this

Hi,
Thanks for the feedback.  It's strange though, I am using the transform_file_path in exactly the same (which could result in duplicates), and I haven't seen this issue. Even if I do this:
      transform_file_path = function(file)
        return "test.txt"
      end,

The picker shows test.txt just once.
Maybe the duplicates are showing because of something else?

Hmm, now that I test it again after clearing my vim.v.oldfiles, I can't reproduce it. Also, I have a path_display set for telescope, I might have had a path that was different but displaying the same.


AMD update 13.3 prompt Unable to repair.

Kernel->Patch[0] has different Find and Replace size (6 vs 5)!
Kernel->Patch[0] has ReplaceMask set but its size is different from Replace (6 vs 5)!
Kernel->Patch[1] has different Find and Replace size (6 vs 5)!
Kernel->Patch[1] has ReplaceMask set but its size is different from Replace (6 vs 5)!
Kernel->Patch[2] has different Find and Replace size (6 vs 5)!
Kernel->Patch[2] has ReplaceMask set but its size is different from Replace (6 vs 5)!
CheckKernel returns 6 errors!
Completed validating /Volumes/EFI/EFI/OC/config.plist in 2 ms. Found 6 issues requiring attention.

Sorry, I don't have any experience with this, so I may not be able to help you.
In my opinion, you should follow the prompts to modify and experiment accordingly.
In addition, OCAT's validation mechanism completely calls OC's "ocvalidate" program. You can try this on the relevant OC forums for help. Good luck.


Fix typo in Readme

Fixed with PR https://github.com/KaizeNodeLabs/SafeSwap/pull/21 by omahs https://app.onlydust.com/u/omahs


Add option to use on demand shared cache

Before that we supported only on demand exclusive cache.
Required part for https://github.com/gradle/gradle/pull/26660.
Reviewing cheatsheet
Before merging the PR, comments starting with

❌ ❓must be fixed
🤔 💅 should be fixed
💭 may be fixed
🎉 celebrate happy things

@bot-gradle test this

I've triggered the following builds for you. Click here to see all build failures.

PullRequestFeedback build

There is one issue with this implementation: we don't have a protocol to inform processes that share a lock that they need to release the lock. This protocol is more complex than what we have now and we won't implement it right now.
So I am closing this PR.


Proper playground code

After my PR was rejected in the old repo, would it make sense to reopen it here?
https://github.com/gobridge/gopher/pull/39

Sure! I'd be happy to review that here.

Thanks for reviewing and merging. Farewell!


New World API

If we'd had a 1.0, this would be 2.0.
The old API was ambiguous and confusing. Use of goTo could easily lead
to unintentionally having the same path on the backstack twice. The
difference between resetTo and replaceTo was not easy to understand
and the two were easily confused.
Those three methods are all now deprecated (along with the previously
deprecated goUp), and there are two new methods to replace them:

set(Path) will make the given path the top of your backstack,
inferring push or pop operations and dispatching with the inferred
Direction. This is your Just Go Here option.

set(Backstack, Direction) will simply throw away the current
backstack, install the given backstack, and dispatch with the given
direction. This is your Total Control option.

One doc nit. LGTM

:shipit:


Support loss accumulation on gpu

Speeding up ptb example.

LGTM. This code works well in my environment.


Remove config parents tagging, rename config field

Warning - This PR introduces breaking changes in the config karton task format/behavior and should be merged/deployed at the same time as https://github.com/CERT-Polska/karton-config-extractor/pull/35
Changes
The sample field in config tasks is renamed to executed_sample. The old name was very confusing, especially with the additional parent field. The current semantics should be easier to grasp:

executed_sample - the sample that was executed in order to extract this specific config (optional)
parent - the executable/dump that contains the necessary data for config extraction, i.e. you can run malduck on this file and get the config (optional)

Apart from that, karton-config-extractor now spawns tagging tasks for the executed_sample and parent on its own so we don't have to have this hacky tag appending in the reporter anymore
Testing plan
Setup new versions of karton-config-extractor and karton-mwdb-reporter in mwdb-playground and check that each scenario behaves the same way as it does currently:

[x] A config that is extracted from an executable - should be tagged with both {family} and ripped:{family}
[x] A config that is extracted from a dump file (analysis) - dump should be tagged with {family} and the executable with ripped:{family}
[x] A config without executed_sample and parent specified - repoter should still upload the config to mwdb and issue no dditional tags

Everything looks good.
Testing suite:
from mwdblib import MWDB
from typing import Any
from karton.core import Producer, Task, Resource
from tempfile import TemporaryDirectory
from pathlib import Path

mwdb = MWDB()

p = Producer(identity="test-producer")

def fetch_sample(sha256: str) -> Resource:
    s = mwdb.query_file(sha256, raise_not_found=True)
    return Resource(name=s.name, content=s.content)

def fetch_config(sha256: str) -> Any:
    return mwdb.query_config(sha256, raise_not_found=True)

def test_sample_executable() -> None:
    sample = fetch_sample("8a0d5d151ea835c7292ec6eb95d4aca6a6ea4148b6e7c3aef027941a4e019e37")
    t = Task({
        "kind": "raw",
        "type": "sample"
    }, payload={
        "sample": sample
    })
    p.send_task(t)

def test_sample_dump() -> None:
    executable = fetch_sample("6b1f2aa5d292b7820a7234b6bb89bb1553073bcf9b5b2537e080f1a7176369b6")
    dump = fetch_sample("971e7b94f81316322780dd09acf050f8d6c6b5c821c2e59dfa6ccd6af2cdbfdd")

    with TemporaryDirectory() as tmp_dir:
        tmp_dir_p = Path(tmp_dir)

        (tmp_dir_p / dump.name).write_bytes(dump.content)

        t = Task({
            "type": "analysis",
        }, payload={
            "sample": executable,
            "dumps.zip": Resource.from_directory(name="dumps.zip", directory_path=tmp_dir),
            "dumps_metadata": [{"filename": dump.name, "base_address": dump.name.split("_")[0]}]
        })
        p.send_task(t)

def test_alone_config() -> None:
    config = fetch_config("a6366bfc650b91243ee32ba19d4c06520fd5ecb5853dd5ef8450a44a9b62038a")

    t = Task({
        "type": "config",
        "family": config.family,
    }, payload={
        "config": config.config,
    })
    p.send_task(t)

def main() -> None:
    test_sample_executable()
    test_sample_dump()
    test_alone_config()

if __name__ == "__main__":
    main()

Results:

Log outputs:
karton-config-extractor.log
karton-mwdb-reporter.log

I know users should be informed of what code they are running, but is there a reason why this was not packaged in a new release with potential release notes indicating a breaking change? Just wondering, appreciate the plugin :).


Native Base Custom Tab Height is not increasing

Hi,
I am trying to create a custom tab using native base Tab . I am adding an image inside the custom tab but unfortunately tab height is not increasing according to the image . Please help me out
this is my code
    <Tabs >
          <Tab style={{height :250}}
                heading={ <TabHeading  ><Image source={require('../../res/images/tab_selected_1.png')}/></TabHeading>}>

          </Tab>

    </Tabs>

I've this exact problem too

Me too. How to adjust tab heading height?

Figured it out myself, share it to you guys:

Follow the Customize Guide to eject themes.
Edit height in native-base-theme/components/TabContainer.js.

closing this issue due to inactivity

tabContainerStyle={{ height: 60 }}
tabBarUnderlineStyle={{
backgroundColor: "black",
height: 5,
}}

this is work for me
tabContainerStyle={{ height: 60 }}
tabBarUnderlineStyle={{
backgroundColor: "black",
height: 5,
}}


Better error messages for DBC file decoding

At the moment it just says Could not parse dbc file and prints a lot of numbers in debug mode. No idea what is actually wrong with the file.
It should be better after can-dbc  migrated from nom 4 to nom 6: https://github.com/marcelbuesing/can-dbc/commits/nom6

Yeah it's kind of in can-dbc/#7. Thanks for the hint regarding nom_supreme I wasn't aware of that. I'm stuck in some nom "Incomplete" errors in the nom6 branch currently, maybe it also helps me with that.


spaces are causing problems

I could not get the xml2hash sample you gave to work with this code:
my $xml = <<'XML';
    <root>
       <item1>111</item1>
       <item2>222</item2>
       <item3>333</item3>
    </root>
XML

I had a peek at your unit tests and got one of them that was similar to work. Eventually I figured out that having a space for the first character was the culprit.
For example: say Dumper(xml2hash(' <root>boom</boom>'));
will result in a "Can't open file" error.
Likewise, the here doc above won't work unless the first line is justified against the left column.

Thanks for report.
I have improved XML detection in v0.39

Wow, that was fast (just like the module)... thanks!


[feature] Jadx script: jump to symbol and rename listener

Background:
I'm the author of  Graffiti, a tool for creating graphs interactively from the IDE.
In order to support JADX, those features in jadx-script are required:

[x] Keybinding/menu for code area - Using Gui.addPopupMenuAction.
[x] Get the method name and class from ICodeNodeRef
[x] Detect renames so I could update the graph
[ ] Given an "address" (full method signature), show it on JADX.

So those features are missing for support:
Rename listener
It would be nice to able to listen to the events of user renaming a symbol (class, method, field, package, package for class)
JEB for example, uses the following api:
dex = prj.findUnit(IDexUnit)
dex.addListener(RenameListener())

class DexListener(IEventListener):
    def onEvent(self, e):
        if isinstance(e, JebEvent) and e.type == J.UnitChange and e.data is not None:
            if e.data.type == UnitChangeEventData.NameUpdate:
                target = e.data.target

Jump to symbol
Given a symbol (For example ICodeNode), it would be really helpful if we could be able to open its code in the view programmatically. Same as The implementation in CommonSearchDialog probably.
JEB for example, uses:
dex = prj.findUnit(IDexUnit)
ctx.navigate(dex, FULL_DEX_ADDRESS)

on IDA, it is:
ida_kernwin.jumpto(addr)

@yoavst recently I make a jadx example plugin and also implement a way to install/manage plugins from cli and gui.
So, if you want, you can create a plugin instead script :slightly_smiling_face:

I'll try this, thanks!
Closing the issue as it is fixed.


Add cloudflare reciver to contrib

Jumped the gun a bit here, will add this once it's hit the next release :)


NLLB vocabulary missing common Chinese character/tokens

Hi, I use the released NLLB checkpoint to decode flroes Chinese testset, overall the results looks good. However, I found that a lot of very common Chinese characters/tokens are missing from the dictionary, leading to those words never generated from other languages to Chinese and OOV tokens when translating from Chinese to other languages.
For example, "The eagle catches the chickens" translates to "老鹰捉小鸡" in Chinese, but NLLB model generates "▁ <unk> 抓 住 了 <unk>" since tokens for the two species are absent from the dictionary. This can be a practical problem, because the missing tokens are absolutely common in real-world situation.
The following is part of the tokens highly-frequent but missing from the dictionary:
饱
畅
湍
滩
岭
舱
诩
阔
荫
鸽
勋
鸡
鹰
裙
艳
哦
毋庸
稻
蔗
熔
亥
裤
氢
《
》
...

I thought nllb uses a byte-level sentencepiece. Am I wrong?
Is the dict you talked about is this https://dl.fbaipublicfiles.com/large_objects/nllb/models/spm_200/dictionary.txt  ?
Since it is a byte-level dictionary, there is no actual word/character inside. They are meant to be decoded back to normal strings later.
So I think those  are generated just by chance.

I thought nllb uses a byte-level sentencepiece. Am I wrong? Is the dict you talked about is this https://dl.fbaipublicfiles.com/large_objects/nllb/models/spm_200/dictionary.txt ?
Since it is a byte-level dictionary, there is no actual word/character inside. They are meant to be decoded back to normal strings later. So I think those are generated just by chance.

Yes, I did use this dict. The sentencepiece model and translation model dictionary are downloaded from https://github.com/facebookresearch/fairseq/tree/nllb#preparing-datasets-for-training
Here is the translation output from Chinese to English on flroes devtest, there are a total of 447  in source language across 1012 sentences.
log.flores-test.checkpoint.NLLB-200-Distilled-600M.zh2en.txt

Confirmed.
The downloaded dictionary.txt does not have all byte chars.
So there are actually a lot of words/characters considered as .
the original one will produce unk like this:
original: 老鹰捉小鸡
after spm: ▁老 鹰 捉 小 鸡
2022-07-10 13:45:57 | INFO | fairseq_cli.preprocess | ▁老 鹰 捉 小
鸡
2022-07-10 13:45:57 | INFO | fairseq_cli.preprocess | 鹰 not found
in self.indices
# Since it is not found, I transfer 鹰 to its equivlent byte char string.
2022-07-10 13:45:57 | INFO | fairseq_cli.preprocess | 鹰 := é¹°
2022-07-10 13:45:57 | INFO | fairseq_cli.preprocess | 鸡 not found
in self.indices
2022-07-10 13:45:57 | INFO | fairseq_cli.preprocess | 鸡 := é¸¡
2022-07-10 13:45:57 | INFO | fairseq_cli.preprocess | tensor([230393, 248132,      3, 250174, 252996, 250014, 248132,      3, 249934,
             2], dtype=torch.int32)
decoded: ▁老 é <unk> ° 捉 小 é <unk> ¡
bchar converted: ▁老<unk>捉小<unk>

The result means byte char "¹" and "¸" does not exist inside the downloaded dictionary.txt as well.
In sum there are 36 byte chars missing in the dictionary.txt,

Thank you for your nice explanation!

In my case, I expanded fairseq/data/dictionary.py to overwrite least used tokens in the dict.txt with missing byte chars and put unknown words into byte char string.
With these 2 setup, there are no more <unk>. The vocab count is kept the same. Probably this is the minimum changes done to the pretrained model.
It would be a disaster if those least used tokens are what you want most. However this time it is 36 tokens, which may be ignored.
Then, yes you can finetune the model for your case without worrying any unknown words/symbols.

Other character base language, such as Japanese, may have similar problems too.

@huihuifan @edunov do you guys know if it could be possible to "update" the vocab for CJK and continue training so that those issues might be fixed ?
https://discuss.huggingface.co/t/nllb-3-3b-poor-translations-from-chinese-to-english/27695

I would wonder how the authors deal with those unknonw words. It feels like a huge hole and they would not have overlooked this.
In my case, I expanded fairseq/data/dictionary.py to overwrite least used tokens in the dict.txt with missing byte chars and put unknown words into byte char string. With these 2 setup, there are no more . The vocab count is kept the same. Probably this is the minimum changes done to the pretrained model. It would be a disaster if those least used tokens are what you want most. However this time it is 36 tokens, which may be ignored. Then, yes you can finetune the model for your case without worrying any unknown words/symbols.

how did you come up with 36 ?

@pluiez I am working on a curated version of NLLB-200 to include these 26 symbols. Are you sure there are no other missing symbols for Chinese/Japanese/Korean ?

I have a tutorial on expanding NLLB with a new language, which also has a section of adding new tokens to its vocabulary.
https://cointegrated.medium.com/a37fc706b865
However, I have to warn you that the model needs to be fine-tuned in order to use the new tokens adequately, and, unless you fine-tune it with enough data in all 202 languages, some forgetting may happen and the model would perform less well on some languages that didn't participate in the fine-tuning.


spring.application.name has no effect when set as an environment variable in 2022.0.3

We upgraded from 2022.0.2 to 2022.0.3 and our Spring Cloud Config clients no longer seem to work.
2022.0.2
2023-05-31T15:57:45.777-07:00  INFO 22800 --- [  restartedMain] org.example.ConfigTestApplication        : The following 1 profile is active: "dev"
2023-05-31T15:57:45.823-07:00  INFO 22800 --- [  restartedMain] o.s.c.c.c.ConfigServerConfigDataLoader   : Fetching config from server at : http://XXXX
2023-05-31T15:57:45.823-07:00  INFO 22800 --- [  restartedMain] o.s.c.c.c.ConfigServerConfigDataLoader   : Located environment: name=notification-service, profiles=[dev], label=null, version=baa976431f5c111b2bed9a8b7643212c5f302f9e, state=null
2023-05-31T15:57:45.824-07:00 DEBUG 22800 --- [  restartedMain] o.s.c.c.c.ConfigServerConfigDataLoader   : Environment notification-service has 1 property sources with 18 properties.

2022.0.3
2023-05-31T15:55:45.211-07:00  INFO 32892 --- [  restartedMain] org.example.ConfigTestApplication        : The following 1 profile is active: "dev"
2023-05-31T15:55:45.249-07:00  INFO 32892 --- [  restartedMain] o.s.c.c.c.ConfigServerConfigDataLoader   : Fetching config from server at : http://XXXX
2023-05-31T15:55:45.250-07:00  INFO 32892 --- [  restartedMain] o.s.c.c.c.ConfigServerConfigDataLoader   : Located environment: name=application, profiles=[default], label=null, version=baa976431f5c111b2bed9a8b7643212c5f302f9e, state=null
2023-05-31T15:55:45.250-07:00 DEBUG 32892 --- [  restartedMain] o.s.c.c.c.ConfigServerConfigDataLoader   : Environment application has 0 property sources with 0 properties.
2023-05-31T15:55:45.250-07:00  INFO 32892 --- [  restartedMain] o.s.c.c.c.ConfigServerConfigDataLoader   : Fetching config from server at : http://XXXX
2023-05-31T15:55:45.250-07:00  INFO 32892 --- [  restartedMain] o.s.c.c.c.ConfigServerConfigDataLoader   : Located environment: name=application, profiles=[dev], label=null, version=baa976431f5c111b2bed9a8b7643212c5f302f9e, state=null
2023-05-31T15:55:45.250-07:00 DEBUG 32892 --- [  restartedMain] o.s.c.c.c.ConfigServerConfigDataLoader   : Environment application has 0 property sources with 0 properties.
2023-05-31T15:55:45.250-07:00  INFO 32892 --- [  restartedMain] o.s.c.c.c.ConfigServerConfigDataLoader   : Fetching config from server at : http://XXXX
2023-05-31T15:55:45.250-07:00  INFO 32892 --- [  restartedMain] o.s.c.c.c.ConfigServerConfigDataLoader   : Located environment: name=application, profiles=[default], label=null, version=baa976431f5c111b2bed9a8b7643212c5f302f9e, state=null
2023-05-31T15:55:45.250-07:00 DEBUG 32892 --- [  restartedMain] o.s.c.c.c.ConfigServerConfigDataLoader   : Environment application has 0 property sources with 0 properties.

I didn't drill down too deeply in the code, but I do see that it is definitely passing in "application" to the restTemplate in ConfigServerConfigDataLoader instead of the value from spring.application.name ("notification-service").
This was tested against Spring Boot 3.0.7.
The profile and server uri are being passed through env variables
SPRING_PROFILES_ACTIVE=dev
spring.config.import=configserver:http://u:p@XXXX
We have no additional config. I created a dummy project to reproduce this, but unless I am missing something, this seems to be fundamentally broken. I didn't see any required config changes in the release notes.

Can you provide the sample that reproduces the problem?

How are you setting spring.application.name?  Can you try setting it via an environment variable?  Or placing spring.config.import in your application configuration file as opposed to an environment variable.

All of our code has spring.application.name in the application.yml with SPRING_PROFILES_ACTIVE and spring.config.import being passed into the docker container as env variables at runtime.
I did some more testing. If I put spring.application.name, spring.config.import, and spring.profiles.active in the application.yml then it works with the odd behavior that it makes 3 calls out to to the config server. Once without a profile, then with the profile, and then again without. In 2022.0.2 (and all previous versions that we have used), it correctly only made 1 call with the profile.
If I add spring.config.import and SPRING_PROFILES_ACTIVE as env variables (with all 3 values still in the application.yml) then it fails and also makes 6 calls. All with "application" instead of the application name and with the pattern of no profile/no profile/profile/no profile/profile/no profile. It works if I also add spring.application.name as an env variable, but again it makes 6 calls.
Doing just env variables works (as long as I have all 3) but again it is making 3 calls instead of 1.
I had assumed initially that this had something to do with the code that was added to ConfigServerConfigDataLoader to handle application names with a "-" so I tested using a name without dashes but that didn't fix it.
Here is a dummy client and server implementation to demonstrate this.
config-client-test.zip
config-server-test.zip
I didn't test to see what those extra calls are doing if I had a default config. Not sure if it would try merging it or what, which would definitely be different behavior than previous versions.

This sounds similar to what I posted in Gitter: https://matrix.to/#/!ZDFMLMDSttmhtcDdOk:gitter.im/$Pviy47HRPd1ltjPVna94VZ59v745LHvCL6O4bPgkNQA?via=gitter.im&via=matrix.org&via=matrix.freyachat.eu

Hi Everyone. I have also experienced  the same issue with the ConfigServerConfigDataLoader resolving to "application" instead of the spring.application.name value that I have provided :
o.s.c.c.c.ConfigServerConfigDataLoader   : Fetching config from server at : http://configserver:8080
o.s.c.c.c.ConfigServerConfigDataLoader   : Located environment: name=application, profiles=[default], label=null, version=null, state=null

Working with 2022.0.2 :
# No need to specify env variable for SPRING_APPLICATION_NAME as the one included in the application.properties is taken into account
SPRING_CONFIG_IMPORT: 'configserver:http://configserver:8080'
SPRING_PROFILES_ACTIVE: dev

Not working with 2022.0.3 :
SPRING_CONFIG_IMPORT: 'configserver:http://configserver:8080'
SPRING_PROFILES_ACTIVE: dev

But if we add the env variable SPRING_APPLICATION_NAME with the valye, it works again.
Another test I did also is to remove the SPRING_CONFIG_IMPORT env variable (and set it directly in the application.properties) makes it work again.
After digging in the code, my gut feeling goes towards this commit : https://github.com/spring-cloud/spring-cloud-config/commit/6ec9c432cb2c02840b85fdd8bcd0859eabc0ba46
Here is a sample project with a config server and a client ( spring cloud 2022.0.3, java 17). You can run it with docker compose. There is an additionnal step to build the docker image of the client with jib plugin, it's explain in the readme.md
demo-issue-cloud-config.tar.gz

Yes this change introduced the problem https://github.com/spring-cloud/spring-cloud-config/commit/6ec9c432cb2c02840b85fdd8bcd0859eabc0ba46
The workaround for now is to

Set spring.application.name and spring.config.import as environment variables (or as command line arguments)
Set spring.application.name and spring.config.import in the same configuration properties file

There is always going to be a slight caveat here with how we load configuration data from the config server and how spring.config.import is set.  When spring.config.import is set via an environment variable or as a command line argument configuration will be fetched BEFORE any application configuration properties files have been loaded.
The reason why there is a different behavior as of 2022.0.3 is because with https://github.com/spring-cloud/spring-cloud-config/commit/6ec9c432cb2c02840b85fdd8bcd0859eabc0ba46 now the config client fetches config data in two separate phases.  The first phase is gathers any profiles that may be activated in configuration from the config server.  The second phase will fetch the configuration again with all the activated profiles.  This is why you see multiple requests now.
However as a side effect of this if you set spring.config.import=optional:configsever in an environment variable the config client will try to fetch configuration for activated profiles before any other configuration properties are loaded and will not have a value for spring.application.name.  This behavior is dictate by Spring Boot and is out of our control.  However the subsequent request with all the activate profiles will have a value for spring.application.name.  The problem is that we created ConfigClientProperties in the first request and never update it.
PR #2275 addresses this and will now update ConfigClientProperties with the value of spring.application.name allowing the second request to succeed.
I know that might be hard to follow, but essentially after PR #2275 is merged it will still be possible that the first request fails, however the second request will succeed and that should yield the same behavior as 2022.02.

Hi! Thank you for your reactivity and explanations on this one. I'm going to apply the first workaround with the two env variables until 2022.0.4 is released.

If anyone wants to try the fix it should now be available in 2022.0.4-SNAPSHOT

If anyone wants to try the fix it should now be available in 2022.0.4-SNAPSHOT

I see it following the behavior you mentioned where it makes the incorrect call followed by a correct one, but I see it still making that extra call without a profile. Is that going to cause issues? 2022.0.2 does not make any calls without the profile. If I put spring.config.import in both application.yml and env, then it goes up to 6 calls.
2023-06-02T10:27:00.843-07:00  INFO 24236 --- [           main] org.example.ConfigTestApplication        : The following 1 profile is active: "dev"
2023-06-02T10:27:00.888-07:00  INFO 24236 --- [           main] o.s.c.c.c.ConfigServerConfigDataLoader   : Fetching config from server at : http://localhost:8888
2023-06-02T10:27:00.888-07:00  INFO 24236 --- [           main] o.s.c.c.c.ConfigServerConfigDataLoader   : Located environment: name=application, profiles=[default], label=null, version=null, state=null
2023-06-02T10:27:00.889-07:00 DEBUG 24236 --- [           main] o.s.c.c.c.ConfigServerConfigDataLoader   : Environment application has 0 property sources with 0 properties.
2023-06-02T10:27:00.889-07:00  INFO 24236 --- [           main] o.s.c.c.c.ConfigServerConfigDataLoader   : Fetching config from server at : http://localhost:8888
2023-06-02T10:27:00.889-07:00  INFO 24236 --- [           main] o.s.c.c.c.ConfigServerConfigDataLoader   : Located environment: name=config-client-test, profiles=[dev], label=null, version=null, state=null
2023-06-02T10:27:00.889-07:00 DEBUG 24236 --- [           main] o.s.c.c.c.ConfigServerConfigDataLoader   : Environment config-client-test has 2 property sources with 2 properties.
2023-06-02T10:27:00.889-07:00  INFO 24236 --- [           main] o.s.c.c.c.ConfigServerConfigDataLoader   : Fetching config from server at : http://localhost:8888
2023-06-02T10:27:00.889-07:00  INFO 24236 --- [           main] o.s.c.c.c.ConfigServerConfigDataLoader   : Located environment: name=config-client-test, profiles=[default], label=null, version=null, state=null
2023-06-02T10:27:00.889-07:00 DEBUG 24236 --- [           main] o.s.c.c.c.ConfigServerConfigDataLoader   : Environment config-client-test has 1 property sources with 1 properties.

Are you expecting the spring.config.import environment variable to override the one in application.yml?

No. Just an observation that it suddenly starts making another 3 calls. With both places set, 2022.0.2 still only made 1 call.

With both places set, 2022.0.2 still only made 1 call.

That is not what I see, I see 2 calls being made when set in both places
2023-06-02T14:27:55.851-04:00  INFO 53821 --- [           main] com.example.demo.Demo1Application        : Starting Demo1Application using Java <IP_ADDRESS> with PID 53821 (/Users/ryanjbaxter/temp/issues/2274/demo1/target/classes started by ryanjbaxter in /Users/ryanjbaxter/temp/issues/2274/demo1)
2023-06-02T14:27:55.852-04:00  INFO 53821 --- [           main] com.example.demo.Demo1Application        : No active profile set, falling back to 1 default profile: "default"
2023-06-02T14:27:55.864-04:00  INFO 53821 --- [           main] o.s.c.c.c.ConfigServerConfigDataLoader   : Fetching config from server at : http://localhost:8888
2023-06-02T14:27:55.864-04:00  INFO 53821 --- [           main] o.s.c.c.c.ConfigServerConfigDataLoader   : Exception on Url - http://localhost:8888:org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://localhost:8888/testapp/default": Connection refused. Will be trying the next url if available
2023-06-02T14:27:55.864-04:00  WARN 53821 --- [           main] o.s.c.c.c.ConfigServerConfigDataLoader   : Could not locate PropertySource ([ConfigServerConfigDataResource@419a20a6 uris = array<String>['http://localhost:8888'], optional = true, profiles = list['default']]): I/O error on GET request for "http://localhost:8888/testapp/default": Connection refused
2023-06-02T14:27:55.864-04:00  INFO 53821 --- [           main] o.s.c.c.c.ConfigServerConfigDataLoader   : Fetching config from server at : http://localhost:8888
2023-06-02T14:27:55.865-04:00  INFO 53821 --- [           main] o.s.c.c.c.ConfigServerConfigDataLoader   : Exception on Url - http://localhost:8888:org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://localhost:8888/testapp/default": Connection refused. Will be trying the next url if available
2023-06-02T14:27:55.865-04:00  WARN 53821 --- [           main] o.s.c.c.c.ConfigServerConfigDataLoader   : Could not locate PropertySource ([ConfigServerConfigDataResource@62e70ea3 uris = array<String>['http://localhost:8888'], optional = true, profiles = list['default']]): I/O error on GET request for "http://localhost:8888/testapp/default": Connection refused
2023-06-02T14:27:56.025-04:00  INFO 53821 --- [           main] o.s.cloud.context.scope.GenericScope     : BeanFactory id=c1d60177-05a1-370d-9f44-ce3e49de8cf2
2023-06-02T14:27:56.134-04:00  INFO 53821 --- [           main] com.example.demo.Demo1Application        : Started Demo1Application in 0.592 seconds (process running for 0.901)

That is not what I see, I see 2 calls being made when set in both places

Sorry, looks like you are right. I didn't have the env variable set correctly for the 2022.0.2 test.

If you have any issues when the client doesn't see server configuration just check "spring.application.name" in client service. I renamed the service name from my-service to my_service and forgot to rename "spring.application.name" param.


How to use MQ-2 Sensor?

Please tell me how to use the sensor MQ-2  + arduino + RPI3? Its Real?

the first question is simple, but has a very complex answer.
if you had an MQ-2 sensor and wanted to export it via homebridge, you would need to do two things:
first, you need to find the appropriate homekit characteristics that allow you to model the sensor. here is the "standard' list ... if you can't find what you need, feel free to submit an issue to the "non-standard list of extensions" repository.
in the case of the MQ-2 sensor, your best bet is to call it an air quality sensor because as the "Note" says on the page:
     The sensor value only reflects the approximated trend of gas concentration in a
     permissible error range, it DOES NOT represent the exact gas concentration.
     The detection of certain components in the air usually requires a more precise and
     costly instrument, which cannot be done with a single gas sensor. If your project is
     aimed at obtaining the gas concentration at a very precise level, then we don’t
     recommend this gas sensor.

your job would be to decide how to map the voltage reported by the sensor to an air quality value. or perhaps you can use the Rs/R0 ratio as the basis for the calculation. the key thing is to understand how to model the sensor using the characteristics and services available via homekit, preferably using the standard objects rather than the non-standard ones.
second,  you just need to have something that periodically reads the sensor and then updates a plugin. depending on your hardware, you may have to do this in two steps
find a way of getting periodic readings from the sensor. the grove seed sites suggests using an arduino and that's a good way of doing it. the only problem is that you're not going to be running homebridge on an arduino, so you're going to need a way of getting it to a system capable of running homebridge. you can look at homespun-arduino to see one way of doing that.
two years ago i started, but never finished, work on homespun-grovepi which talks to a grove shield for the RPi. the basic idea again is to talk to the sensor. both of these repos read from the sensor and then send out a multicast UDP packet using a format called TSRP.
finally, you need to have a way of receiving the sensor reading over the network. you don't have to use TSRP, knowing that it came from an MQ-2 and then implementing a plugin that corresponds to the sensor.
doable, but not simple. good luck!


Configure ide

Reference issue
Fixes https://github.com/mne-tools/mne-python/issues/9112
What does this implement/fix?
Creating a page that steps through the various IDE settings that are helpful when contributing to MNE-Python.
Additional information
Picking up and resolving an idle ticket. Tried implementing some of the changes as suggested. Not sure if :menuselection: by Sphinx is working as rst is not displaying correctly for me when installing MNE right out of the box. May require additional setup to view it correctly.
**Fixed duplicate label

@th3linja you don't need to open a new PR for each new change, just push commits to your branch and they should show up on GitHub. cc @britta-wstnr this is why #11613 was closed I think

I've made some changes to reflect the suggestions made. Let me know what you think. Also if someone using VSCode could double down on the steps with their IDE version, that would be great.

Given @hoechenberger's latest comment and before we venture further, can I raise the point again that I raised above and that @mscheltienne seemed to agree with?
I think giving such specific settings for IDEs will be very hard to keep up-to-date - so, do we want to do that? Or do we want to keep this more general, i.e. mention what settings are useful but not give directions on how to set them in different IDE's.
Pinging @larsoner @hoechenberger and @drammock who opened the original issue.

mention what settings are useful but not give directions on how to set them in different IDE's

+1.  I imagine something like this would be more future-proof:
With the right configuration, using an Integrated Development Environment (IDE) can make contributing to MNE-Python much easier.
This page describes the most common IDE settings that are recommended for MNE-Python contributors.

## syntax highlighting
<most users find it helpful, usually enabled by default, if not search "python syntax highlight [name of IDE]" to learn how to do it.>

## Linting
<describe what linting is, mention what linter we use.>

### Rulers
<describe what & why of rulers>

### Line endings and whitespace
<4 spaces (not tabs), LF not CRLF (also mention relevant git setting), trailing whitespace,
 trailing newlines at EOF, whitespace around classes/functions/methods, whitespace in docstrings.>

### code completion / introspection
<what & how; should probably mention at least jedi and pylance>

## Configuring Specific IDEs

### Spyder
<general comments about which config sections to look in for things (tools -> preferences -> { completion and linting | source code | advanced settings } ) but don't try to say exactly what is found where.>

### VSCode
<which extensions should be installed, general comments about how to configure this IDE (GUI dialog vs config JSON file)>

### Pycharm
<similar advice to Spyder / VSCode>

Hi @th3linja, I agree with @drammock's suggestion to update this - are you able to work on this? 🙂

For sure! I'll send an update as soon as I get the chance.

Hi all, I made some revisions. Let me know how it looks!

Should I add Vim to this list or would that be overly archaic?


Implement about() for subreddits.

Hi, I needed a way to obtain a subreddit's subscriber count so I implemented the about API endpoint. Unfortunately this is not documented besides a very short description:

Return information about the subreddit.
Data includes the subscriber count, description, and header image.

So I had to guess the type for some of the response fields. I used Option<T> everywhere since the API might change and there's no guarantee about which fields are present.
Thanks!

LGTM, thanks!


Why do you need this feature?
There is no command like envd shutdown to deinitialize envd, which allows users closing envd as they want, keeping the development environment tidy and neat.
Additional context
No response

We have envd v1 which can use moby builder directly (doesn't need to run a buildkitd container). I think that's the trend. However, we don't have a plan to release v1 recently.

Okay, that's fine.


AccessLogging: fix the issue where disable accesslogging does not take effect

Please provide a description of this PR:
Fix a bug introduced in #43372
This configuration did not take effect.
accessLogging:
- disabled: true
  providers:
  - name: envoy

need backport to v1.17

@zirain This bug is introduced after v1.17.1. Do we not need to prepare a release note for this?

@zirain This bug is introduced after v1.17.1. Do we not need to prepare a release note for this?

yes

In response to a cherrypick label: #43770 failed to apply on top of branch "release-1.16":
Applying: AccessLogging: fix the issue where disable accesslogging does not take effect.
Using index info to reconstruct a base tree...
M	pilot/pkg/model/telemetry.go
M	pilot/pkg/model/telemetry_logging_test.go
M	pilot/pkg/networking/core/v1alpha3/accesslog.go
M	pilot/pkg/networking/core/v1alpha3/accesslog_test.go
Falling back to patching base and 3-way merge...
Auto-merging pilot/pkg/networking/core/v1alpha3/accesslog_test.go
CONFLICT (content): Merge conflict in pilot/pkg/networking/core/v1alpha3/accesslog_test.go
Auto-merging pilot/pkg/networking/core/v1alpha3/accesslog.go
Auto-merging pilot/pkg/model/telemetry_logging_test.go
Auto-merging pilot/pkg/model/telemetry.go
CONFLICT (content): Merge conflict in pilot/pkg/model/telemetry.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 AccessLogging: fix the issue where disable accesslogging does not take effect.
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to a cherrypick label: new issue created for failed cherrypick: #43783

In response to a cherrypick label: #43770 failed to apply on top of branch "release-1.17":
Applying: AccessLogging: fix the issue where disable accesslogging does not take effect.
Using index info to reconstruct a base tree...
M	pilot/pkg/model/telemetry.go
M	pilot/pkg/networking/core/v1alpha3/accesslog.go
Falling back to patching base and 3-way merge...
Auto-merging pilot/pkg/networking/core/v1alpha3/accesslog.go
Auto-merging pilot/pkg/model/telemetry.go
CONFLICT (content): Merge conflict in pilot/pkg/model/telemetry.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 AccessLogging: fix the issue where disable accesslogging does not take effect.
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to a cherrypick label: new issue created for failed cherrypick: #43784


Integrating Migration endpoints into virt-api

retest this please

retest this please

retest this please

retest this please


[filter] Extract inference wrapper header

This patch extracts inference wrapper header for testability and better
maintainability.
Self evaluation:

Build test: [X]Passed [ ]Failed [ ]Skipped
Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee<EMAIL_ADDRESS>

:memo: TAOS-CI Version: 1.4.20191203. Thank you for submitting PR #957. Please a submit 1commit/1PR (one commit per one PR) policy to get comments quickly from reviewers. Your PR must pass all verificiation processes of cibot before starting a review process from reviewers. If you are new member to join this project, please read manuals in documentation folder and wiki page. In order to monitor a progress status of your PR in more detail, visit http://nnsuite.mooo.com/.

:octocat: cibot: @zhoonit, A builder checker could not be completed because one of the checkers is not completed. In order to find out a reason, please go to http://nnsuite.mooo.com/nntrainer/ci/repo-workers/pr-checker/957-202102181411570.81881999969482-b6ec7170213334a62a77f65857c6fd08bdd8ca27/.


Docs: reintroduce aria-describedby advice for help text

Based on v3 http://getbootstrap.com/css/#forms-help-text, adapted to make examples more "real life"

Seems like these should use .form-group for consistency with the other form examples.

Added a <div class="form-group"> to the first example, as that also had the full <form> around it. For the second one, I'd say it can remain without the <div>, as it's only a fragment and not a complete form (similar to some of the more focused previous examples like radio buttons etc)
Also spotted two instances of help text in the main form controls example which lacked aria-describedby...rectified now


split addindex real-tikv test into multiple directory

currently, test in real-tikv-test is run in serial, if there're too many test in one directory, test will run slow and delay pr merge, or even timeout
so we add 4 addindextestX into real tikv test, we will split the test later, to avoid test timeout as we will add more test later.

/hold

/unhold


Update projection.md

Added missing name column in projection example

@robinlehrmann thanks :)

@robinlehrmann thanks :)


rclone mount leads to DELETE when appending to a remotely locked file

What is the problem you are having with rclone?
I've noticed that when I write (append) to a file over rclone mount in quick succession, sometimes ownCloud returns a 423 (file locked). This results in rclone giving a DELETE on the file.
What is your rclone version (output from rclone version)
rclone --version
rclone v1.47.0

os/arch: linux/amd64
go version: go1.11.5

Which OS you are using and how many bits (eg Windows 7, 64 bit)
CentOS Linux release 7.6 (64 bit)
Which cloud storage system are you using? (eg Google Drive)
ownCloud (10.0.10 and 10.1)
The command you were trying to run (eg rclone copy /tmp remote:tmp)
Mounting:
rclone mount -vvv --debug-fuse --vfs-cache-mode=full owncloud:/ /mnt/rclone/ 2>&1|tee /tmrclone_append-full-surfdrive.log

Then I run:
time for i in {1..100}
do
    echo run $i >> /mnt/rclone/rclone_appending.txt
done

This works for a time, but after a while it will start erroring:
-bash: echo: write error: Input/output error

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)
Apache logs:
<IP_ADDRESS> -  [14/Jun/2019:12:06:45 +0200] "PUT /remote.php/nonshib-webdav/rclone_appending.txt HTTP/1.1" 204 - "-" "rclone/v1.47.0" microsecs:2407940 response_size:0 bytes_received:660 bytes_sent:947 XQNxtXtu9FMVBQEOlV-QkQAAAMY - 17497
<IP_ADDRESS> -  [14/Jun/2019:12:06:47 +0200] "PROPFIND /remote.php/nonshib-webdav/rclone_appending.txt HTTP/1.1" 207 733 "-" "rclone/v1.47.0" microsecs:333823 response_size:733 bytes_received:735 bytes_sent:1647 XQNxt3tu9FMVBQEOlV-QkgAAAMY - 17497
<IP_ADDRESS> -  [14/Jun/2019:12:06:47 +0200] "PUT /remote.php/nonshib-webdav/rclone_appending.txt HTTP/1.1" 204 - "-" "rclone/v1.47.0" microsecs:2254121 response_size:0 bytes_received:668 bytes_sent:939 XQNxt3tu9FMVBQEOlV-QkwAAAMY - 17497
<IP_ADDRESS> -  [14/Jun/2019:12:06:50 +0200] "PROPFIND /remote.php/nonshib-webdav/rclone_appending.txt HTTP/1.1" 207 734 "-" "rclone/v1.47.0" microsecs:299133 response_size:734 bytes_received:735 bytes_sent:1656 XQNxuntu9FMVBQEOlV-QlQAAAMY - 17497
<IP_ADDRESS> -  [14/Jun/2019:12:06:50 +0200] "PUT /remote.php/nonshib-webdav/rclone_appending.txt HTTP/1.1" 423 240 "-" "rclone/v1.47.0" microsecs:281125 response_size:240 bytes_received:675 bytes_sent:1098 XQNxuntu9FMVBQEOlV-QlgAAAMY - 17497
<IP_ADDRESS> -  [14/Jun/2019:12:06:51 +0200] "DELETE /remote.php/nonshib-webdav/rclone_appending.txt HTTP/1.1" 204 - "-" "rclone/v1.47.0" microsecs:1952813 response_size:0 bytes_received:415 bytes_sent:843 XQNxu3tu9FMVBQEOlV-QmAAAAME - 17497
<IP_ADDRESS> -  [14/Jun/2019:12:06:53 +0200] "GET /remote.php/nonshib-webdav/rclone_appending.txt HTTP/1.1" 404 248 "-" "rclone/v1.47.0" microsecs:265505 response_size:248 bytes_received:412 bytes_sent:1109 XQNxvXtu9FMVBQEOlV-QmQAAAME - 17497

The owncloud log itself gives some more detail (though not more information):
{
  "reqId": "XQNxuntu9FMVBQEOlV-QlgAAAMY",
  "level": 1,
  "time": "14/Jun/2019:12:06:50",
  "remoteAddr": "<IP_ADDRESS>",
  "user": "wybrandl",
  "app": "admin_audit",
  "method": "PUT",
  "url": "/remote.php/nonshib-webdav/rclone_appending.txt",
  "message": "User wybrandl logged into Miskatonic University from IP address: <IP_ADDRESS> [CLIENT_IP: <IP_ADDRESS>] [USER_AGENT: rclone/v1.47.0]"
}
{
  "reqId": "XQNxuntu9FMVBQEOlV-QlgAAAMY",
  "level": 4,
  "time": "14/Jun/2019:12:06:50",
  "remoteAddr": "<IP_ADDRESS>",
  "user": "wybrandl",
  "app": "webdav",
  "method": "PUT",
  "url": "/remote.php/nonshib-webdav/rclone_appending.txt",
  "message": "Exception: HTTP/1.1 423 \"rclone_appending.txt\" is locked: {\"Exception\":\"OCA\\\\DAV\\\\Connector\\\\Sabre\\\\Exception\\\\FileLocked\",\"Message\":\"\\\"rclone_appending.txt\\\" is locked\",\"Code\":0,\"Trace\":\"#0 \\/var\\/www\\/owncloud\\/lib\\/composer\\/sabre\\/dav\\/lib\\/DAV\\/Server.php(1130): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\File->put(Resource id #143)\\n#1 \\/var\\/www\\/owncloud\\/lib\\/composer\\/sabre\\/dav\\/lib\\/DAV\\/CorePlugin.php(513): Sabre\\\\DAV\\\\Server->updateFile('rclone_appendin...', Resource id #143, NULL)\\n#2 [internal function]: Sabre\\\\DAV\\\\CorePlugin->httpPut(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#3 \\/var\\/www\\/owncloud\\/lib\\/composer\\/sabre\\/event\\/lib\\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\\n#4 \\/var\\/www\\/owncloud\\/lib\\/composer\\/sabre\\/dav\\/lib\\/DAV\\/Server.php(479): Sabre\\\\Event\\\\EventEmitter->emit('method:PUT', Array)\\n#5 \\/var\\/www\\/owncloud\\/lib\\/composer\\/sabre\\/dav\\/lib\\/DAV\\/Server.php(254): Sabre\\\\DAV\\\\Server->invokeMethod(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#6 \\/var\\/www\\/owncloud\\/apps\\/dav\\/appinfo\\/v1\\/webdav.php(65): Sabre\\\\DAV\\\\Server->exec()\\n#7 \\/var\\/www\\/owncloud\\/apps\\/user_shibboleth\\/appinfo\\/remote.php(15): require('\\/var\\/www\\/ownclo...')\\n#8 \\/var\\/www\\/owncloud\\/remote.php(165): require_once('\\/var\\/www\\/ownclo...')\\n#9 {main}\",\"File\":\"\\/var\\/www\\/owncloud\\/apps\\/dav\\/lib\\/Connector\\/Sabre\\/File.php\",\"Line\":191}"
}

So what seems to be happening, is that every append leads to a PUT and a PROPFIND. Then, apparently, once every so often ownCloud is not done yet with releasing the file lock, and returns a 423. And what happens next is that rclone gives a DELETE. Now the file is gone, locally and remotely. In ownCloud, you can see the file in the trashbin, and restore it from there. But locally the file stays gone, and both rclone and my loop are unable to write to it.
This is the rclone output:
2019/06/14 12:06:50 INFO  : rclone_appending.txt: Copied (replaced existing)
2019/06/14 12:06:50 DEBUG : rclone_appending.txt: transferred to remote
2019/06/14 12:06:50 DEBUG : rclone_appending.txt(0xc00056dda0): >close: err=<nil>
2019/06/14 12:06:50 DEBUG : &{rclone_appending.txt (rw)}: >Flush: err=<nil>
2019/06/14 12:06:50 DEBUG : fuse: -> [ID=0x52] Flush
2019/06/14 12:06:50 DEBUG : fuse: <- Open [ID=0x54 Node=0x2 Uid=0 Gid=0 Pid=5753] dir=false fl=OpenWriteOnly+OpenAppend
2019/06/14 12:06:50 DEBUG : rclone_appending.txt: Open: flags=OpenWriteOnly+OpenAppend
2019/06/14 12:06:50 DEBUG : rclone_appending.txt: Open: flags=O_WRONLY|O_APPEND
2019/06/14 12:06:50 DEBUG : rclone_appending.txt: >Open: fd=rclone_appending.txt (rw), err=<nil>
2019/06/14 12:06:50 DEBUG : rclone_appending.txt: >Open: fh=&{rclone_appending.txt (rw)}, err=<nil>
2019/06/14 12:06:50 DEBUG : fuse: -> [ID=0x54] Open 0x1 fl=0
2019/06/14 12:06:50 DEBUG : fuse: <- Release [ID=0x53 Node=0x2 Uid=0 Gid=0 Pid=0] 0x2 fl=OpenWriteOnly+OpenAppend rfl=0 owner=0x0
2019/06/14 12:06:50 DEBUG : &{rclone_appending.txt (rw)}: Release:
2019/06/14 12:06:50 DEBUG : rclone_appending.txt(0xc00056dda0): RWFileHandle.Release nothing to do
2019/06/14 12:06:50 DEBUG : &{rclone_appending.txt (rw)}: >Release: err=<nil>
2019/06/14 12:06:50 DEBUG : fuse: -> [ID=0x53] Release
2019/06/14 12:06:50 DEBUG : fuse: <- Flush [ID=0x55 Node=0x2 Uid=0 Gid=0 Pid=5753] 0x1 fl=0x0 lk=0xd1b4980456f80000
2019/06/14 12:06:50 DEBUG : &{rclone_appending.txt (rw)}: Flush:
2019/06/14 12:06:50 DEBUG : &{rclone_appending.txt (rw)}: >Flush: err=<nil>
2019/06/14 12:06:50 DEBUG : fuse: -> [ID=0x55] Flush
2019/06/14 12:06:50 DEBUG : fuse: <- Write [ID=0x56 Node=0x2 Uid=0 Gid=0 Pid=5753] 0x1 7 @103 fl=0 lock=0 ffl=OpenWriteOnly+OpenAppend
2019/06/14 12:06:50 DEBUG : &{rclone_appending.txt (rw)}: Write: len=7, offset=103
2019/06/14 12:06:50 DEBUG : rclone_appending.txt: Size and modification time the same (differ by 846.265491ms, within tolerance 1s)
2019/06/14 12:06:50 DEBUG : rclone_appending.txt: Unchanged skipping
2019/06/14 12:06:50 DEBUG : rclone_appending.txt(0xc00050e3c0): Opened existing cached copy with flags=O_WRONLY|O_APPEND
2019/06/14 12:06:50 DEBUG : &{rclone_appending.txt (rw)}: >Write: written=7, err=<nil>
2019/06/14 12:06:50 DEBUG : fuse: -> [ID=0x56] Write 7
2019/06/14 12:06:50 DEBUG : fuse: <- Flush [ID=0x57 Node=0x2 Uid=0 Gid=0 Pid=5753] 0x1 fl=0x0 lk=0xd1b4980456f80000
2019/06/14 12:06:50 DEBUG : &{rclone_appending.txt (rw)}: Flush:
2019/06/14 12:06:50 DEBUG : rclone_appending.txt(0xc00050e3c0): close:
2019/06/14 12:06:50 DEBUG : rclone_appending.txt: Sizes differ (src 110 vs dst 103)
2019/06/14 12:06:53 ERROR : rclone_appending.txt: Failed to copy: "rclone_appending.txt" is locked: OCA\DAV\Connector\Sabre\Exception\FileLocked: 423 Locked
2019/06/14 12:06:53 ERROR : rclone_appending.txt(0xc00050e3c0): failed to transfer file from cache to remote: "rclone_appending.txt" is locked: OCA\DAV\Connector\Sabre\Exception\FileLocked: 423 Locked
2019/06/14 12:06:53 DEBUG : rclone_appending.txt(0xc00050e3c0): >close: err=failed to transfer file from cache to remote: "rclone_appending.txt" is locked: OCA\DAV\Connector\Sabre\Exception\FileLocked: 423 Locked
2019/06/14 12:06:53 ERROR : rclone_appending.txt(0xc00050e3c0): RWFileHandle.Flush error: failed to transfer file from cache to remote: "rclone_appending.txt" is locked: OCA\DAV\Connector\Sabre\Exception\FileLocked: 423 Locked
2019/06/14 12:06:53 DEBUG : &{rclone_appending.txt (rw)}: >Flush: err=failed to transfer file from cache to remote: "rclone_appending.txt" is locked: OCA\DAV\Connector\Sabre\Exception\FileLocked: 423 Locked
2019/06/14 12:06:53 DEBUG : fuse: -> [ID=0x57] Flush error=EIO: failed to transfer file from cache to remote: "rclone_appending.txt" is locked: OCA\DAV\Connector\Sabre\Exception\FileLocked: 423 Locked
2019/06/14 12:06:53 DEBUG : fuse: <- Open [ID=0x59 Node=0x2 Uid=0 Gid=0 Pid=5753] dir=false fl=OpenWriteOnly+OpenAppend
2019/06/14 12:06:53 DEBUG : rclone_appending.txt: Open: flags=OpenWriteOnly+OpenAppend
2019/06/14 12:06:53 DEBUG : rclone_appending.txt: Open: flags=O_WRONLY|O_APPEND
2019/06/14 12:06:53 DEBUG : rclone_appending.txt: >Open: fd=rclone_appending.txt (rw), err=<nil>
2019/06/14 12:06:53 DEBUG : rclone_appending.txt: >Open: fh=&{rclone_appending.txt (rw)}, err=<nil>
2019/06/14 12:06:53 DEBUG : fuse: -> [ID=0x59] Open 0x2 fl=0
2019/06/14 12:06:53 DEBUG : fuse: <- Release [ID=0x58 Node=0x2 Uid=0 Gid=0 Pid=0] 0x1 fl=OpenWriteOnly+OpenAppend rfl=0 owner=0x0
2019/06/14 12:06:53 DEBUG : &{rclone_appending.txt (rw)}: Release:
2019/06/14 12:06:53 DEBUG : rclone_appending.txt(0xc00050e3c0): RWFileHandle.Release nothing to do
2019/06/14 12:06:53 DEBUG : &{rclone_appending.txt (rw)}: >Release: err=<nil>
2019/06/14 12:06:53 DEBUG : fuse: -> [ID=0x58] Release
2019/06/14 12:06:53 DEBUG : fuse: <- Flush [ID=0x5a Node=0x2 Uid=0 Gid=0 Pid=5753] 0x2 fl=0x0 lk=0xd1b4980456f80000
2019/06/14 12:06:53 DEBUG : &{rclone_appending.txt (rw)}: Flush:
2019/06/14 12:06:53 DEBUG : &{rclone_appending.txt (rw)}: >Flush: err=<nil>
2019/06/14 12:06:53 DEBUG : fuse: -> [ID=0x5a] Flush
2019/06/14 12:06:53 DEBUG : fuse: <- Write [ID=0x5b Node=0x2 Uid=0 Gid=0 Pid=5753] 0x2 7 @110 fl=0 lock=0 ffl=OpenWriteOnly+OpenAppend
2019/06/14 12:06:53 DEBUG : &{rclone_appending.txt (rw)}: Write: len=7, offset=110
2019/06/14 12:06:53 DEBUG : rclone_appending.txt: Sizes differ (src 103 vs dst 110)
2019/06/14 12:06:53 ERROR : rclone_appending.txt: Failed to copy: failed to open source object: File with name rclone_appending.txt could not be located: Sabre\DAV\Exception\NotFound: 404 Not Found
2019/06/14 12:06:53 DEBUG : &{rclone_appending.txt (rw)}: >Write: written=0, err=open RW handle failed to update cached file: failed to open source object: File with name rclone_appending.txt could not be located: Sabre\DAV\Exception\NotFound: 404 Not Found
2019/06/14 12:06:53 DEBUG : fuse: -> [ID=0x5b] Write error=EIO: open RW handle failed to update cached file: failed to open source object: File with name rclone_appending.txt could not be located: Sabre\DAV\Exception\NotFound: 404 Not Found
2019/06/14 12:06:53 DEBUG : fuse: <- Flush [ID=0x5c Node=0x2 Uid=0 Gid=0 Pid=5753] 0x2 fl=0x0 lk=0xd1b4980456f80000
2019/06/14 12:06:53 DEBUG : &{rclone_appending.txt (rw)}: Flush:
2019/06/14 12:06:53 DEBUG : &{rclone_appending.txt (rw)}: >Flush: err=<nil>
2019/06/14 12:06:53 DEBUG : fuse: -> [ID=0x5c] Flush
2019/06/14 12:06:53 DEBUG : fuse: <- Open [ID=0x5e Node=0x2 Uid=0 Gid=0 Pid=5753] dir=false fl=OpenWriteOnly+OpenAppend
2019/06/14 12:06:53 DEBUG : rclone_appending.txt: Open: flags=OpenWriteOnly+OpenAppend
2019/06/14 12:06:53 DEBUG : rclone_appending.txt: Open: flags=O_WRONLY|O_APPEND
2019/06/14 12:06:53 DEBUG : rclone_appending.txt: >Open: fd=rclone_appending.txt (rw), err=<nil>
2019/06/14 12:06:53 DEBUG : rclone_appending.txt: >Open: fh=&{rclone_appending.txt (rw)}, err=<nil>
2019/06/14 12:06:53 DEBUG : fuse: -> [ID=0x5e] Open 0x1 fl=0
2019/06/14 12:06:53 DEBUG : fuse: <- Release [ID=0x5d Node=0x2 Uid=0 Gid=0 Pid=0] 0x2 fl=OpenWriteOnly+OpenAppend rfl=0 owner=0x0
2019/06/14 12:06:53 DEBUG : &{rclone_appending.txt (rw)}: Release:
2019/06/14 12:06:53 DEBUG : rclone_appending.txt(0xc00050ea80): RWFileHandle.Release closing
2019/06/14 12:06:53 DEBUG : rclone_appending.txt(0xc00050ea80): close:
2019/06/14 12:06:53 DEBUG : rclone_appending.txt(0xc00050ea80): not modified so not transferring
2019/06/14 12:06:53 DEBUG : rclone_appending.txt(0xc00050ea80): >close: err=<nil>

What is happening here I think is

the upload fails with a 423 error - this is when the webdav backend deletes the file

2019/06/14 12:06:50 DEBUG : rclone_appending.txt: Sizes differ (src 110 vs dst 103)
2019/06/14 12:06:53 ERROR : rclone_appending.txt: Failed to copy: "rclone_appending.txt" is locked: OCA\DAV\Connector\Sabre\Exception\FileLocked: 423 Locked

the next append rclone notices that the nextcloud file is a different length

2019/06/14 12:06:53 DEBUG : rclone_appending.txt: Sizes differ (src 103 vs dst 110)

however it can't find the file when trying to read it from next cloud back to the cache

2019/06/14 12:06:53 ERROR : rclone_appending.txt: Failed to copy: failed to open source object: File with name rclone_appending.txt could not be located: Sabre\DAV\Exception\NotFound: 404 Not Found

The DELETE is issued from the webdav backend when a PUT request fails.
	if err != nil {
		// Give the WebDAV server a chance to get its internal state in order after the
		// error.  The error may have been local in which case we closed the connection.
		// The server may still be dealing with it for a moment. A sleep isn't ideal but I
		// haven't been able to think of a better method to find out if the server has
		// finished - ncw
		time.Sleep(1 * time.Second)
		// Remove failed upload
		_ = o.Remove()
		return err
	}

This was put in as part of this change eb84b58d3c728be0834bb3bb39282346a41e6b1c
So to get some internal testing working properly
I've fixed that here - can you give it a go?
https://beta.rclone.org/branch/v1.47.0-131-gb1f1b45b-fix-3263-webdav-delete-beta/ (uploaded in 15-30 mins)
I think the decision for backends to delete half uploaded files is probably a bad one and one I need to correct.  I think this affects the ftp and sftp backends too.

Excellent! Thanks!
I tried it, and now, I was able to run that append loop a few times without issues. But it seems the problem is not gone yet.
One of my tests started complaining about file corruption:
2019/06/14 17:20:39 DEBUG : &{rclone_appending-beta2.txt (rw)}: Release:
2019/06/14 17:20:39 DEBUG : rclone_appending-beta2.txt(0xc00046d920): RWFileHandle.Release closing
2019/06/14 17:20:39 DEBUG : rclone_appending-beta2.txt(0xc00046d920): close:
2019/06/14 17:20:39 DEBUG : rclone_appending-beta2.txt(0xc00046d920): not modified so not transferring
2019/06/14 17:20:39 DEBUG : rclone_appending-beta2.txt(0xc00046d920): >close: err=<nil>
2019/06/14 17:20:39 DEBUG : &{rclone_appending-beta2.txt (rw)}: >Release: err=<nil>
2019/06/14 17:20:39 DEBUG : fuse: -> [ID=0x6c0] Release
2019/06/14 17:20:39 DEBUG : fuse: <- Flush [ID=0x6c2 Node=0x4 Uid=0 Gid=0 Pid=5753] 0x2 fl=0x0 lk=0xdc5111b9253357c6
2019/06/14 17:20:39 DEBUG : &{rclone_appending-beta2.txt (rw)}: Flush:
2019/06/14 17:20:39 DEBUG : &{rclone_appending-beta2.txt (rw)}: >Flush: err=<nil>
2019/06/14 17:20:39 DEBUG : fuse: -> [ID=0x6c2] Flush
2019/06/14 17:20:40 DEBUG : fuse: <- Write [ID=0x6c3 Node=0x4 Uid=0 Gid=0 Pid=5753] 0x2 7 @61 fl=0 lock=0 ffl=OpenWriteOnly+OpenAppend
2019/06/14 17:20:40 DEBUG : &{rclone_appending-beta2.txt (rw)}: Write: len=7, offset=61
2019/06/14 17:20:40 DEBUG : rclone_appending-beta2.txt: Couldn't find file - need to transfer
2019/06/14 17:20:40 ERROR : rclone_appending-beta2.txt: corrupted on transfer: sizes differ 54 vs 61
2019/06/14 17:20:40 INFO  : rclone_appending-beta2.txt: Removing failed copy
2019/06/14 17:20:40 DEBUG : &{rclone_appending-beta2.txt (rw)}: >Write: written=0, err=open RW handle failed to cache file: corrupted on transfer: sizes differ 54 vs 61
2019/06/14 17:20:40 DEBUG : fuse: -> [ID=0x6c3] Write error=EIO: open RW handle failed to cache file: corrupted on transfer: sizes differ 54 vs 61
2019/06/14 17:20:40 DEBUG : fuse: <- Flush [ID=0x6c4 Node=0x4 Uid=0 Gid=0 Pid=5753] 0x2 fl=0x0 lk=0xdc5111b9253357c6
2019/06/14 17:20:40 DEBUG : &{rclone_appending-beta2.txt (rw)}: Flush:
2019/06/14 17:20:40 DEBUG : &{rclone_appending-beta2.txt (rw)}: >Flush: err=<nil>
2019/06/14 17:20:40 DEBUG : fuse: -> [ID=0x6c4] Flush

From the client side, the file is now only partially readable:
cat /mnt/rclone/rclone_appending-beta2.txt
run 1
run 2
run 3
run 4
run 5
run 6
run 7
run 8
run 9
run 10
cat: /mnt/rclone/rclone_appending-beta2.txt: Input/output error

And on the server:
<IP_ADDRESS> -  [14/Jun/2019:17:20:31 +0200] "PUT /remote.php/nonshib-webdav/rclone_appending-beta2.txt HTTP/1.1" 204 - "-" "rclone/v1.47.0-131-gb1f1b45b-fix-3263-webdav-delete-beta" microsecs:2601331 response_size:0 bytes_received:666 bytes_sent:945 XQO7P1i7UxbKFqghXtJCPgAAAIs - 17982
<IP_ADDRESS> -  [14/Jun/2019:17:20:34 +0200] "PROPFIND /remote.php/nonshib-webdav/rclone_appending-beta2.txt HTTP/1.1" 207 739 "-" "rclone/v1.47.0-131-gb1f1b45b-fix-3263-webdav-delete-beta" microsecs:370438 response_size:739 bytes_received:783 bytes_sent:1657 XQO7Qli7UxbKFqghXtJCPwAAAIs - 17982
<IP_ADDRESS> -  [14/Jun/2019:17:20:34 +0200] "PUT /remote.php/nonshib-webdav/rclone_appending-beta2.txt HTTP/1.1" 423 246 "-" "rclone/v1.47.0-131-gb1f1b45b-fix-3263-webdav-delete-beta" microsecs:2360389 response_size:246 bytes_received:673 bytes_sent:1112 XQO7Qli7UxbKFqghXtJCQAAAAIs - 17982
<IP_ADDRESS> -  [14/Jun/2019:17:20:37 +0200] "GET /remote.php/nonshib-webdav/rclone_appending-beta2.txt HTTP/1.1" 200 61 "-" "rclone/v1.47.0-131-gb1f1b45b-fix-3263-webdav-delete-beta" microsecs:1648823 response_size:61 bytes_received:460 bytes_sent:1203 XQO7RVi7UxbKFqghXtJCQQAAAIs - 17982
<IP_ADDRESS> -  [14/Jun/2019:17:20:37 +0200] "GET /remote.php/nonshib-webdav/rclone_appending-beta2.txt HTTP/1.1" 200 61 "-" "rclone/v1.47.0-131-gb1f1b45b-fix-3263-webdav-delete-beta" microsecs:1648823 response_size:61 bytes_received:460 bytes_sent:1203 XQO7RVi7UxbKFqghXtJCQQAAAIs - 17982

About that sleep, ownCloud does some processing after a file upload has been received by the server. And if ownCloud has an objectstore as backend (as opposed to a local filesystem), it needs to upload the file to the objectstore as well. From the client perspective, this is seen as part of the PUT. It's best not to expect that this will be a quick and fast process.
Since from the client side it's impossible to know what storage backend is behind ownCloud, a possible solution might be to retry n times when the error returned is 423?
Not sure if that is a proper solution, I'm just trying to think around the problem that there is so much processing going on behind the scenes that make the process unpredictably slow for the client.

I've made webdav retry on 423 errors and I've stuck that in the latest beta (uploaded in 15-30 mins) which is shortly to become v1.48.  This is controlled by the --low-level-retries flag.
I haven't merged the delete fix yet as I want to work on it some more after the 1.48 release but I think the retry on 423 will probably fix the issue...

Ah nice, thank you!
I've been running some tests, and the simple append test I'd been using now goes flawlessly. It still encounters the same 'file locked' situations, but the retry does the trick. Also, the data inside the file is as expected (no missing lines or so)
2019/06/17 09:59:58 DEBUG : &{rclone_appending-latest-beta.txt (rw)}: Flush:
2019/06/17 09:59:58 DEBUG : rclone_appending-latest-beta.txt(0xc0004f9aa0): close:
2019/06/17 09:59:58 DEBUG : rclone_appending-latest-beta.txt: Sizes differ (src 1096 vs dst 1089)
2019/06/17 09:59:58 DEBUG : pacer: low level retry 1/1 (error "rclone_appending-latest-beta.txt" is locked: OCA\DAV\Connector\Sabre\Exception\FileLocked: 423 Locked)
2019/06/17 09:59:58 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2019/06/17 10:00:11 DEBUG : pacer: Reducing sleep to 15ms
2019/06/17 10:00:11 DEBUG : rclone_appending-latest-beta.txt: Received error: "rclone_appending-latest-beta.txt" is locked: OCA\DAV\Connector\Sabre\Exception\FileLocked: 423 Locked - low level retry 1/10
2019/06/17 10:00:15 DEBUG : pacer: Reducing sleep to 11.25ms
2019/06/17 10:00:15 DEBUG : pacer: Reducing sleep to 10ms
2019/06/17 10:00:15 INFO  : rclone_appending-latest-beta.txt: Copied (replaced existing)
2019/06/17 10:00:15 DEBUG : rclone_appending-latest-beta.txt: transferred to remote
2019/06/17 10:00:15 DEBUG : rclone_appending-latest-beta.txt(0xc0004f9aa0): >close: err=<nil>
2019/06/17 10:00:15 DEBUG : &{rclone_appending-latest-beta.txt (rw)}: >Flush: err=<nil>
2019/06/17 10:00:15 DEBUG : fuse: -> [ID=0x211] Flush

I ran this a couple of times without issues, and so now I've also tried appending to a bigger file, one that's bigger than the chunk size of the objectstore backend (ie. 150MB) because I already know that is slow. And that's also going fine now with the retries.
So I can confirm, the issue is gone. Thanks again for your quick responses and action :)

I'm glad it is working now :-)
I factored the whole deleting thing into another issue: #3274
I think this needs to be resolved separately.


Added chakra theme, html meta, and home page

Adds chakra theme, and home page with components:

Header
Footer
Hero carousel
Recent posts
Recent events (WIP)

Favicon report: https://realfavicongenerator.net/favicon_checker?protocol=https&site=fe-f506p5ov1-cse-it.vercel.app


Update go version in github actions workflows to 1.17

Current Behavior
Many of places in the workflows still mention go version 1.16 yet the adapters use go version 1.17
Desired Behavior
Workflows shouls also use go version 1.17
Implementation

Acceptance Tests

Mockups

Contributor Guides and Resources

🛠 Meshery Build & Release Strategy
📚 Instructions for contributing to documentation

Meshery documentation site and source

🎨 Wireframes and designs for Meshery UI in Figma
🙋🏾🙋🏼 Questions: Layer5 Discussion Forum and Layer5 Community Slack

Can I take this up?

There you go.

Closed by #149


shellout to find is traversing /proc

This is kind of crazy:
https://github.com/sous-chefs/selinux_policy/blob/master/providers/fcontext.rb#L41
You occasionally run into errors like this:
[15:08:20][Step 4/7]          ---- Begin output of ["find", "/", "-regextype", "posix-egrep", "-regex", "/var/cache/nova(/.*)?", "-execdir", "restorecon", "-iRv", "{}", ";"] ----
[15:08:20][Step 4/7]          STDOUT:
[15:08:20][Step 4/7]          STDERR: find: ‘/proc/11889’: No such file or directory
[15:08:20][Step 4/7]          ---- End output of ["find", "/", "-regextype", "posix-egrep", "-regex", "/var/cache/nova(/.*)?", "-execdir", "restorecon", "-iRv", "{}", ";"] ----
[15:08:20][Step 4/7]          Ran ["find", "/", "-regextype", "posix-egrep", "-regex", "/var/cache/nova(/.*)?", "-execdir", "restorecon", "-iRv", "{}", ";"] returned 1

If you're going to shellout to find, you could determine the filesystem you need to search, and then use -xdev to avoid crossing filesystem boundaries, otherwise you could start searching large network storage, or hitting issues like this with ephemeral files in /proc.
Also using -execdir {}; forks for each file found.  xargs or {}+ passes multiple arguments to the utility being invoked, thereby reducing overhead.

@daften can you take a look at this? thoughts?

lol
We should probably add -xdev (Don't descend directories on other
filesystems.)
Should have thought about that
Nitzan Raz
http://backslasher.net
On Tue, May 2, 2017 at 10:21 PM, Grant Ridder<EMAIL_ADDRESS>wrote:

@daften https://github.com/daften can you take a look at this? thoughts?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/sous-chefs/selinux_policy/issues/71#issuecomment-298734435,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AG3He8m4tvX_LrCcCxKqLL7ApjWho0Jrks5r14IegaJpZM4NOkWm
.

Ah, did not know about the -xdev option :)

@acranox you have a chance at everlasting glory here. Translate your recommendations into a PR and submit! :smile:

One thought though... if the find is executed on / like it currently is and you add -xdev then won't the find fail if the directly (new_resource.file_spec) is not on the root partition?

that's true. We should probably eliminate virtual filesystems though.

That's correct, about running find on / if your resource isn't in that FS.
You'll need to determine what filesystem new_resource.file_spec is on first.
Don't count on me for a PR.  at least not in a timely fashion.  :)

Probably can use df <dir> and parse the mount point

I might still want to operate on other filesystems.
I think the best solution would be to avoid doing smart things, and having
some attribute ("restorecon_single_filesystem") to add this behavior.
Maybe we can by default avoid proc with something like this
https://unix.stackexchange.com/a/193277
Nitzan Raz
http://backslasher.net
On Tue, May 2, 2017 at 10:39 PM, Grant Ridder<EMAIL_ADDRESS>wrote:

Probably can use df  and parse the mount point
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/sous-chefs/selinux_policy/issues/71#issuecomment-298738899,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AG3HewyJMf4Kpafrkm06SeN6MJnnjpLWks5r14ZkgaJpZM4NOkWm
.

stat --format '%m'   might be lighter weight than df.

I got bit by this again today.  It looks like the find command starting traversing into an Amazon EFS volume, and after 600s, it timed out and the chef run failed.  I created a PR that may address it.

See #72

This can be closed now.


Rewrite module to be less monolithic

Currently, the python-vipaccess module does too many things. Ideally, it should just be a simple wrapper around Symantec's API with a command-line example program to generate an OTP URL, but as of the current version it also requires Pillow (and Django, which Pillow depends on) to be installed.

[ ] Remove QR code-generation functionality
[ ] Direct messages to STDERR and reserve STDOUT for the OTP URI

Closing this issue since I'm no longer maintaining this project and will be archiving it soon. Please use this fork instead.


ecole-isitech.fr école d'expert info

ecole-isitech.fr école d'expert info

http://ecole-isitech.com

@shoognight69 As far as I can see, this is a duplicate of #2861. I am closing this request without merging, because the same request has been already satisfied. Thank you.


Can't install to insomnia 2022.6.0

Failed to install it to 2022.6.0.

Somehow I managed to install the plugin manually.


Fix content uploader

Description
The latest content-uploader container did not work correctly in certain scenarios. This hotfix addresses such inssue.

/merge


test: add tests for CLIPMediaSim

as per title

The code part looks good to me! One thing I was thinking though to make the interface a little easier, instead of passing a timestamp per frame just pass FPS. I think that's a lot easier for the user to handle and is a common video-related argument

We synced offline and decided to remove the timestamps and return indices. I made the changes for that and added an example to get the timestamp in the documentation.


Not able to change IP/Port when run as root on macOS

We are using Satellite on a machine that is used by many people with different user accounts, in a space where sometimes people bring their own machine. Because of this, I'd like Satellite to run before any user has logged in. I've done this in the past with LaunchDaemons running as root, but for some reason I can't get the IP and port arguments to work as root.
Comparing the logs between running as root vs running as a user, there isn't any "Connecting to xxx.xxx.xxx.xxx:xxxx" line at all, so I'm not sure what the issue is.
Here's the log output from running as root:
sh-3.2# ./Companion\ Satellite <IP_ADDRESS> 16622
Starting
adding new device: IOService:/AppleACPIPlatformExpert/PCI0@0/AppleACPIPCI/XHC1@14/XHC1@14000000/HS14@14600000/4-Port USB 2.0 Hub@14600000/AppleUSB20Hub@14600000/AppleUSB20HubPort@14610000/Stream Deck Mini@14610000/IOUSBHostInterface@0/AppleUserUSBHostHIDDevice
existing = []
Registering key events for BL44H1B05882
App ready
set tray
objc[7974]: Class WebSwapCGLLayer is implemented in both /System/Library/Frameworks/WebKit.framework/Versions/A/Frameworks/WebCore.framework/Versions/A/Frameworks/libANGLE-shared.dylib (0x7ffb4846fec8) and /Applications/Companion Satellite.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Libraries/libGLESv2.dylib (0x112effce0). One of the two will be used. Which one is undefined.

The 'desktop' builds of satellite don't support passing the ip and port on the command line like this. They will only read them from the config file. Perhaps you can edit that file to how you want it setup?
Im surprised that it even runs at all, I would have expected it to not be able to open due to being run in the background without a ui

I can try that, where is the config file? I couldn't find the location of a config file originally, that's what got me started passing arguments

On my mac, it is at /Users/julus/Library/Application Support/companion-satellite/config.json

I'll give that a shot, I'll go ahead and close this since it really isn't an intended use case