[RFC] #[ramfunc]

This is a 2 in 1 RFC and implementation PR. This builds on top of #90 so you can ignore the first commit. See #42 for some background information.
@rust-embedded/cortex-m Please review this RFC and if you are in favor approve this PR.

Summary
Add a #[ramfunc] attribute to this crate. This attribute can be applied to
functions to place them in RAM.
Motivation
Running functions from RAM may result in faster execution times. Thus placing
"hot code" in RAM may improve the performance of an application.
Design and rationale
Expansion
The #[ramfunc] attribute will expand into a #[link_section = ..]
attribute that will place the function in a .ramfunc.$hash section. Each
#[ramfunc] function will be placed in a different linker section (in a
similar fashion to -ffunction-sections) to let the linker GC (Garbage Collect)
the unused functions . To this end, #[ramfunc] won't accept generic functions
because if the generic function has several instantiations all of them would end
in the same linker section.
link.x will be tweaked to instruct the linker to collect all the input
sections of the form .ramfunc.* into a single .ramfunc output section that
will placed in RAM.
The Reset handler will be updated to initialize the .ramfunc section before
the user entry point is called.
Why an attribute?
This is implemented as an attribute to make it composable with the attributes to
be added in RFC #90. For example you can place an exception handler in RAM. The
compiler won't generate a veener in this case as the address to the RAM location
will be placed in the vector table.
#[exception(SysTick)]
#[ramfunc]
fn systick() {
    // do stuff
}

Why a separate linker section?
Using a separate section lets the compiler / linker assign the right ELF flags
to .ramfunc (i.e. "AX"). That way .ramfunc ends up showing in the output of
objdump -C.
$ readelf -S app
  [ 1] .vector_table     PROGBITS        08000000<PHONE_NUMBER>00 00   A  0   0  4
  [ 2] .text             PROGBITS        08000400<PHONE_NUMBER>ea 00  AX  0   0  2
  [ 3] .rodata           PROGBITS        080007ec<PHONE_NUMBER>00 00  WA  0   0  4
  [ 4] .data             PROGBITS        20000000<PHONE_NUMBER>04 00  WA  0   0  4
  [ 5] .ramfunc          PROGBITS        20000004<PHONE_NUMBER>1c 00  AX  0   0  4
  [ 6] .bss              NOBITS          20000020<PHONE_NUMBER>00 00  WA  0   0  4

$ arm-none-eabi-objdump -Cd app
Disassembly of section .text:

08000400 <Reset>:
 8000400:       f000 f9ee       bl      80007e0 <DefaultPreInit>
 (..)

Disassembly of section .ramfunc:

20000004 <SysTick>:
20000004:       b580            push    {r7, lr}
(..)

This also means that .data would only contain ... data so the section can be
inspected (not disassembled) separately.
$ arm-none-eabi-objdump -s -j .data app
Contents of section .data:
 20000000 01000000                             ....

The downside (?) is that the default format of size will report .ramfunc
under text.
$ size app
   text    data     bss     dec     hex filename
   2066       4       4    2074     81a app

But in any case it's (usually) better to look at the output in System V format.
$ size -Ax app
app  :
section             size         addr
.vector_table      0x400    0x8000000
.text              0x3ea    0x8000400
.rodata              0x0    0x80007ec
.data                0x4   0x20000000
.ramfunc            0x28   0x20000004
.bss                 0x4   0x2000002c

Unresolved questions?

Is there a reliable way to initialize both .data and .ramfunc in a single
for loop? (see init_data in the Reset function) Right now I initialize
each one separately and this produces more code. I haven't tried initializing
them in a single pass.

Should #[ramfunc] imply #[inline(never)]? Right now the compiler is free
to inline functions marked as #[ramfunc] into other functions. That could
cause the function to not end in RAM.

Bikeshed the attribute name and linker section name? IAR and TI use the name
ramfunc for this functionality.

Oh and I have not tested this on hardware. I have only looked at the output of objdump so it would be good if someone tested it to see if it actually works ...

@japaric This is fantastic! Especially the ability to have exception/interrupt handlers without veneer.

Should #[ramfunc] imply #[inline(never)]? Right now the compiler is free
to inline functions marked as #[ramfunc] into other functions. That could
cause the function to not end in RAM.

Yes, we should do that.

Something that should be mentioned in the docs is that #[ramfunc] does not automatically propagate through the call stack. By this I mean that code like:
#[ramfunc]
fn foo() {
    // ..
    bar();
    // ..
}

fn bar() { .. }

Could result in foo being executed from RAM and bar being executed from Flash unless bar gets inlined into foo but that would be up to the compiler.
Sure you can add #[ramfunc] to bar if you wrote it but if you are calling third party code, like libm, you have pretty much no control over whether something like sinf will end in Flash or RAM.

Also I agree with @therealprof, a ramfunc should not be inlined, that destroys the purpose. :)

Sounds good to me. I'm quite used to this sort of thing from working with big-name RTOS from a vendor that rhymes with "Nentor" and the little pitfalls tend to be fairly well known by people who have a need for this.

Should #[ramfunc] imply #[inline(never)]?

:+1:

Bikeshed the attribute name and linker section name?

I'm fine with ramfunc.

To this end, #[ramfunc] won't accept generic functions
because if the generic function has several instantiations all of them would end
in the same linker section.

Is that actually a problem? Presumably if the instantiation was generated it's because it's in use, so wouldn't be eliminated by the linker later anyway, but I might be misunderstanding how the generic instantiations get made. In any event it seems like it might be more useful to be able to have ramfunc generics than not. Since calling a function from a ramfunc doesn't make the called function live in RAM unless it gets inlined, you can't even make your own type-specific instantiations which call the generic and be confident it will still be in RAM.
Otherwise this looks good. As @korken89 mentioned it would be really nice if we could specify what section ramfuncs end up in, but I think we also want that for statics, and for things like #59, and even being able to move stacks to CCM. The same consideration about being able to initialise in a single loop would apply to non-.data statics too.

@korken89
Continuing thread https://github.com/rust-embedded/cortex-m-rt/pull/100#discussion_r214315147 here because review comments get hidden when new commits are pushed.
We could support placing functions in CCRAM by adding a #[ccramfunc] attribute but this AFAICT this would require all users to define a CCRAM.
An alternative is to extend the #[ramfunc] to take an integer argument instead of adding a new attribute. This could be generalized to support any number of RAM regions. That is, #[ramfunc(0)] would place a function in .ramfunc0, #[ramfunc(1)] would place a function in .ramfunc1 and so on. Then .ramfunc0 would go in region RAM0, .ramfunc1 in RAM1, and so on.
This can be further generalized to allow placing static mut variables in different RAM regions. At that point, it would make more sense to rename the attribute to just #[ram]. Then #[ram] fn foo() would put foo in .ramfunc0 (no argument is equivalent to passing 0 as the argument), and #[ram(1)] static mut FOO: u32 = 0 (*) would put FOO in .data1 (**). This opens the question of how many regions to support by default? Just 1? And the question of how to let users configure the number of regions? Cargo features (being booleans) seems like a bad fit for this kind of configuration.
(*) Using #[ram] with static variables is tricky and potentially error prone. A static variable could end in .rodata (e.g. a string literal) or in .data (if it contains something with interior mutability or an atomic). The attribute doesn't have enough information (just the item AST) to determine if a static should really go in RAM so a misuse could cause a string literal to be stored in RAM, which would be wasteful.
(**) Similarly the attribute doesn't have enough information to determine if something should go in .bss or .data so putting everything in .data is the safe and conservative choice although it can unnecessarily increase the initialization time.
On the one hand, support for multiple memory regions looks like a breaking change so it would be good to batch it with breaking addition of attributes. On the other hand, I don't have the bandwidth to sit down and design something reasonable at the moment so I would prefer to postpone it for v0.7.0.

I'm wondering if having #[ramfunc] imply #[inline(never)] is more error prone or at least less flexible than leaving #[inline(never)] as a separate attribute. Imagine one of the simplest scenario where you need to call a function foo that calls another function bar. You want the call to foo to execute from RAM for $reasons. I might write this:
#[ramfunc]
fn foo() {
    // ..
    bar(args);
    // ..
}

#[ramfunc]
fn bar(args: Args) { .. }

If #[ramfunc] implies #[inline(never)] you forbid LLVM from inlining bar into foo which could hamper optimizations. In this case inlining is something that may be favorable so it's best to leave LLVM do its thing.
OTOH, if #[ramfunc] doesn't imply #[inline(never)] then you really should mark foo as #[inline(never)] otherwise it may get inlined into the caller function that's running from Flash. So if we go this route we should have guideline to mark "entry" RAM functions as #[inline(never)].
Or, instead of that guideline we could add a ramcall!(fun) macro that calls fun through a thunk:
// `ramcall!(fun)` expands into:
{
    #[ramfunc]
    #[inline(never)]
    fn thunk() { fun() } // LLVM will likely inline `fun` into `thunk`
    thunk();
}

Then the recommendation would be to use ramcall!(fun) when you want to execute fun from RAM and the caller is being executed from Flash. This may work correctly even if fun is not a #[ramfunc]. The problem here is how to implement ramcall! so that it supports inputs for fun ergonomically.

@therealprof

True, but why would you do that? If you wanted to make sure it's inlined

Because you are not 100% sure whether you inlining it is better or not -- that's the job of LLVM. What you want to be sure of is that if it's not inlined then bar should still be placed in RAM and you would accomplish that with #[ramfunc] #[inline] fn bar(..) but #[ramfunc] must not imply #[inline(never)].
@adamgreig

On inline

That might work. It seems #[ramfunc] can see all the other attributes regardless of whether they come before or after it.

On sections:

But having more memory regions is not zero cost. The loop for initializing .{bss,data}{0,1,...N} will be kept in the binary even if the section is empty. You already see this today with .bss and .data: the compiler doesn't know if a section is empty because that information is known only after linking. Thus the number should be configurable and not fixed.
Manually placing things in .bss is unsafe (e.g. placing static mut FOO: Enum = Enum::NonZeroVariant is UB) so the attribute should indicate that the operation is unsafe and using the attribute should be rejected by #[deny(unsafe_code)].
re: generics

Is that actually a problem? Presumably if the instantiation was generated it's because it's in use, so wouldn't be eliminated by the linker later anyway

Not necessarily. The default is to use multiple codegen units -- basically the Rust source code is split in chunks and each one is codegen-ed independently of the other. A simple example of instances lowered to machine code but still GC-ed is: you have foo that on some condition calls bar which in turns calls baz::<i32>. In a multiple codegen scenario foo and bar are processed separately. bar causes baz to be instantiated. While translating foo is observed that the branch that calls bar is unreachable so it gets eliminated. Now it's the job of the linker to discard the unused baz::<i32> instance.
The scenario you are picturing is probably only achievable using fat LTO (codegen-units = 1), but I'm not 100% sure it's guaranteed to occur.

The main aspect here is respecting the declared wish of the developer: if a function is declared #[ramfunc] it has to go to RAM, the way to ensure that is by declaring #[inline(never)]. If there's a better way to ensure that -- no matter what --  a #[ramfunc] function will always end up in RAM (inlined in another #[ramfunc] or standalone), that's fine by me. Something that should not happen is that the compiler decides it would be better to inline a #[ramfunc] into code residing in flash...

@japaric wrote this:

Sure you can add #[ramfunc] to bar if you wrote it but if you are calling third party code, like libm, you have pretty much no control over whether something like sinf will end in Flash or RAM.

I recently had the need to write a function that is guaranteed to run from RAM (no access to Flash allowed before the function is finished). I found it impossible to do that in Rust, for the reason mentioned in the quote. Unfortunately that makes this feature useless for my use case. I don't know what could be done to change that.
I left a pull experience report here: https://github.com/rust-embedded/cortex-m-rt/issues/42#issuecomment-559061416

Context_for_my_usecase - just wanted to see if there are updates for this thread. I'm happen to be working on a driver that writes a blob of data to internal flash. As ramfunctions require all dependent functions to also be located in RAM, I was wondering if its makes sense to have board specific HALs containing an NVMC abstraction to use the attribute for relevant flash operations.

I'm assuming, not all vendors or boards include support for code in flash writing to flash. In such a case, does it make sense to use a board-specific hal (even if one's available) or would it be better to roll your own low-level register-based impl, given that ramfunctions only work if all dependent functions are also in RAM.
I'm not well-versed with C, does C (or c compiler) have better support for this?


JarResolver related build tool refactor

Description:
JarResolver was introduces to resolve all the native dependencies of modules. We need to refactor the other places of the build tasks to use the JarResolver.

[x] Run Test task as to use the JarResolver
[x] Copy Native lib will go away
[x] Copy module jar task will go away
[x] Remove executable jar data holder
[x] Remove bre/lib loading in RunExecutable task

[ ] CreateExecutableTask has to use the JarResolver
[ ] Modify createJarTask according to the above changes
[ ] Add methods to get allTestDependecies and getRuntimeJar in jarResolver
[ ] Validate balo dependencies during compile task


Controlling tabindex focus upon re-render

Question
Is there a way to control the location of the tabindex focus after a Sprig render?
For example, if I have three Sprigified tab links, I would like the tabindex focus to remain on the clicked tab link. This is especially important for accessibility use cases where a user tabs to a specific tab (the tab receives keyboard focus) and hits Return. Instead of the tab focus remaining on the activated tab link, the tab focus seems to lock focus on links in my footer. It even skips the links from the returned, rerendered content—a bit mysterious and bizarre.
Additional context
The tabindex focus seems to work as it should with dropdown (and checkbox fields). When you keyboard select a dropdown option, the focus still remains on the same dropdown field after a Sprig re-render.
In short, I'm trying to replicate the same behavior with non-dropdown links.

I believe that the focus is maintained for elements that have IDs, but you'll need to ask over in https://github.com/bigskysoftware/htmx/issues since this is a htmx issue.


Add round_up_to() function for convenience.

Fixes #15 — Convenience functions

Not going to do this since it’s better to make things explicit. I can always add it later.


Add font-family declaration for the checkmark used on checkboxes

Currently, the checkbox checkmark uses the checkmark CSS character but if the font doesn't have this character or the designer wants to use something else, it falls apart.
We originally thought about switching to an "image" based solution. Ideally, it would be a data-uri and not require a separate image asset. And if could be added as default and client SCSS variable where the necessary CSS are mapped to custom properties.
Instead I think we can just add a font-family property to the checkbox set to inherit the current font, but which can be overridden if needed.
https://github.com/4site-interactive-studios/engrid-scripts/blob/a4794e411f6d664e9891aab7b5cf094517199332/packages/styles/src/_engrid-fields.scss#L753

I see the un-encoded checkmark rarely, but it's still happened. Not sure if we want to change this, but still worth a review.


Game terminating without any errors when logging into the game

After updating to v<IP_ADDRESS>, my game crashes after freezing for 2 seconds after getting through the login queue and I see the loading screen. I don't get any error message pop ups, the game just terminates itself and I'm at my desktop screen shortly after. I currently only have GShade installed (no mods installed) and followed the instruction to install GShade first before installing XIVAlexander.
I've tried launching the game without Alexander (only GShade installed) and managed to get into game just fine. I've also uninstalled v49 and installed v48, and I was able to get into the game with v48.
One weird issue I'm seeing is I have an alt on the Chaos DC and I'm able to log into the game no problem on that character, but my Elemental DC, which is my main, is always the character crashing on logging in.

Update: After doing additional tests and multiple reinstalls, I've managed to pin point the issue was being caused by XIVLauncher and not XIVAlexander. Closing this issue.


Update the appendix

Very in-depth disk space example
Explanation of miniwdl and Cromwell's configuration files, how to use them, and links to the templates that explain everything within then
Brief mention of hog factors, which are too complicated to explain in detail but worth linking out to Cromwell's explanation

Broken URL error appears to be a red herring; it's components that have since been updated on main.

Note to self: Merge this after https://github.com/fhdsl/WDL_Workflows_Guide/pull/46 gets merged!


Use flake8 instead of pyflakes and pep8

When trying to contribute, I struggled with unhelpful pep8/pyflakes issues on https://github.com/stephenmcd/mezzanine/pull/1734 and eventually found the workarounds to 'fix' the issue - https://github.com/stephenmcd/mezzanine/blob/c7bc3e74d8487c6b23417ac099a46aa5c486c369/mezzanine/utils/tests.py#L22
It seems like flake8 would be a much better option:

It has a builtin, standard way of ignoring specific errors, for both PEP8 and its own, rather than requiring people to dig around and eventually find the custom workarounds like I did.
It has a builtin way of ignoring errors on a specific line, or even just a specific error on a specific line, rather than having to blanket ignore a whole set of errors.
It is configurable and extensible, and well documented.
It has various very useful plugins. For example, I use flake8-future-import to help eliminate a whole set of unicode issues.
In my experience it generally has better editor integration and better adoption in the community, making it easier for people to use it immediately, and obey it without having to run tests. Because it can be configured from a config file, warnings that appear inline in the editor are the same as required by the test suite, which is vital IMO.

I've been using flake8 to lint Python directly on my text editor and find
it very useful. Seems like a good idea.
On Sat, Feb 18, 2017 at 4:51 AM, Luke Plant<EMAIL_ADDRESS>wrote:

When trying to contribute, I struggled with unhelpful pep8/pyflakes issues
on #1734 https://github.com/stephenmcd/mezzanine/pull/1734 and
eventually found the workarounds to 'fix' the issue -
https://github.com/stephenmcd/mezzanine/blob/
c7bc3e74d8487c6b23417ac099a46aa5c486c369/mezzanine/utils/tests.py#L22
It seems like flake8 would be a much better option:

It has a builtin, standard way of ignoring specific errors, for both
PEP8 and its own, rather than requiring people to dig around and eventually
find the custom workarounds like I did.
It has a builtin way of ignoring errors on a specific line, or even
just a specific error on a specific line
http://flake8.readthedocs.io/en/latest/user/ignoring-errors.html#in-line-ignoring-errors,
rather than having to blanket ignore a whole set of errors.
It is configurable and extensible, and well documented.
It has various very useful plugins. For example, I use
flake8-future-import
https://pypi.python.org/pypi/flake8-future-import to help eliminate
a whole set of unicode issues.
In my experience it generally has better editor integration and
better adoption in the community, making it easier for people to use it
immediately, and obey it without having to run tests. Because it can be
configured from a config file, warnings that appear inline in the editor
are the same as required by the test suite, which is vital IMO.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/stephenmcd/mezzanine/issues/1736, or mute the thread
https://github.com/notifications/unsubscribe-auth/ACR8Pkaovg7qQ0GKW_1rfTx3sw1oBIwoks5rds09gaJpZM4MFFLi
.

Sounds good to me.

As of https://github.com/stephenmcd/mezzanine/commit/82dd9f1d4752da402573adc37b53b2e8573fe76a we're now using black + pep8


Backend Create Endpoint for User Sign up

Implement Signup endpoint-  User fills out sign up form. User submits form. Server sends email to validate user. When user clicks link in email, user gets validated and added to the server permanently.

done. Thank you, Arun.


Add "see also" section to readme

Lots of alternatives...

❯ np patch
 ✔ Prerequisite check
 ✔ Git
 ✔ Cleanup
 ✔ Installing dependencies
 ✔ Running tests
 ✔ Bumping version
 ✔ Publishing package
 ✔ Pushing tags

 similarity 1.1.1 published 🎉


Bring in _forms.scss styling to toolkit

Two of our three apps have duplicate _forms.scss files (the buyer app, and the supplier app), and instead of adding it to the admin app (to correct the correct spacing above the log in button), it made more sense to bring the styles into the toolkit.
Once this is merged, we can delete the duplicated CSS in the other two apps.

👍


Playing memory stream with text "Third Activation" over DirectSoundOut plays a weird sound at the end of playback

I am trying to play memory stream of data through DirectSoundOut. Everything works fine. but i am hearing unwanted "beep" sound at the end of playback. If I try to play "First Activation", "Second Activation" and "Third Activation" . The issue happens only with "Third Activation". Something to do with this text. Any help would be appreciated.
Here is the code
///
/// Plays the audio stream.
///
/// The player.
public bool TransmitAudio(MemoryStream audioStream)
{
IWavePlayer waveOut = CreateWaveOut(_audioDeviceGuid);
        waveOut.PlaybackStopped += _playBackStopped;

        ISampleProvider sampleProvider = CreateInputStream(audioStream);

        //Initialize the WavePlayer
        waveOut.Init(sampleProvider);

        //Play audio
        waveOut.Play();

        _interfaceResetEvent.WaitOne();

        //Reset the manual reset event
        _interfaceResetEvent.Reset();

        //Return true
        return true;
    }

I'm afraid I don't understand how the text "third activation" relates to the code you have shown.


Add argument to remove only pods violating allowlisted taints

Fixes #1353.
Adds the option to apply the RemovePodsViolatingNodeTaints strategy only for an explicitly allowlisted set of taints.
When a list of includedTaints is provided, a taint will be considered if and only if it matches an included key or key=value from the list. Otherwise it will be ignored. Leaving includedTaints unset will include any taint by default.

/ok-to-test

should we let users set both included and excluded taints? When we look at namespace inclusion/exclusion we make inclusion be something like: "this and nothing else". Shouldn't we keep consistency here?
@a7i @damemi @ingvagabund

should we let users set both included and excluded taints? When we look at namespace inclusion/exclusion we make inclusion be something like: "this and nothing else". Shouldn't we keep consistency here?
@a7i @damemi @ingvagabund

I'm for using one or the other until there's a use case for both. I had mentioned in the original issue that it didn't seem harmful to do both, but I agree that it's safer and more consistent to make them exclusive for now.

should we let users set both included and excluded taints? When we look at namespace inclusion/exclusion we make inclusion be something like: "this and nothing else". Shouldn't we keep consistency here?

@damemi also pointed this out in the issue.
I started looking into how it's done for namespaces, but I wasn't sure where would be the right place to check in this case. If it would be within the strategy plugin or if there is some linter parsing the deschedulerPolicy resource before (which I didn't find). So I proposed the change without it, but I'm happy to add it, if you could give me some guidance on what you consider the best practice for it!

+1 for allowing to set only one of include/exclude lists, not both.

If it would be within the strategy plugin

Yeap. Each strategy/plugin is responsible for its own arg validation. E.g. https://github.com/kubernetes-sigs/descheduler/blob/master/pkg/framework/plugins/removeduplicates/validation.go

Thanks @ingvagabund! I added the validation to allow only one of include/exclude taints. Please have a look.
cc @knelasevero @damemi

After Jan's comments are addressed this lgtm

/approve
/lgtm

Thank you all for your quick responses and helpful feedback! This was a great experience as a first time contributor. :)


Fix translations

Description
Changes proposed in this pull request:

...
...
...

Related issue(s)

/retest


isort and black pre-commit conflict not solved by changing setup.cfg

Running into a problem where isort and black have a conflict, so failing pre-commit. Adding
[isort]
profile = black

to the setup.cfg fixes at the CLI but not pre-commit.

This is now fixed using pyproject.toml


Thumbnail function returns in the JS-API are empty.

It returns a basic BASE64 result which is not the image that's being requested.
I just enabled HTTPS on my server, could that have something to do with it? The rest of Cockpit works completely fine.

Could it be that image GD isn't working on PHP 7.0?


Add realistic example

Fixes #32(?)
Replaces the example ones tensor with a more realistic example, based on a Pytorch Resnet example.
To do:

[ ] Update resnet_infer_fortran.f90 to read in the saved text file for the image
[ ] Update resnet_infer_fortran.f90 to highest probabilities with labels (possibly via saving the loaded labels in python, in a similar manner to the image array?)

Thanks for starting this. It looks great so far :+1:

* Whilst it is arguably 'cleaner' in terms of storage to fetch the image from url, there is no guarantee it will always be available. Since it is only a single image I would consider just adding it to the repository to make the example self-contained. This will also arguably make the python a bit cleaner.

What about something in between (which I'm halfway along, having added the .jpg to the PR): try to download the data, but, if necessary, catch the HTTPError for both the image and the categories, as we can direct users to the pre-downloaded files.
I quite like having some form of the fetch present, as, for example, I wanted to test a different dog, which only requires changing one line of code at the moment, and I think it makes an example better if the barrier to trying out different data is as low as possible.

Yes, testing alternative images did cross my mind, but the purpose of this example is to serve as an (ideally minimal) example of how to save a pytorch net and couple it to Fortran. To this end I'd maybe favour having the image in the repo to read from and removing the requests code, perhaps with a note in the Readme to describe how a user might go about trying other images.

@jatkinson1000 What sort of thing would you consider to be nicer? My feeling was that flattening it would make it the least ambiguous, or do you mean the way it is flattened/saved? (I'll add a comment about writing/reading anyway)

For the MiMA case we wrote out with the indices with the value (e.g. see here) to remove any ambiguity when reading back in and make interpretable.

At some point @TomMelt mentioned that it would be nice to have an assert to check that the results are actually consistent e.g. call assert_real_2d(big_array, big_result/2., test_name=msg) (currently unused) in the resnet benchmarking branch.
Given that the probabilities are more complex than that example, and seem to only match to ~5sf, is it worth just adding an assert for the label? Or maybe we should check the first 4-5 significant figures of the probability?

Also, the python code could be more black, pylint, and mypy compliant.

Was there anything obvious that you haven't mentioned? I'd already run it my main changes to resnet18.py through flake8 and black, and I think there were only a couple of minor things in resnet_infer_python.py, which I hadn't looked at enough yet (as part of the last set of changes, it's now updated to use the new image, as it was still using the ones.)
I've made a few more minor changes based on a few things from pylint/mypy too in the last commit, so hopefully it's ok now?

At some point @TomMelt mentioned that it would be nice to have an assert to check that the results are actually consistent e.g. call assert_real_2d(big_array, big_result/2., test_name=msg) (currently unused) in the resnet benchmarking branch.
Given that the probabilities are more complex than that example, and seem to only match to ~5sf, is it worth just adding an assert for the label? Or maybe we should check the first 4-5 significant figures of the probability?

The assert_real_2d subroutine has an optional argument which allows you to change the relative tolerance. So I think a good idea would be to use assert_real_2d (or something like it) but just lower the tolerance to acknowledge that the variable in question has higher inaccuracy.
snippet of assert_real_2d

  subroutine assert_real_2d(a, b, test_name, rtol_opt)

    implicit none

    character(len=*) :: test_name
    real, intent(in), dimension(:,:) :: a, b
    real, optional :: rtol_opt
    real :: relative_error, rtol

Thanks @TomMelt! I think addressed all of your Fortran variable suggestions.
I also wanted to double check you were happy with the changes just before/at the same time as your review along side the assertion checks, which included the addition of wp separately to c_wp (the possibility of which was part of the intention for c_wp in the first place), since I'd changed probabilities to be a normal Fortran real.


[pulsar]Delete folders that do not need to be submitted

Motivation
This folder is used where node dependency modules are stored, it appears when using yarn and npm, there is no need to keep it, I think it was committed by mistake.


KongCredentials are not associated immediately or never with KongConsumers

NOTE: GitHub issues are reserved for bug reports only.
For anything else, please join the conversation
in Kong Nation https://discuss.konghq.com/c/kubernetes.

Summary
When a KongConsumer is created along with an associated KongCredential, it can take anywhere between 0 to infinite time in sec for the KongCredential records to get stored in Cassandra. The KongConsumer info is immediately stored whereas the credential info shows an unacceptable amount of latency almost every time.
As a result, Authentication fails as Kong can't find a valid consumer-credential pair.
Note:
Latency is mostly seen from creation of the second consumer and credential onwards or if CRDs are created back to back one after the other.
Kong Ingress controller version
0.3.0
Kong or Kong Enterprise version
1.0.3
Kubernetes version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.5", GitCommit:"f01a2bf98249a4db383560443a59bed0c13575df", GitTreeState:"clean", BuildDate:"2018-03-19T15:59:24Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.5", GitCommit:"f01a2bf98249a4db383560443a59bed0c13575df", GitTreeState:"clean", BuildDate:"2018-03-19T15:50:45Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Environment

Cloud provider or hardware configuration: Vmware Vsphere/K8s
OS (e.g. from /etc/os-release): CentOS 7
Kernel (e.g. uname -a): Linux ****-master-01 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Install tools: YAML/Jenkins
Others:

What happened
Unacceptable latency occurred whenever a new consumer and credential crd objects were created and authentication kept failing.
I created the following CRDs :
apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
  name: consumer-pqr
username: pqr
---

apiVersion: configuration.konghq.com/v1
kind: KongCredential
metadata:
  name: credential-pqr
consumerRef: consumer-pqr
type: basic-auth
config:
  username: pqr
  password: pqr

I then checked cassandra records and there is no credential for consumer with username 'pqr'.

API call result:

Expected behavior
KongConsumers and KongCredentials should get created at the same time and authentication plugins should be effective immediately.
Steps To Reproduce

Create a KongConsumer and KongCredential and apply/create deployment.
Make sure both the CRDs are created without any error.
Check cassandra records or try authenticating
Authentication Failed due to "Invalid authentication credentials"

I also encounter the issue.
Has anyone any idea to fit it?

I found that if we recreated consumer, it will sync the credentials.
Hope developer team to fix it.
Thx.

Hey. So, I tried recreating the consumer. I simply deleted the previous consumer and created another with the same config and name using kubectl create. It doesn't sync the credentials. I also tried kubectl apply without deleting the KongConsumer crd. However, that doesn't help either.
What steps did you follow to make the sync work?

I found that if we recreated consumer, it will sync the credentials.
Hope developer team to fix it.
Thx.

@HsinHeng
Hey. So, I tried recreating the consumer. I simply deleted the previous consumer and created another with the same config and name using kubectl create. It doesn't sync the credentials. I also tried kubectl apply without deleting the KongConsumer crd. However, that doesn't help either.
What steps did you follow to make the sync work?

versions:

kong-ingress-controller:0.3.0 for "kong-ingress-controller" Deployment.
kong:1.0.3-alpine for "kong-proxy"  Deployment.

Steps for me:
Use  "kubectl delete" to delete related "KongPlugin" & "KongConsumer".
And then use  "kubectl apply" to create "KongPlugin" & "KongConsumer".
Hope it could help you.

Also experiencing this issue. I find the credentials do eventually populate but it takes a couple of hours. Is this expected behavior?

Also experiencing this issue. I find the credentials do eventually populate but it takes a couple of hours. Is this expected behavior?

@srleyva Can you share your steps to reproduce?
I've not been able to reproduce this locally yet.

Thanks for the response @hbagdi  I'll give some context as well. I am wokring with an existing set of users that I am attempting to import into kong via a utility script I've written.

My script generates k8s resource definitions for each consumer and credentials associated with each. (70 for our current environment)
# Example of one consumer but 70 are created
apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
  name: johndoe
  namespace: kong
username: johndoe
custom_id:  some-custom-id

# Example of one credential but 70 are created
apiVersion: configuration.konghq.com/v1
kind: KongCredential
metadata:
 name: johndoe-jwt
 namespace: kong
consumerRef: johndoe
type: jwt
config:
 secret: some-base64-encoded-key
 key: johndoeiss

kubectl apply -f consumers/ at this point all the consumers exist in Kong

kubectl apply -f credentials/ k8s successfully creates all 70 credential resources and they exist in k8s. However, only the first credential exists in kong with all subsequent credentials being created over next few hours.

To reproduce locally I would think creating bulk consumers and credentials is where the issues lies. I would use the Kong API directly but waiting on #246

I reproduced this issue above with a significant number of kong consumers (2750 consumers), using kong-ingress-controller v0.3.0
When I create the 2751th consumer, it is added to Kong's postgresql database by kong-ingress-controller pod but its jwt and basic auth credentials are never added, although I can list the credentials as custom resources from /apis/configuration.konghq.com/v1/namespaces//kongcredentials API route.
No error message is given either.
i was wondering if this had something to to with kong consumers synchronisation not being over before its next round starts?

An update here:
The way consumers and credentials are synced with 0.5.0 have significantly changed in 0.5.0.
I ran the tests ranging up to 4000 consumers and was not successful in reproducing this problem.
Please test it with 0.5.0 (expect at least a release candidate to be out next week) and let us know if you run into this yet again. I'll leave the issue open for now.

Closing this issue.
Please re-open if the issue persists for Kong Ingress Controller >=0.5.0.

An update here:
The way consumers and credentials are synced with 0.5.0 have significantly changed in 0.5.0.
I ran the tests ranging up to 4000 consumers and was not successful in reproducing this problem.
Please test it with 0.5.0 (expect at least a release candidate to be out next week) and let us know if you run into this yet again. I'll leave the issue open for now.

However，I reproduce this problem with 0.5.0，
postgresql database ：

k8s kongcredentials：

k8s Kong Ingress Controller：

Is there a problem with my configuration?
echo "
apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
  name: user2
username: user2
" | kubectl apply -n ai-test -f -

echo "
apiVersion: configuration.konghq.com/v1
kind: KongCredential
metadata:
  name: user2-key1
consumerRef: user2
type: key-auth
config:
  key: user2-secret1
  " | kubectl apply -n ai-test -f -

echo "
apiVersion: configuration.konghq.com/v1
kind: KongCredential
metadata:
  name: user2-key2
consumerRef: user2
type: key-auth
config:
  key: user2-secret2
  " | kubectl apply -n ai-test -f -

And api call failed :

Ask for help , thx

An update here:
The way consumers and credentials are synced with 0.5.0 have significantly changed in 0.5.0.
I ran the tests ranging up to 4000 consumers and was not successful in reproducing this problem.
Please test it with 0.5.0 (expect at least a release candidate to be out next week) and let us know if you run into this yet again. I'll leave the issue open for now.

However，I reproduce this problem with 0.5.0，
postgresql database ：

k8s kongcredentials：

k8s Kong Ingress Controller：

Is there a problem with my configuration?
echo "
apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
  name: user2
username: user2
" | kubectl apply -n ai-test -f -

echo "
apiVersion: configuration.konghq.com/v1
kind: KongCredential
metadata:
  name: user2-key1
consumerRef: user2
type: key-auth
config:
  key: user2-secret1
  " | kubectl apply -n ai-test -f -

echo "
apiVersion: configuration.konghq.com/v1
kind: KongCredential
metadata:
  name: user2-key2
consumerRef: user2
type: key-auth
config:
  key: user2-secret2
  " | kubectl apply -n ai-test -f -

And api call failed :

Ask for help , thx

While ,After about seven minutes , postgresql has credential record,

I am facing the same issue with 0.5.0 and Kong 1.3.0 with key-auth and Postgres.
It's been over 10 minutes, but the credential didn't sync.
However, if I first delete the keys (all keys for now) from Kong directly and then deleting one of the Credentials that synced, then the missing credential appears in Kong. Unfortunately, the deleted Credential (in K8s) also appears in Kong. (I know the Update and Delete are not yet implemented).
I hope I am clear. Please ask if it's confusing.
Thanks for your help! @hbagdi

@shugydw With https://github.com/Kong/kubernetes-ingress-controller/pull/371 merged in, the syncing of credentials is completely revamped and that should address this issue. Also, see #230.

@hbagdi i'm testing the new 0.6.0 with 1.3.0 kong and somehow the issue is now much worse. I'm not sure how i arrived at this problem but it seems that kong no longer picks up any consumers that are created either via kubectl or helm.
Thinking i somehow botched the upgrade from older kong-ingress i wiped everything. pulled the latest helm chart and reinstalled everything from scratch. When i try to create a consumer with the following:
apiVersion: v1
items:
- apiVersion: configuration.konghq.com/v1
  kind: KongConsumer
  metadata:
    name: dev-ml-trainer-user
    namespace: dev
  username: dev-ml-trainer-user
kind: List

Nothing happens at all. Kong ingress logs don't show anything at all. No error message of any kind.

@dcherniv Are you using the default ingress.class or a custom one?

@hbagdi kong-internal is the class. These are startup params:
      - args:
        - /kong-ingress-controller
        - --publish-service=kong-internal/kong-internal-kong-ingress-kong-proxy
        - --ingress-class=kong-internal
        - --election-id=kong-ingress-controller-leader-kong-internal
        - --kong-url=http://localhost:8444

Okay, then you need to specify kubernetes.io/ingress.class: "kong-internal" annotation on the KongConsumer (and every other custom) resources.

This is documented here: https://github.com/Kong/kubernetes-ingress-controller/blob/master/docs/references/annotations.md#kubernetesioingressclass
PR to make it clearer welcome!

@hbagdi What do you think about namespace level annotation? We need to change two many stuff's annotation when using different staging namespaces.

@hbagdi What do you think about namespace level annotation? We need to change two many stuff's annotation when using different staging namespaces.

Not totally against the idea. Another idea would be to watch a bunch of namespace. PRs/Issues to discuss this at length are welcome.
Let's not further discuss this on this closed issue.


Is there a mechanism to insert custom directives in the generated form's input elements

Question / Enhancement
As a user/developer, when I use angular-schema-form I want a mechanism to a mechanism to insert custom directives in the generated form's input elements
See this plunker for an example. Therein I used the dlErrorValidationHandlerCombined directive, in conjunction with a custom service and a couple of additional supporting directives to meet my validation requirements.

You could inject a new builder to apply the directives on elements with an ng-model jqLite query perhaps.
angular.module('myform', ['SchemaForm'])
  .config(configSchemaForm)
  .directive(sfNewDirective);

configSchemaForm.$inject = ['schemaFormDecoratorsProvider', 'sfBuilderProvider'];
function configSchemaForm(schemaFormDecoratorsProvider, sfBuilderProvider) {
  // Capture any existing builders if you want to inject the new builder in a specific order...
  var conditionBuilder = sfBuilderProvider.builders.condition;

  // For this example, we want to inject our new builder after everything except condition.
  angular.forEach(schemaFormDecoratorsProvider.decorator(), function (decorator, name) {
    if (angular.isArray(decorator.builder)) {
      var i = decorator.builder.indexOf(conditionBuilder);
      if (i > -1)
        decorator.builder.splice(i, 0, newBuilder);
      else
        decorator.builder.push(newBuilder);
    } else if (angular.isFunction(decorator.builder) {
      if (decorator.builder === conditionBuilder)
        decorator.builder = [newBuilder, decorator.builder];
      else
        decorator.builder = [decorator.builder, newBuilder];
    } else {
      decorator.builder = newBuilder;
    }
  }
}

function newBuilder(args) {
  var nodes = args.fieldFrag.querySelectorAll('[ng-model]');
  for (var i = 0; i < nodes.length; i++) {
    nodes[i].setAttribute('sf-new-directive','');
  }
}

sfNewDirective.$inject = [];
function sfNewDirective() {
  return {
    restrict: 'A',
    require: 'ngModel',
    scope: false,
    priority: 2,
    link: function(scope, element, attrs, ngModel) {
      if (!ngModel)
        return;

      // Do validation stuff here?  You have access to the ngModel controller if your doing validation through it.
    }
  };
}

@kyse thanks!
@dematic-rodrigo-silveira what he said :) Actually a better answer than mine to your other question too.
Please re-open if you have any further questions on the topic.


DH returns 415 when no Content-Type header defined

Please make Content-Type header non-required and use JSON request parser by default. Right now if you supply a valid JSON and no Content-Type the API will return HTTP 415.
This seems to be redundant  on this stage jsut because JSON media type is used everywhere and there is no alternative media types implemented in DH. This will also help users who like accessing DeviceHive API directly (using curl or Postman or whatever)
This issue (or you may call this a feature :smiley:) is reproducible at least for Network POST method.
Error example:
{
  "error": 415,
  "message": "HTTP 415 Unsupported Media Type"
}

I think we should not make Content-Type header optional for the number of reasons.
First of all it's not about having a some header string in a request. We need to explicitly check that sender wants to send exactly JSON string before we even try to parse it.
For example, default Content-Type for POST request is application/x-www-form-urlencoded which is used to send serialized HTML forms data. This means that the following code will produce the following payload: command=command&parameters[prop1]=val1&parameters[prop2]=val2&parameters[prop3]=etc which will cause JSON parsing error (assuming we are accepting all requests) on server-side and unexpected 500 Internal Server Error instead of code 415 with user-friendly error message:
$.ajax({
    type: "POST",
    url: "http://<IP_ADDRESS>:8080/dh/rest/device/e50d6085-2aba-48e9-b1c3-73c673e414be/command",
    headers: {"Authorization": "Bearer 1jwKgLYi/CdfBTI9KByfYxwyQ6HUIEfnGSgakdpFjgk="},
    data: {
        "command": "command",
        "parameters": {
            "prop1": "val1",
            "prop2": "val2",
            "prop3": "etc"
        }
    }
});

Sticking to standards also makes it easier for us to integrate with new tools, libraries and applications.
On the other hand, it's a good idea to transfer less bandwidth. For example, we can return 415 code on parsing error.
@demon-xxi @zubrabubra @SorJEF What do you think? Let's discuss and make some decision regarding this issue.

I would go with defaulting content type to application json and then making it the only choice for now.  What are the reasons device would decide to send us data form encoded?
As from device perspective both json and form encoded are not native, while json accepted by human beings easily, and when someone looks on command=command&parameters[prop1]=val1&parameters[prop2]=val2&parameters[prop3]=etc he become disappointed with a fact that he needs to catch symbols in this spagetti :)

@zubrabubra What are the reasons device would decide to send us data form encoded?
I was talking about

This will also help users who like accessing DeviceHive API directly (using curl or Postman or whatever)
Majority of tools will use default Content-Type if it's not omitted.

But yeah, from the device perspective it makes perfect sense to avoid sending redundant data.

Checked situation with curl and with Postman (form chrome). Check in debug mode, showed that curl send header Content-Type: application/x-www-form-urlencoded and Postman send header Content-Type: text/plain;charset=UTF-8. I assume that the client generates a default Content-Type. And it can cause problems. Therefore, the header must be specified certainly. I propose to close issue.

@YermakovSergiy i think this is specific behaviour only of curl, postman etc, and it is not available in low level http libraries. So it's better to go with Content-type: application/json or text/json if no header specified.

Added filter that checks request headers. If there is no Content-Type header then will be added  Content-Type : application/json. Can be cheked at branch: https://github.com/devicehive/devicehive-java-server/tree/defect/90

@YermakovSergiy  why it is in branch 90?


Refactor exact match

The current implementation of exact match is very inefficient, and is causing the NLP server to hit memory limits (#270).
We should:

add a flag to the model to decide whether the exact match should be used or not
regardless, load only one copy of the exact match per language
store exact match datasets on disk / S3 instead of mysql
use a mmap-able binary format that can be swapped out as needed and avoids building the trie in memory

The last two changes would make us lose incremental updates. At the same time, the generation is not incremental, and the process is currently CPU bound by the generation step and not IO bound by writing to the database. Hence, it's probably ok to do non-incremental.

(Online updates to the exact match coming from Train Almond could be supported with a separate exact match trie in memory, that would periodically be flushed to disk)


UnityXR SDK Support

This pull request will add UnityXR support, which is much easier to set up than any of the other SDKs and is a little more flexible.
Usage guide is already posted on the wiki, but a quick note: this is only being actively support for 2019.1 and newer. I included the existing implementation for 2018 from VRTK (and verified that it works), but I will not be maintaining it.
When complete this will resolve: #3

One additional thing, and this is for all future PRs and such, but the build list of scenes should be cleared if we're not including the scenes. Otherwise, Unity only specifies that they are deleted but it does not let you select and remove them.


Set-CIPolicySetting is missing

Set-CIPolicySetting is missing from the list of commands. It is used to add class ids to the policy xml file.
Example available at this link : https://docs.microsoft.com/en-us/windows/security/threat-protection/windows-defender-application-control/allow-com-object-registration-in-windows-defender-application-control-policy#how-to-configure-settings-for-the-clsids. P.S. I am not able to create a pull request that is why I have created a new issue. If anyone is able to do it. please go ahead.

Is there any update on this? @e0i

@TJ2215 This is under investigation. Thank you for the feedback.

Hello @TJ2215,
yes you are correct, Set-CIPolicySetting article is missing in the list of commands of ConfigCI module and the article is not presented on GitHub.
I only see the description of cmdlet on the powershell help:
Word | Binary | String} -Value   []
Set-CIPolicySetting [-FilePath] <string> -Provider <string> -Key <string> -ValueName <string> -Delete
[<CommonParameters>]

PARAMETERS
-Delete
-FilePath <string>

-Key <string>

-Provider <string>

-Value <string>

-ValueName <string>

-ValueType <string>

<CommonParameters>
    This cmdlet supports the common parameters: Verbose, Debug,
    ErrorAction, ErrorVariable, WarningAction, WarningVariable,
    OutBuffer, PipelineVariable, and OutVariable. For more information, see
    about_CommonParameters (https:/go.microsoft.com/fwlink/?LinkID=113216).

ALIASES
None


Fix pipon definition merge

This PR fixes #412, where both the verb and the noun pipon were wrongly associated with same 3 definitions.
Now it's correct:

Potentially this also fixes some other cases.
I'll update production database after this is merged to master.

@eddieantonio pretty technical changes. I do have comments if you want to scrutinize.

Yes, it now works as it should for kîsikâw as well, see:

This also seems to mostly be the case for also kimiwan,

... though there I think the major cause of the remaining mix-up is in the XML source.
N.B. This should NOT be fixed programmatically, but in the source (XML or otherwise), by linguists. The reason for this is probably that the comparison script I used didn't properly disambiguate word-class-wise ambiguous dictionary entries.

Generally, there is a small but non-insignificant number of dictionary entries which are ambiguous for the head word (e.g. II verbs and nouns, some II and TA verbs). Because of this, any combination of the dictionary entries may need to be sensitive as deep down as the inflectional subcategory (sometimes the general word class is sufficient, but not always). In the XML, the inflectional subcategory is provided in the lc variable. Something similar(ly named) will be used in whatever format the new, aggregated crk dictionary source will look like.

@aarppe get it


Download progress and create folder

Hi,
I want to know how to get the progress of download and create an empty folder.
It's possible?
Thanks in advance!

Hi Andrea,
For creating an empty folder, you can follow the instruction at the last part in this [doc] (https://github.com/OneDrive/onedrive-sdk-android/blob/master/docs/collections.md#create-a-folder) about creating an empty folder.
For showing download progress, there are two ways but  I am not sure which one could meet your needs:
1. In the example app we use Android's Download Manager to handle download progress, here is the example.
2. you can also track the progress by yourself as you get the inputstream. Here is the example shows how to get inputstream from SDK.
Dabo

Sounds like this should be resolved, closing out this old issue.


[FLINK-4379]  Rescalable non-partitioned state

This pull request introduces rescalable non-partitioned operator state as described in issue [FLINK-4379] and makes following major changes:
1) Introducing interfaces PartitionableStateBackend and SnapshotProvider
For rescaling purposes, non-partitioned state is now stored in a PartitionableStateBackend, which provides a method to register operator states under unique names and store state partitions in a list state:
<S> ListState<S> getPartitionableState(
	String name,
	Collection<PartitionableOperatorStateHandle> restoreSnapshots,
	TypeSerializer<S> partitionStateSerializer) throws Exception;

Notice that a TypeSerializer is provided on a per-state basis so that we can realize backward compatibility if the serialization format changes.
Furthermore, we introduce SnapshotProvider which offers a method to create snapshots:
RunnableFuture<S> snapshot(
	long checkpointId,
	long timestamp,
	CheckpointStreamFactory streamFactory) throws Exception;

DefaultPartitionableStateBackend is an implementation of both, PartitionableStateBackend and SnapshotProvider, where the first interface gives a view that is exposed to user code and the later interface is only used by the system to trigger snapshots.
Similar to other state backends (e.g. KeyedStateBackend), a DefaultPartitionableStateBackend can be created and restored through the class AbstractStateBackend:
public DefaultPartitionableStateBackend createPartitionableStateBackend(
	Environment env,
	String operatorIdentifier)

public DefaultPartitionableStateBackend restorePartitionableStateBackend(
	Environment env,
	String operatorIdentifier,
	Collection<PartitionableOperatorStateHandle> restoreSnapshots)

2) Interface PartitionableCheckpointed as replacement for Checkpointed
User code can use rescalable non-partitioned state by implementing the PartitionableCheckpointed interface. This interface is meant to replace the Checkpointed interface, which would become deprecated.
void storeOperatorState(long checkpointId, PartitionableStateBackend backend) throws Exception;

void restoreOperatorState(PartitionableStateBackend backend) throws Exception;

Through the store/restore methods, user code can interact with PartitionableStateBackend and register/restore states as previously explained.
Methods from this interface are invoked in StreamTask::performCheckpoint(...) and StreamTask::restoreState() respectively.
3) Interface StateRepartitioner
This interface allows implementations of repartitioning strategies the redistribute non-partitioned state when for changing parallelism. Currently, there is only one default strategy implemented in RoundRobinStatePartitioner, but in general it is possible to implement different strategies, e.g. a StatePartitioner that distributes the union of all previous states parallel subtasks to each parallel subtask. This interface is used in CheckpointCoordinator::restoreLatestCheckpointedState().
4) Implementation for KafkaConsumer/Producer
The PR contains implementations of rescalable non-partitioned state for Kafka consumer and producers.
TODO: After review, a description of the new features must still be added to the Flink documentation.

Please review @tillrohrmann @StephanEwen

Hi,
I have some suggestions for renaming some of the interfaces and their methods in this pull request to come up with some clearer, more consistent naming schemes. I suggest the following changes:
1) Renaming the state handle that points to operator state: PartitionableStateHandle -> OperatorStateHandle
2) Rename: PartitionableStateBackend -> OperatorStateStore
	/**
	 * User-side interface for storing (partitionable) operator state.
	 */
	public interface OperatorStateStore {

		/**
		 * Creates (or restores) the partitionable state in this backend. Each state is registered under a unique name.
		 * The provided serializer is used to de/serialize the state in case of checkpointing (snapshot/restore).
		 */
		<S> ListState<S> getListState(String name, TypeSerializer<S> partitionStateSerializer) throws Exception;
	}

3) Rename: PartitionableSnapshotStateBackend -> OperatorStateBackend. I propose that the term backend now refers to some (i) store with the ability to (ii) snapshot.
	/**
	 * Interface that combines both, the user facing {@link OperatorStateStore} interface and the system interface
	 * {@link SnapshotProvider}
	 */
	public interface OperatorStateBackend
			extends OperatorStateStore, SnapshotProvider<PartitionableOperatorStateHandle> {
	}

4) Rename: PartitionableCheckpointed -> CheckpointedOperator
- `storeOperatorState` -> `snapshotState`
- `restoreOperatorState` -> `restoreState`

	public interface CheckpointedOperator {

		/**
		 * This method is called when state should be stored for a checkpoint. The state can be registered and written to
		 * the provided state store.
		 */
		void snapshotState(long checkpointId, OperatorStateStore stateStore) throws Exception;

		/**
		 * This method is called when state should be restored from a checkpoint. The state can be obtained from the
		 * provided state store.
		 */
		void restoreState(OperatorStateStore stateStore) throws Exception;
	}

5) Rename: StateRepartitioner -> OperatorStateRepartitioner
	/**
	 * Interface that allows to implement different strategies for repartitioning of operator state as parallelism changes.
	 */
	public interface OperatorStateRepartitioner {

		List<Collection<OperatorStateHandle>> repartitionOperatorState(
				List<OperatorStateHandle> previousParallelSubtaskStates,
				int parallelism);
	}

6) Add new interface that allows user-friendly checkpointing code for simple cases that to not require custorm serializer
	/**
	 * Simplified interface as adapter to the more complex CheckpointedOperator
	 */
	public interface ListCheckpointed<T extends Serializable> {

		List<T> snapshotState(long checkpointId) throws Exception;

		void restoreState(List<T> state) throws Exception;
	}

7) OperatorStateBackend lifecycle
Another point that we might want to discuss is the life cycle of OperatorStateBackend. Currently, a new backend is created (+restored) for each invocation of the methods in CheckpointedOperator. This always provides a clean backend to take the operator state for a snapshot. I wonder if it could make sense to create OperatorStateBackend just once for each AbstractStreamOperator, similar to the KeyedStateBackend. This would give users the option to actually keep operator state only in the OperatorStateBackend. However, we need a way to signal that all state must be passed to the backend before a snapshot. For example, large operator states could be managed in RocksDB this way, and we could provide more proxy collections (currently we only support a list of substates) over time.
What do you think @aljoscha @StephanEwen ?

I think most of the proposed renames are good. One thing that would be good to change is to remove SnapshotProvider and just have that method directly on OperatorStateBackend.
I really like the separation between the user facing state store and the backend. I've been meaning to change that for keyed state as well, i.e. the user only gets a way to access state and not all the methods for snapshotting, closing, etc. from the backend.
When I say I like the changes, I mean everything except that they have Operator in all of them. In the end, all state is at an operator, plus there is the existing interface OperatorState that is an alias for ValueState. I don't have a good alternative now, but I think the Operator could be confusing.

+1, I like the changes.
Concerning (7) - I think creating the state backend once sounds good. It is more in line with the keyed state backend.
Concerning Aljoscha's comments: Removing SnapshotProvider makes sense, if it is only an internal interface and never exposed to the user.
Conderning the name OperatorState - I actually like the name. It is state that exists in the instance of an operator, assuming that functions are also (special cases of) operators.
I would add to this pull request that it removes the deprecated OperatorState interface (superclass of ValueState).

I think we should also update the documentation as part of this PR (there's a page dedicated to working with state)

I would like to go for a follow-up PR. This is currently blocking some other work, so getting it in soon would help.

Had an in-depth code walkthrough with @StefanRRichter - comments are addresses as additional commits.
Comments:

Use StateDescriptor as well for operator state
For the snapshotState() and prepareSnapshot() calls, the checkpoint timestamp should be passed as well.
The Kafka Consumer should use the same state as the snapshotState() interface, so as to allow for backwards compatible checkpoints/savepoints.


Append enter after snippet insertion

In some cases it could be useful to append enter after snippet.
Eg. pasting snippet into command prompt (without need modify it) could immediately run it.

Using Ctrl+Enter is error prone as apps might be bindings for pressing Ctrl+Enter

Record that enter was hit and wait for ctrl to release before firing the command


Setting slider to current image

Hi Thomas,
thanks a lot four your code! I have managed to incorporate your slider back into my theme :-)
Now I have run into the following: Seems like you try to set the slider to the image with this
$(".slick-slider").slick('goTo', currentThumbnailIndex, true);

On my site that makes the thumb to be the first in the slider, see http://www.feuster.com/piwigo/picture?/8970/category/714-lissabon_2016. But I would like to have it centered. How could I do that?
I have tried the centerMode option but that doesn't seem to work with slidesToScroll. Any other ideas?
Thanks in advance,
Thomas

Hi Thomas,
"centerMode: true" forces slidesToScroll to 1, no matter what you set it to in the options. See https://github.com/kenwheeler/slick/pull/659.
What I like most currently is "centerMode: true" together with "swipeToSlide: true", that way the buttons scroll only by 1, BUT you can use the mouse to drag along more than 1 slide.

On my Windows-Box this leads to a behavior that at the end of a mouse-drag the slider moves back to center the left-most (when dragging to the left) thumb. When dragging to the right a few more thumbs are scrolled to the right afterwards. Not necessarily what I would want :-(

I'll close this for now as it's not really theme related.


pivot: resize column ux

Updates the UX for resizing columns in the pivot table by extending the handle vertically across the entire table.

@jkhwu

Works well!
Just a minor nit, it seems to not be horizontally aligned:

The resizing is a bit buggy for me. After expanding or contracting the column width, it takes the default width and is not respecting the user defined width
https://github.com/user-attachments/assets/320abfc8-f442-4c2f-bb7c-e3a3df93a45a

This was being caused by the behavior and placement of the removeActiveCell function.

I've moved this active cell cancellation behavior to the PivotToolbar component, which is the only place it can practically be. Its use with the clickOutside action meant that it was being triggered on mouse up when dragging. Click outside does not really capture the intended cancellation behavior as clicking things like the panel toggle, export button or search bar would clear the active cell. It's now only cancelled when clicking the empty space above the pivot table (between the panel toggle and export buttons).
Toggling the active cell causes the entire config object to be re-derived, which in turn causes the table to re-render and the column widths to be reset. I've changed the implementation of the measureLengths so that this is avoided, but I think we should isolate the activeCell state so that this isn't happening in the background.


Fixed travis soft fail versions

Oh. Looks like it did not work. I'll have to look into this.

It took a couple of tries but now the build seems to allow failures from Ruby 2.2 on all Redmine versions.

Thanks for your work!


Fix compilation issue for arm macOS native compilation

Compiling CppMicroServices natively on an M1 mac fails with the following warning-as-error message:
In file included from /Users/alexander/Development/CppMicroServices/compendium/LogServiceImpl/src/LogServiceImpl.cpp:3:
In file included from /Users/alexander/Development/CppMicroServices/third_party/spdlog/include/spdlog/sinks/stdout_color_sinks.h:9:
In file included from /Users/alexander/Development/CppMicroServices/third_party/spdlog/include/spdlog/sinks/ansicolor_sink.h:8:
In file included from /Users/alexander/Development/CppMicroServices/third_party/spdlog/include/spdlog/sinks/sink.h:6:
/Users/alexander/Development/CppMicroServices/third_party/spdlog/include/spdlog/details/log_msg.h:17:5: error: definition of implicit copy assignment operator for 'log_msg' is deprecated because it has a user-declared copy constructor [-Werror,-Wdeprecated-copy]
    log_msg(const log_msg &other) = default;
    ^
/Users/alexander/Development/CppMicroServices/third_party/spdlog/include/spdlog/details/log_msg_buffer-inl.h:36:14: note: in implicit copy assignment operator for 'spdlog::details::log_msg' first required here
    log_msg::operator=(other);
             ^
In file included from /Users/alexander/Development/CppMicroServices/compendium/LogServiceImpl/src/LogServiceImpl.cpp:3:
In file included from /Users/alexander/Development/CppMicroServices/third_party/spdlog/include/spdlog/sinks/stdout_color_sinks.h:9:
In file included from /Users/alexander/Development/CppMicroServices/third_party/spdlog/include/spdlog/sinks/ansicolor_sink.h:8:
In file included from /Users/alexander/Development/CppMicroServices/third_party/spdlog/include/spdlog/sinks/sink.h:6:
/Users/alexander/Development/CppMicroServices/third_party/spdlog/include/spdlog/details/log_msg.h:17:5: error: definition of implicit copy assignment operator for 'log_msg' is deprecated because it has a user-declared copy constructor [-Werror,-Wdeprecated-copy]
    log_msg(const log_msg &other) = default;
    ^
/Users/alexander/Development/CppMicroServices/third_party/spdlog/include/spdlog/details/log_msg_buffer-inl.h:36:14: note: in implicit copy assignment operator for 'spdlog::details::log_msg' first required here
    log_msg::operator=(other);
In file included from /Users/alexander/Development/CppMicroServices/compendium/LogServiceImpl/test/TestLogService.cpp:21:
In file included from /Users/alexander/Development/CppMicroServices/third_party/spdlog/include/spdlog/sinks/ostream_sink.h:7:
In file included from /Users/alexander/Development/CppMicroServices/third_party/spdlog/include/spdlog/sinks/base_sink.h:13:
/Users/alexander/Development/CppMicroServices/third_party/spdlog/include/spdlog/details/log_msg.h:17:5: error: definition of implicit copy assignment operator for 'log_msg' is deprecated because it has a user-declared copy constructor [-Werror,-Wdeprecated-copy]
    log_msg(const log_msg &other) = default;
    ^
/Users/alexander/Development/CppMicroServices/third_party/spdlog/include/spdlog/details/log_msg_buffer-inl.h:36:14: note: in implicit copy assignment operator for 'spdlog::details::log_msg' first required here
    log_msg::operator=(other);

This PR fixes this issue by removing the explicitly defined, defaulted copy constructor for spdlog::details::log_msg.
Signed-off-by: The MathWorks, Inc<EMAIL_ADDRESS>

Codecov Report

Merging #696 (132d56a) into development (954936f) will decrease coverage by 0.18%.
The diff coverage is n/a.

@@               Coverage Diff               @@
##           development     #696      +/-   ##
===============================================
- Coverage        90.46%   90.27%   -0.19%
===============================================
  Files              248      230      -18
  Lines            13643    12550    -1093
===============================================
- Hits             12342    11330    -1012
+ Misses            1301     1220      -81

Impacted Files
Coverage Δ

...endium/ServiceComponent/src/ComponentException.cpp
0.00% <0.00%> (-100.00%)
:arrow_down:

...m/ServiceComponent/src/ServiceComponentRuntime.cpp
0.00% <0.00%> (-100.00%)
:arrow_down:

...m/DeclarativeServices/src/manager/BundleLoader.cpp
63.82% <0.00%> (-6.32%)
:arrow_down:

...mework/src/service/ServiceReferenceBasePrivate.cpp
86.16% <0.00%> (-5.93%)
:arrow_down:

...pendium/tools/SCRCodeGen/ManifestParserFactory.hpp
83.33% <0.00%> (-5.56%)
:arrow_down:

framework/src/service/ServiceListeners.cpp
95.10% <0.00%> (-4.90%)
:arrow_down:

...iveServices/src/metadata/MetadataParserFactory.hpp
66.66% <0.00%> (-3.34%)
:arrow_down:

...ndium/ConfigurationAdmin/src/CMBundleExtension.cpp
85.71% <0.00%> (-3.18%)
:arrow_down:

framework/src/service/ServiceRegistry.cpp
90.76% <0.00%> (-2.57%)
:arrow_down:

.../ConfigurationAdmin/src/ConfigurationAdminImpl.cpp
90.98% <0.00%> (-2.04%)
:arrow_down:

... and 87 more


Implemented Viper, .env for better contributor experience

Are you sure that this is the right approach? I guess we should discuss how we want to host the backend in the end. You can also set the ENV in your run configuration in Goland / IntelliJ

Not everyone uses InteliJ/Goland, for the sake of anyone wanting to contribute we should have a file for .env like most projects you can find on GitHub. Unless we set up docker and stuff we can remove it but for the time being it'll be good


Update README.md

Change read-me

Build fails because there are no tests to run and fail/pass yet - so this will need to be ignored until we start working in this repo


Add link to the »Awesome JavaScript Learning« list

Hej I guess that list will fit quite well in here:

https://github.com/micromata/awesome-javascript-learning

It’s featured in the learn section of Sindres meta master list.
Cheers, Michael

Looks awesome! Thanks for the PR @mischah - merging now!

🍾


⚠ ClamAV has degraded performance

In d6f5576, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 88 ms

Resolved: ClamAV performance has improved in 9da32ce.


Esphome sonoff rf bridge empty binary state after restart

Home Assistant release with the issue:

0.101
Last working Home Assistant release (if known):
Operating environment (Hass.io/Docker/Windows/etc.):

Docker
Integration:

Esp home
Description of problem:
Esp home sonoff rf bridge binary states are null after restart in previous version this was not affected. This happened after update.
Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):

Traceback (if applicable):

Additional information:

Otto, thanks for patching this but the state active/inactive problem still persists. Now the states are showing after reboot but they are not active HA does not know how they are and wont trigger automations after restart you need to trigger them for 2-3 times to HA detect them correctly.

system is up to date to: 0.101.1.

The fix (and original issue) was in the esphome project. It only started appearing with an Home Assistant update because of some internal changes.
If you update your esphome install to the latest version (1.14.0), the problem should be gone.

Hi Otto i have done the update but no nothing changed the issue persists the same after restart Binary_sensors are all unavailable. I have to trigger them manually to appear in badges.

Please confirm you did the ESPHome update (not HA update).
Please also add more info - is it really showing unavailable or just unknown (screenshot is best). By restart do you mean ESP restart?

Hi Otto sorry for late response ive done the ESPHome update and also reflashed the Sonoff RF to the latest version but no avails nodes on RF after restart are showing as N/A when you trigger them manually they set up and appear normal. ESP updated to 1.14.2

Please also add more info - is it really showing unavailable or just unknown (screenshot is best). By restart do you mean ESP restart?
Both. Its the same effect if you restart the ESPHome they dissapear or if i reflash sonoff RF they would also dissapear after the HA restart they would dissapear also. Here is the screen the binaries that are showing ok are Xiaomi/Aqara door window sensors.

Hi Otto update some of the sensors re appear by them selves there is no rule/pattern how this happen  or at which sometimes water leakage come on sometimes door/window come back.

I have these issues on production and a vanilla installation after update.

Cruicial update to situation found a fix for my problem.
Had to remove esphome and reinstall it something must have gone wrong after update. After reinstall all nodes set correctly and everything is back to normal. Except there is issue in ESPHOME panel now some nodes show red dot unavaliable but i can reflash them and they work in HA frontend.

Everything is fine now. It was a bad container after update it done a real mess. Now i have reinstalled and reflashed devices and it works all ok.

Wrote too soon something weird is going on connection is being restarted on all devices and the states get mesed up the devices change to offline and online randomly. I hoped it would all change after update but it did to worse.


use directory-like syntax for imports

I think using /, ., and .. for imports would be quite nice
for example
// import math file from current directory
import ./math
from ./math import add

// import math from parent directory
import ../math
from ../math import add

// libraries use their name instead of .
import std/math
from std/math import add

this makes it very clear what you intend to import and doesn't conflate it with the dot access operand .

should projects be able to import from themselves?
if so using self to refer to the base could be cool

also this brings up the idea of reserving library names like std and self; for std it may be nice to have a way to override it


fix _send invocation in post method

the _send method takes three arguments, where the second is params, and the third the cb.
this was wrong in the post method, causing post to send the contents of the callback function to the nntp server.

Landed in 01beea8ace. Thanks!


experiment/phasemix + CNN and CNN_LSTM

We should add comparise experiment for phasemix method.
Will add:

phasemix + CNN
phasemix + CNN_LSTM

I found that can directly set train.backbone=2dcnn/cnn_lstm and train.temporal_mix=true to implement the phasemix with cnn/cnn_lstm method.

finish the comparise experiment, close.


Dependency update

Fixes #
Proposed Changes

Codecov Report

Merging #99 (1d607d2) into main (6048372) will decrease coverage by 0.07%.
The diff coverage is 100.00%.

Additional details and impacted files

Impacted Files
Coverage Δ

narrow_down/_minhash.py
100.00% <100.00%> (ø)

narrow_down/scylladb.py
98.27% <100.00%> (ø)

narrow_down/similarity_store.py
96.63% <100.00%> (ø)

narrow_down/sqlite.py
97.29% <100.00%> (ø)

narrow_down/storage.py
100.00% <100.00%> (ø)

narrow_down/__init__.py
100.00% <0.00%> (ø)


Refactor conversion lists into Rust

Refactor conversion lists into Rust

Coverage remained the same at 94.946% when pulling 005ec51e2b30a10e53308a6cc685c5b492560044 on list-to-rust-refactoring into 83c71fbc8b285be8b9e661f2bdc3d6fab53d50c7 on master.

Coverage remained the same at 94.946% when pulling 005ec51e2b30a10e53308a6cc685c5b492560044 on list-to-rust-refactoring into 83c71fbc8b285be8b9e661f2bdc3d6fab53d50c7 on master.

Coverage remained the same at 94.946% when pulling 005ec51e2b30a10e53308a6cc685c5b492560044 on list-to-rust-refactoring into 83c71fbc8b285be8b9e661f2bdc3d6fab53d50c7 on master.

Coverage remained the same at 94.946% when pulling 005ec51e2b30a10e53308a6cc685c5b492560044 on list-to-rust-refactoring into 83c71fbc8b285be8b9e661f2bdc3d6fab53d50c7 on master.

Codecov Report

Merging #114 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #114   +/-   ##
=======================================
  Coverage   95.56%   95.56%
=======================================
  Files          18       18
  Lines        1870     1870
=======================================
  Hits         1787     1787
  Misses         83       83

Impacted Files
Coverage Δ

src/types/data_serialization_types.rs
28.57% <ø> (ø)
:white_check_mark:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 83c71fb...5407d4f. Read the comment docs.

Coverage remained the same at 94.946% when pulling 005ec51e2b30a10e53308a6cc685c5b492560044 on list-to-rust-refactoring into 83c71fbc8b285be8b9e661f2bdc3d6fab53d50c7 on master.

Coverage remained the same at 94.946% when pulling 005ec51e2b30a10e53308a6cc685c5b492560044 on list-to-rust-refactoring into 83c71fbc8b285be8b9e661f2bdc3d6fab53d50c7 on master.

Coverage remained the same at 94.946% when pulling 005ec51e2b30a10e53308a6cc685c5b492560044 on list-to-rust-refactoring into 83c71fbc8b285be8b9e661f2bdc3d6fab53d50c7 on master.

Coverage remained the same at 94.946% when pulling 005ec51e2b30a10e53308a6cc685c5b492560044 on list-to-rust-refactoring into 83c71fbc8b285be8b9e661f2bdc3d6fab53d50c7 on master.

Coverage remained the same at 94.946% when pulling 005ec51e2b30a10e53308a6cc685c5b492560044 on list-to-rust-refactoring into 83c71fbc8b285be8b9e661f2bdc3d6fab53d50c7 on master.

Coverage remained the same at 94.946% when pulling 5407d4f744484e5e78ed3d1d9bbd1b0df8ce6027 on list-to-rust-refactoring into 83c71fbc8b285be8b9e661f2bdc3d6fab53d50c7 on master.


Gradle spends lots of time in Waiting\Aquiring\Releasing lock on daemon addresses registry.

During gradle build with cmake on jenkins there are some heavy freezes from 20secs to 1.5min with following messages in loop:
[2021-07-02T10:28:12.542Z] 2021-07-02T13:28:11.775+0300 [LIFECYCLE] [org.gradle.cache.internal.DefaultFileLockManager]
[2021-07-02T10:28:12.542Z] 2021-07-02T13:28:11.775+0300 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.
[2021-07-02T10:28:12.542Z] 2021-07-02T13:28:11.775+0300 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry.
[2021-07-02T10:28:12.542Z] 2021-07-02T13:28:11.776+0300 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
[2021-07-02T10:28:12.542Z] 2021-07-02T13:28:11.776+0300 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.
[2021-07-02T10:28:12.542Z] 2021-07-02T13:28:11.776+0300 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry.
[2021-07-02T10:28:12.542Z] 2021-07-02T13:28:11.776+0300 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
//some more iterations omitted
[2021-07-02T10:28:44.305Z] 2021-07-02T13:28:08.657+0300 [LIFECYCLE] [class org.gradle.internal.buildevents.TaskExecutionLogger] > Task :generateJsonModelMultRelease
Problem occurrs only on our build server, that is running in docker. On local machine this locking also occurs on the same places of build script, but takes 1-2sec.
It seems that most heavy lags occurs on launching subprocesses - cmake, python scripts called from cmake, linker.
I was unable to create minimal bulding example for this bug.
Problem was partially fixed with adding gradle property -XX:MaxHeapSize=2G, with increased heap size there were no loops during build, but still there was 1.5 min freeze at the end of build.
Environment:
Gradle 6.8.3 (6.7.1 also have such behavior)
CentOS Linux release 7.5.1804 (Core) running in docker
NDK 21.1.6352462
openjdk version "1.8.0_212-release"
OpenJDK Runtime Environment (build 1.8.0_212-release-1586-b4-5784211)
OpenJDK 64-Bit Server VM (build 25.212-b4-5784211, mixed mode)

The fact that increased heap size helped a bit suggests that the issue was (at least partially) in the insufficient memory. You can verify that by running jstat -gc <pid of the daemon process> 5s that will dump GC statistics of the Gradle daemon every 5 seconds. Increasing values of FGC/FGCT (full GC count/full GC time) columns indicate a lack of available memory.
As for the end-build hanging - it would be great if you could provide a profiler report (with Java Flight Recorder/async profiler/YourKit) of the daemon process during the hang so we can figure out what is going on. A simpler substitution for the provider might be just capturing a dozen of thread dumps with jstack -l -e <pid of the daemon process>. You can share profile data or dumps with me privately at mlopatkin[at]gradle.com.

We cannot really proceed there without additional information. I'm closing the issue as non-actionable. Feel free to comment there with the information requested if you thing it should be reopened.


fix selecting non-existing user in offline mode

Explanation of Change
Fixed Issues
$ https://github.com/Expensify/App/issues/51651
PROPOSAL: https://github.com/Expensify/App/issues/51651#issuecomment-2448833349
Tests

Open Expensify app
Go offline
Click on the search icon at the top right
Search for a non-existing user
Click on the search result
Go online
Make sure the not here page should not appear

[X] Verify that no errors appear in the JS console

Offline tests
Same as tests
QA Steps
Same as tests

[X] Verify that no errors appear in the JS console

PR Author Checklist

[X] I linked the correct issue in the ### Fixed Issues section above
[X] I wrote clear testing steps that cover the changes made in this PR

[X] I added steps for local testing in the Tests section
[X] I added steps for the expected offline behavior in the Offline steps section
[X] I added steps for Staging and/or Production testing in the QA steps section
[X] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[X] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[X] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[X] I included screenshots or videos for tests on all platforms
[X] I ran the tests on all platforms & verified they passed on:

[X] Android: Native
[X] Android: mWeb Chrome
[X] iOS: Native
[X] iOS: mWeb Safari
[X] MacOS: Chrome / Safari
[X] MacOS: Desktop

[X] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[X] I followed proper code patterns (see Reviewing the code)

[X] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[X] I verified that comments were added to code that is not self explanatory
[X] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[X] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method

[X] If any non-english text was added/modified, I verified the translation was requested/reviewed in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message:

[X] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[X] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
[X] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[X] I verified the JSDocs style guidelines (in STYLE.md) were followed

[X] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[X] I followed the guidelines as stated in the Review Guidelines
[X] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[X] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[X] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[X] I verified that if a function's arguments changed that all usages have also been updated correctly
[X] If any new file was added I verified that:

[X] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[X] If a new CSS style is added I verified that:

[X] A similar style doesn't already exist
[X] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(theme.componentBG))

[X] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[X] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[X] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[X] If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
[X] If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:

[X] I verified that all the inputs inside a form are aligned with each other.
[X] I added Design label and/or tagged @Expensify/design so the design team can review the changes.

[X] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[X] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.

Screenshots/Videos

Android: Native
https://github.com/user-attachments/assets/cac6803d-7c2d-4756-af94-1b145af53918

Android: mWeb Chrome
https://github.com/user-attachments/assets/d6a03ffb-97bd-44db-9fb6-2521e6c28f4d

iOS: Native
https://github.com/user-attachments/assets/bda901af-35ef-4551-b1fe-56685279b56f

iOS: mWeb Safari
https://github.com/user-attachments/assets/2243fa0a-27ad-486b-a179-2818d08ae850

MacOS: Chrome / Safari
https://github.com/user-attachments/assets/3e7a8c74-8785-43bf-90c7-b6489017a359

MacOS: Desktop
https://github.com/user-attachments/assets/5385bab6-1c58-4462-8479-a14c6ab7c90c

:hand: This PR was not deployed to staging yet because QA is ongoing. It will be automatically deployed to staging after the next production release.


Rancher App List not working on RKE2 clusters

the installed apps section of the local cluster is only showing me the flask demo app. longhorn, neuvector, and monitoring are not showing up in that menu despite being installed via the rancher app chart store.
the imported eks-cluster is showing the cattle-system and fleet-agent apps after import just fine. the eks-cluster also shows longhorn and the longhorn-crds when i install it (with prereqs of course) in the eks-cluster. seems like just the rke2 cluster is not showing installed apps properly.

it should be showing. did you try force reloading or logging out and back into it?

rookie mistake- did not change the namespace filter. 🤦
user error, closing with no changes.

It does bring up a good point... should we call out the namespace filter step a bit more? it happens quite often to users.


Adding Child Supervision feature

Child supervision feature provides a mechanism for parent to ensure that a message is sent to each child within a fixed interval, namely the supervision interval. If there is no transmission to the child within the supervision interval, child supervisor enqueues and sends a supervision message (a data message with empty payload) to the child.
On the child side, this is used to check the connectivity to the parent. If the child does not hear from its parent for a supervision check timeout interval it assumes that it may be disconnected and tries to re-attach to the parent.
The child supervision provides an alternative, more energy-efficient solution compared to requiring the child to periodically perform an MLE Child Update Request/Response exchange with the parent (as a way of verifying that it is still connected to the parent). The child supervision solution puts the burden of message transmissions on the parent instead of the typically more energy-constrained child.

Codecov Report

Merging #1608 into master will increase coverage by 2.95%.
The diff coverage is 34.61%.

@@            Coverage Diff             @@
##           master    #1608      +/-   ##
==========================================
+ Coverage   67.56%   70.52%   +2.95%
==========================================
  Files         166      160       -6
  Lines       21102    19369    -1733
  Branches     2588     2400     -188
==========================================
- Hits        14258    13660     -598
+ Misses       5936     4836    -1100
+ Partials      908      873      -35

Impacted Files
Coverage Δ

src/core/common/message.hpp
96.29% <ø> (ø)
:arrow_up:

src/core/thread/mle_router_ftd.hpp
71.42% <ø> (ø)
:arrow_up:

src/core/thread/topology.hpp
89.02% <ø> (ø)
:arrow_up:

src/core/thread/mesh_forwarder.hpp
66.66% <ø> (ø)
:arrow_up:

src/core/thread/mle.cpp
81.77% <100%> (-0.22%)
:arrow_down:

src/core/thread/thread_netif.cpp
82.92% <100%> (+0.87%)
:arrow_up:

src/core/thread/thread_netif.hpp
96.55% <100%> (+3.69%)
:arrow_up:

src/core/thread/mesh_forwarder.cpp
76.68% <17.5%> (-1.53%)
:arrow_down:

src/core/utils/child_supervision.hpp
85.71% <85.71%> (ø)

src/core/common/logging.cpp
9.55% <0%> (-28.43%)
:arrow_down:

... and 65 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bd18c0f...434badb. Read the comment docs.

Rebased on master to resolve merge conflicts.

@abtink ,

Sorry for careless operations perviously.
@abtink , may I ask your plan for this PR? are you going to limit the supervision model to SED? Thanks.

@librasungirl , the supervision model is between a parent node and its children (sleepy or non-sleepy).

Thread 1.1 Section 4.7.4 states:

A Child specifies its timeout period in the Timeout TLV included in the Child ID Request sent to the Parent. Parents maintain a separate timeout timer for each Child. This timer is reset when a keep-alive message is received from the Child.

[...]

The keep-alive message for an rx-on-when-idle Child is a Child Update Request containing the following TLVs:

Mode TLV
Source Address TLV
Leader Data TLV
Address Registration TLVs (optional)

I do not think we should be suppressing Child Update Request messages from rx-on-when-idle children in this case.

Rebased on master to resolve merge conflicts...

Pushed a changed commit. This now limits the supervision process (on both parent and child) to sleepy children only.

Not sure what's happening with either travis or appvoyer builds (they both seem stuck...)

You are queued behind other OpenThread jobs.  Gotta wait your turn :)


WinRT projections should autorelease all objects returned from create* factory functions

For example; this code leaks because the WFPoint isn't added to the autoreleasepool:
WFPoint* pointInFromLayer = [WXPointHelper fromCoordinates:point.x y:point.y];
It looks like this only pertains to structs WFPoint, WFRect, etc. but we should scrub the rest of the projections to confirm.  Anything that goes through _createRTProxy is fine because it returns an autoreleased object.

Fixed.


Fast flow control

Coverage decreased (-0.02%) to 93.152% when pulling 3afe98e6eb8759f5ff609e4809b35be8eb4b4da5 on fast_flow_control into fd7d3efc18e9ea36539b4c806968479143694f13 on dev.


Create acceptance test helper to fill in a content editable component

Fixes issue #24 with a cool reusable acceptance test helper and acceptance test for the dummy app demonstrating how to use it.

Update: Included @ConorLinehan 's fix and fixed tests for Ember 2.7

ember try:testall passes for me locally. Not sure why it does not pass on Travis.

Yay! Tests Passed!

Great, thanks for this! Looks awesome. I'll get it released as 0.10.0


Support for downloading folders

Would it be possible to implement the ability to download shared folders? Drive usually tries to zip folders before you download them but I'd imagine it would be possible to download each item separately into a folder of the same name.

I'm not sure how to get the list of files in the shared folder without using API.
Maybe you can use https://github.com/wkentaro/gshell.
pip install gshell
gshell init
gshell download --with-id <FILE_ID> --recursive

gshell is not mantained anymore

https://medium.com/@olegkhomenko/how-to-download-multiple-files-from-google-drive-using-terminal-7f0f2ee357b8
In short: $$("[data-id]").map((el) => 'https://drive.google.com/uc?id=' + el.getAttribute('data-id')).join(" ") in console

@Kreijstal,  as said by @AlexRMU that is shown in the article here, you can use the following js code once you are in the desired gdrive folder to get those pesky urls, however it seems that some urls are not working (maybe it's from old files that were in the gdrive folder):
$$("[data-id]").map((el) => 'https://drive.google.com/uc?id=' + el.getAttribute('data-id'))

i will try to use selenium with chrome headless to automate it and post here.

Got it without selenium, only requests,bs4 and regex it's pretty hacky tough don't know if it would work in any gdrive folder link.
Could this be adapted so it can be added to this package @wkentaro ?

If the folder parsing requires Javascript, I don't think it will work without selenium.
I'm not sure if we should add that feature with adding selenium to this project.

If the folder parsing requires Javascript, I don't think it will work without selenium.
I'm not sure if we should add that feature with adding selenium to this project.

Yes, i think it's a bad idea to add selenium dependency in this package, but i am referring to this implementation, with bs4+regex

If the folder parsing requires Javascript, I don't think it will work without selenium.
I'm not sure if we should add that feature with adding selenium to this project.

Yes, i fully agree that would be bad to add selenium dependency, however these links worked in the regex/bs4 version ...

@giuliano-oliveira Yes, sometimes there are links to non-existent files, but you can use them freely because they just won't download!