ExcludeFromOutput only works for one property in class

I.e. applying [SpreadsheetTabName(ExcludeFromOutput = true)] to more than one property in the same class, only one property will be excluded from the spreadsheet.
Based on other experiments, I understand now that  SpreadsheetTabName is only meant to be used once per class; but it's the only option that excludes columns from the results, which led me to posting this issue.
How about a new attribute: [SpreadsheetIgnore]?

Hi @rocjoe,
Thanks for the suggestion! My thinking is that if you don't want to include all the data, you would just make a new type that has only the fields you want, you could then select into the new type using LINQ or else use something like AutoMapper to map your data into the correct format you want before you send it to EPPlusEnumerable.
I can see the argument for using an attribute -- it prevents you from having to define a secondary type which is the same except for having a couple parameters taken out -- but at the same time it'd be easy for EPPlusEnumerable to get unwieldy really quickly by polluting your project with a million single-purpose attributes, and by extension making the EPPlusEnumerable code more complex and slower.
Does that sound reasonable?

Totally reasonable. KISS > convenience.

@bradwestness Hi! I'm new here. Just recently I encountered a situation when having an exclude attribute would be beneficial. I understand your case here, but imho, it's worth thinking through a scenario in which format of the exported data is unknown compile-time and the concrete type is emitted at runtime based on some configuration. I'm dealing with such a system now and having a possibility to just emit [SpreadsheetExcludeFromOutput] would be really a life saver there, cause we are not able to use LINQ nor AutoMapper to map the types easily without introducing several risks to the software while implementing custom reflection logic to delete properties not needed in output.
I forked the repository, and created a pull request to review this concept and small improvements in code. Obviously, this might need tuning further on to adjust my quick draft to proper style and coding guidelines. For now I'd greatly appreciate if you could spend just a few minutes to review the code to decide if we are on the same page here, if my changes do not conform to your plans, just let me know, cause I'm happy to help with other things for some time if that's fine for you.
I'm using VS 2015 so if you would like to merge the changes I might need to adjust the pull request a little to restore original nuget.exe and revert the sln changes, unless you want to start using VS 2015 further on.
Mentioned PR: #13

I'm confused. If you are emitting the types at runtime, can't you just not emit the field you don't want in the spreadsheet rather than including it but also emitting an attribute saying to exclude it?

@bradwestness it's a somewhat bigger customer-developed enterprise solution. We would need to duplicate the code a little to grab types and classes they emit based on some interfaces provided in code used for configuration. We are using the exclude attribute on base interfaces as we have limited to no access to the emit logic, which handles plenty of workflows before actually providing the final assembly. The other reason is performance overhead coming along another assembly generation process.
The issue here is: I understand that ideally we we would just provide properly schemed objects to EPPlusEnumerable, but sometimes it's just not feasible due to the project scope or other requirements when we have no direct control over everything in the code.

The <IP_ADDRESS> release now includes a SpreadsheetExcludeAttribute which can be used on one or more properties on a class to omit those from output in the generated spreadsheet. Note that as part of this change I also removed the ExcludeFromOutput property from the SpreadsheetTabNameAttribute as the same effect can be made by combining the SpreadsheetTabNameAttribute and SpreadsheetExcludeAttribute on a given property.


add Metigator channel

add Metigator channel https://youtube.com/c/Metigator

Thanks for submitting , I already added   it here https://youtubechannels.gatsbyjs.io/metigator-عصام-عبدالنبي
But I will update it with your content.


TextField in tableview cell

I can't see suggestions when typing.. i have tableview cell and textfield in it.
Any solution for this?
Image:

I am also having this problem.

Yeah, what's annoying about this is that the cell constrains the touch events
Damn, it's been one year already and still no solution, any news on this?
@mirzadelic what did you do to fix this issue?

@williamfs were you able to do anything to get this working?


S3 upload?

Resumable js is working good. But it will be great if it as a option to upload file directly to s3 after receiving chunk from frontend.

I suggest you have a look at this:
https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html
Is possible, and I have done it too.

there is evaporate.js for s3


using additional quilljs modules

could you please enable including additional quilljs modules (e.g. https://github.com/kensnyder/quill-image-resize-module, https://github.com/kensnyder/quill-image-drop-module), or provide some workaround for accomplishing this?

This is not possible currently but shouldn't be hard to add. Can you make a PR allowing to specify modules through props ?


File download failure

The CI is failing (in all O.S.), at least one reason is that the file download (for example bathymetry) cannot be completed.
Origin of the problem
I guess it's because DoX URL:

This works:

julia> download("https://dox.uliege.be/index.php/s/k0f7FxA7l5FIgu9/download")
"/tmp/jl_onMfMLiNcN"

This fails:

julia> download("https://dox.ulg.ac.be/index.php/s/k0f7FxA7l5FIgu9/download")
ERROR: RequestError: Could not resolve host: dox.ulg.ac.be while requesting https://dox.ulg.ac.be/index.php/s/k0f7FxA7l5FIgu9/download
Stacktrace:
...

Solved by changing the URL's related to DoX.
Also I propose the have the file paths and URLs in a common file config.jl that is called at the beginning of the notebooks. So that we don't have to repeat the URLs for the bathymetry, the data files etc.


Issues/adding compressors to gas asset yaml

This is not battle tested yet but using the original specs I'm able to set the low_to_high capacities and efficiences with the help of compressors.

[x]  Test it in a real LES

It doesn't freeze a LES except for reasons specified here https://github.com/quintel/etmoses/issues/996 .

@antw ,, hmm I might accidentally removed some commits of yours, could you maybe check if that's the case?

hmm I might accidentally removed some commits of yours, could you maybe check if that's the case?

I don't see anything my old local copy of this branch. It might have been this merge commit, but that's no problem as it's already on master.

I don't see anything my old local copy of this branch.

Than all is fine I guess 👯  was just checking to be sure

Merged manually, thanks!


text

k my code

Coverage decreased (-5.3%) to 43.635% when pulling 09bc76da7a37a4b583a2cfd21fc52369ee7957e6 on sravanthisiri:master into 06657a784bbc1eabdff5b2b3d3cd09a7084eb559 on angular-gantt:master.


No alias for --force

Related: #1059 and #1062

nice, thx for adding a test. can we send a PR for the optimist typings?


Add hashset in Java

Add HashSet in Java Collections with example and comment it properly.

Hey, I'd like to add the HashSet program.


Power Perceiver Refactor

Pull Request
Description
Refactor PP model into own folder and files. Add HuggingFace Hub support
Fixes #183
How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce.
Please also list any relevant details for your test configuration

[ ] Yes

If your changes affect data processing, have you plotted any changes? i.e. have you done a quick sanity check?

[ ] Yes

Checklist:

[ ] My code follows OCF's coding style guidelines
[ ] I have performed a self-review of my own code
[ ] I have made corresponding changes to the documentation
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] I have checked my code and corrected any misspellings

Codecov Report

Merging #190 (ddbfe61) into main (91f878c) will decrease coverage by 0.48%.
The diff coverage is 0.00%.

:exclamation: Current head ddbfe61 differs from pull request most recent head ecbe897. Consider uploading reports for the commit ecbe897 to get more accurate results

@@           Coverage Diff            @@
##            main    #190      +/-   ##
========================================
- Coverage   9.53%   9.04%   -0.49%
========================================
  Files         72      76       +4
  Lines       6977    7355     +378
========================================
  Hits         665     665
- Misses      6312    6690     +378

Impacted Files
Coverage Δ

power_perceiver/hub.py
0.00% <0.00%> (ø)

power_perceiver/production/model.py
0.00% <0.00%> (ø)

power_perceiver/production/pvrnn.py
0.00% <0.00%> (ø)

...ower_perceiver/production/satellite_transformer.py
0.00% <0.00%> (ø)

:mega: Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

Hey Jacob,
Would you be able to add a text to just check the model initialises? Im sure we could write lots more, but I think at least that first one would be handy

Yeah, definetely!

@peterdudfield the last error seems to be the same as you were seeing in loading the weights, how did you get around it?


Feature Request: support download zls

Maybe we can add feat to download zls, and just like using exec with zig, it also can use on zls!

Yeah that's a cool idea.

I'm trying to add zls support
But maybe we need to modify the command line, such as zmv zls i 0.13.0 or zvm i 0.13.0 --zls

I have a little question, why did you add a lot of assert in the source code?
I find that many assert are unnecessary

I have a little question, why did you add a lot of assert in the source code?
I find that many assert are unnecessary

I added it only to two functions. One in update_current to check that the zig_path and symlink_path variables are not empty before attempting to update the current symbolic link or directory.
And the other one in content. IMHO that's not a lot. In both cases, the assertions act as security checks.

I suggest introducing a command line parsing package. The current command function is no longer sufficient.
For example, the clap package is of very high quality.

I suggest introducing a command line parsing package. The current command function is no longer sufficient.
For example, the clap package is of very high quality.

What's the current limitation that we are facing? If there are no major obstacles, I want to avoid adding dependencies.

For example, 'install 0.13.0' is designed to take only one argument, but I think 'install zig 0.13.0' is more intuitive and can be distinguished from 'install zls 0.13.0'. It can be installing zig for default 'install 0.13.0'.

https://github.com/hendriknielaender/zvm/pull/57
Currently we only need that for zls as additional argument, so something like in my draft should work.

@jinzhongjia how do we proceed with the zls feature? IIRC there is still something missing.

Missing support for master branch compilation

But I don't have an additional manager to deal with this issue now.


How to add support for supportsAllDrives?

So in an earlier issue which has automatically closed, I was unable to upload files to shared drives in the actual build (works fine in editor though). https://github.com/Elringus/UnityGoogleDrive/issues/95. Instead of uploading to the shared folder, the files I make end up in my local drive in the build, but in Editor they upload to the shared folder just fine (the code in the build is identical).
Is there a way to add support for 'supportsAllDrives=true' which is required to have write permissions to shared folders? I can't see any mention of it in the plugin/github repo
The 'Shared with me collection' link here in the setup guide is dead so I'm at a loss as to what else I can do to make this work with shared drives if the api has changed.

More reference for this here https://stackoverflow.com/questions/63043156/cannot-interact-with-shared-drives-files-python-google-drive-api-v3

After much investigation and cross-comparing with the sample scenes in this repo, I realised this issue was caused by Managed Stripping Level being higher than 'Low', I have updated the readme for it here in this PR. I hope this is useful to other people


Feature request: Wrapper for Polymer

Would be extremely useful to have a wrapper as a webcomponent for polymer (polymer-project.org).
Thank you for such a great grid.

something like this:
http://localhost/example-web-component-grid/index.php

This is great -  managed to add it as a web component with Polymer.
Many thanks.
On Sun, Aug 9, 2015 at 5:34 PM, Niall Crosby<EMAIL_ADDRESS>wrote:

something like this:
http://localhost/example-web-component-grid/index.php
—
Reply to this email directly or view it on GitHub
https://github.com/ceolter/angular-grid/issues/368#issuecomment-129213617
.

Great. I plan on making it more 'web component' in the future, such as having the api methods directly on the DOM object, and also having the gridOptions as dom attributes where appropriate. any advice on this is welcome, you might know more about web components than i do!


llvm_5: 5.0.1 -> 5.0.2

Motivation for this change
https://lists.llvm.org/pipermail/llvm-announce/2018-May/000078.html

5.0.2  is API and ABI compatible with 5.0.0 and 5.0.1 and includes
mitigations for CVE-2017-5715 (Spectre Variant 2) for X86 and MIPS.

Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[x] Fits CONTRIBUTING.md.

Duplicate of #39979 which was merged to staging and will eventually appear in master.

Success on aarch64-linux (full log)
Attempted: llvm_5
Partial log (click to expand)
/nix/store/lz8mfyd61sjp1z8gqalhjr3gyxr95blj-llvm-5.0.2-python/share/opt-viewer/opt-stats.py: interpreter directive changed from "/usr/bin/env python2.7" to "/nix/store/vhz9s1dzzbndz032dysnixj8mxyfl30s-python-2.7.14/bin/python2.7"
/nix/store/lz8mfyd61sjp1z8gqalhjr3gyxr95blj-llvm-5.0.2-python/share/opt-viewer/opt-diff.py: interpreter directive changed from "/usr/bin/env python2.7" to "/nix/store/vhz9s1dzzbndz032dysnixj8mxyfl30s-python-2.7.14/bin/python2.7"
checking for references to /build in /nix/store/lz8mfyd61sjp1z8gqalhjr3gyxr95blj-llvm-5.0.2-python...
shrinking RPATHs of ELF executables and libraries in /nix/store/94vpmwh35pa0c8bia72p9d5shgzy6rn3-llvm-5.0.2-lib
shrinking /nix/store/94vpmwh35pa0c8bia72p9d5shgzy6rn3-llvm-5.0.2-lib/lib/libLLVM-5.0.so
strip is /nix/store/jk6j4lh9v5mvjdbdc35sj0zffhhf6s56-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/94vpmwh35pa0c8bia72p9d5shgzy6rn3-llvm-5.0.2-lib/lib
patching script interpreter paths in /nix/store/94vpmwh35pa0c8bia72p9d5shgzy6rn3-llvm-5.0.2-lib
checking for references to /build in /nix/store/94vpmwh35pa0c8bia72p9d5shgzy6rn3-llvm-5.0.2-lib...
/nix/store/j0shb5byalhjb63gy984l1y7h4i6453m-llvm-5.0.2

Success on x86_64-linux (full log)
Attempted: llvm_5
Partial log (click to expand)
/nix/store/1q4hxdy0iwrdzd9mhdi77zi4fajdxqrk-llvm-5.0.2-python/share/opt-viewer/opt-stats.py: interpreter directive changed from "/usr/bin/env python2.7" to "/nix/store/l9j2jsc9flrbmpf799nw9wdq59gpwms8-python-2.7.14/bin/python2.7"
/nix/store/1q4hxdy0iwrdzd9mhdi77zi4fajdxqrk-llvm-5.0.2-python/share/opt-viewer/opt-viewer.py: interpreter directive changed from "/usr/bin/env python2.7" to "/nix/store/l9j2jsc9flrbmpf799nw9wdq59gpwms8-python-2.7.14/bin/python2.7"
/nix/store/1q4hxdy0iwrdzd9mhdi77zi4fajdxqrk-llvm-5.0.2-python/share/opt-viewer/optrecord.py: interpreter directive changed from "/usr/bin/env python2.7" to "/nix/store/l9j2jsc9flrbmpf799nw9wdq59gpwms8-python-2.7.14/bin/python2.7"
checking for references to /build in /nix/store/1q4hxdy0iwrdzd9mhdi77zi4fajdxqrk-llvm-5.0.2-python...
shrinking RPATHs of ELF executables and libraries in /nix/store/z4ycndahahwxy2r380v143lrqsn9pisl-llvm-5.0.2-lib
shrinking /nix/store/z4ycndahahwxy2r380v143lrqsn9pisl-llvm-5.0.2-lib/lib/libLLVM-5.0.so
strip is /nix/store/j75dgadrff2d1fyc4fczmcgqkid2imdx-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/z4ycndahahwxy2r380v143lrqsn9pisl-llvm-5.0.2-lib/lib
patching script interpreter paths in /nix/store/z4ycndahahwxy2r380v143lrqsn9pisl-llvm-5.0.2-lib
checking for references to /build in /nix/store/z4ycndahahwxy2r380v143lrqsn9pisl-llvm-5.0.2-lib...


sorting of relations based on date is not working

the sorting of the relations should not happen based on the written date but on the iso-date. though this is the link one get's when clicking on 'End' on top of relations:
https://pmb.acdh.oeaw.ac.at/apis/entities/entity/place/213/edit?PePl-sort=end_date_written

this should be resolved in f99e4732c6a7a3282d6eb2f37fae3b4a4a545825


Create a test library

This PR does following.

Give execute permission to existing test.
Move common functions used by tests in libtest.sh and tests will source that file.
Run tests in separate shell instead of sourcing these into parent shell.

@cgwalters-bot r+ 8c74378

@cgwalters-bot r+ 8c74e78


Multiple Security Schemes in Security Requirement Object

The specification sates that: " ...
Lists the required security schemes to execute this operation. The object can have multiple security schemes declared in it which are all required (that is, there is a logical AND between the schemes).
..."
I would like to specify that exactly one of multiple/any  OAuth 2.0 flows can be used to authenticate the user, but the spec states that these are AND-ed, for example API-key and OAuth Authorization Code flow.
How can I define a security scheme that includes: authorization code, implicit and password OAuth flows - all three are fine, just pick one?

It is a bit confusing, but anything that's defined in a Security Requirement indeed has an AND between them. However, when you actually assign Security Requirements to the operations, there's an OR between those. So define one Security Requirement for each flow, and then mention them all in the array for the security.
See https://github.com/swagger-api/swagger-spec/blob/master/versions/2.0.md#swaggerSecurity.

Thanks.  I specified multiple security schemes for an operation. A couple of things.

As you "Authenticate" each scheme and use the "Try It" function, the online Swagger editor correctly fills in the headers for you, especially the Authorization header as "Basic" or "Bearer" - this is a great assist/affordance!
The online Swagger editor does not give any error for multiple security schemes for an operation, however the references to the second, third, etc on the list do not get resolved, resulting in a not used warning.

OK - so those were other errors - ignore that screen capture!

I removed all the warnings I created.  When using the online Swagger editor, just choose the authentication methods you wish to use for one or more given operations.  It will automatically fill these in for you.
The only problem is that there is no easy way to clear a previously set Authentication if for example you want to test that all methods work.

That sounds like a ticket for the swagger-editor project then ;)

Thanks for your help.


SentinelConnectionManager stops scheduling change check / dns check after error

Expected behavior
SentinelConnectionManager should continue to perform change check and dns check after encountering transient DNS issue like resolve hostname failure.
Actual behavior
Once there is an error (e.g. DNS resolution error), all future calls to SentinelConnectionManager.scheduleChangeCheck and SentinelConnectionManager.scheduleSentinelDNSCheck will be suppressed.
From a high level view, looks like it's due to the next schedule call is called inside .thenAccept. It probably should be .whenComplete.
Steps to reproduce or test case
Redisson.create(
   new Config()
    .useSentinelServers()
    .setMasterName("mymaster")
    .addSentinelAddress("redis://temporarily-unavailable-hostname:26379")
)

We can see that scheduleSentinelDNSCheck is called once and never again, similarly for scheduleChangeCheck .
Redis version
7.0
Redisson version
3.27.2
Redisson configuration
Like in above test case

Thanks for pointing out. Fixed


feat(views/index): add quick-update buttons for incidents

These buttons only appear if the current user is logged in
and are similar to resources/views/dashboard/incidents/index.blade.php.

Are you able to style the placement of the buttons a little better? :)

If I put the div inside the div.panel-heading the spacing looks more correct, thoughts?

That does look much nicer :)

@GrahamCampbell the style checks passed, does that need to be updated? What should the indentation be?

StyleCI doesn't check the Blade files, so it won't be seeing the dodgy spacing :smi

@GrahamCampbell, derp, sorry, didn't realize I had 2 spaces instead of 4 in the inner-div-html.
I've updated the commit to represent the new spacing and new location of the div.

Ok finally put the buttons into .btn-group and we're good :)

@jbrooksuk: done.

the style checks passed, does that need to be updated? What should the indentation be?

StyleCI doesn't check blade.

Smashing! :beers:

As a side note, since the links to the incidents is now used in more than one place in the code, maybe we should eventually refactor to use the route() function if possible? Maybe a partial? I'm not that familiar with the framework, so I will defer to you guys.

Keep up the good work, guys, and thank you for the quick turnaround!

It's something that we'll address in V2. We're going to make Cachet more framework-y.

@jbrooksuk What's the release time for v1 tomorrow?

It'll be after I finish work, so maybe 8PM GMT+1 (London).

@jbrooksuk So 8pm BST for us Londoners? :p

That'll be the one.
I do wish I'd said Saturday for release... Now I'm busy tomorrow night :(

@jbrooksuk Yeah... releasing v1!

:+1:


Keep track of the previous state

Keep track of the previous state to make it possible to react to some specific changes rather than to a given value. For instance, you could do:
func newState(state: AppState) {
  if let previousState = mainStore.previousState {
    if (state.imageIsLoaded && !previousState.imageIsLoaded) {
      print("image has just loaded")
    }
  }
}

[Current coverage][cc-pull] is 98.83%

Merging [#129][cc-pull] into [master][cc-base-branch] will decrease coverage by 1.16%

@@           master       #129   diff @@
========================================
  Files           6          6
  Lines         169        172     +3
  Methods         0          0
  Messages        0          0
  Branches        0          0
========================================
+ Hits          169        170     +1
- Misses          0          2     +2
  Partials        0          0

Powered by Codecov. Last updated by [b095f9e...916d303][cc-compare]
[cc-base-branch]: https://codecov.io/gh/ReSwift/ReSwift/branch/master?src=pr
[cc-compare]: https://codecov.io/gh/ReSwift/ReSwift/compare/b095f9e5d4b640d4ba277fe2ca9a184f62f990b4...916d3031b7f0c4db4ab1e6b105a6b7b310ceaa4f
[cc-pull]: https://codecov.io/gh/ReSwift/ReSwift/pull/129?src=pr

@gsabran Thanks a lot for this PR. 🎉
@ReSwift/reswift I think we hadn't discussed the full tradeoffs of storing the previous state as part of the core framework vs. offering it via some extension, see #93 ?
Sorry for the delay in this @gsabran but it's a somewhat impactful API change that we should discuss thoroughly.
Thanks a lot for the contribution! Let's see what other's have to say about the general direction 😄

Nice simple solution @gsabran 👍
I don't think adding it to the core store is quite the best solution when it's so easy and simple to create a store extension to add the same function. It would also introduce a lot of workarounds required when using replay and jumping to random timeline points. I'd like to keep the PR open though until there are examples in the documentation of how to do this.
The best place to be checking the for previous state is also normally in the ViewModel or even View where you would compare it against the last value displayed and update if they differ. This way the ViewModel can be independent of the Store and also not have to worry about missing a state or double checks on instantiation.

Shouldn't pulling this into a subtype work?
In my project, I was considering a Middleware which takes note of the previous state but haven't looked into it. If that'd work, I'd prefer a Middleware as a "core plugin", if you will.

Nice!
@agentk  are you saying that it'd be better to keep reference to the previous state in the view that'd need it, like
func newState(state: AppState) {
  if let previousState = self.previousState {
    if (state.imageIsLoaded && !previousState.imageIsLoaded) {
      print("image has just loaded")
    }
  }
  self.previousState = state
}

?
Also for context, I'm not very experienced with Swift/ReSwift and I'm coming from redux + React.js so any way you would think off that'd offer the same result and that'd integrate better with the ecosystem is perfect.
With React+redux, a state change in redux would trigger a property change in react components that would trigger componentWillUpdate(newProps, newState) (doc) on the component, and in there you can compare newProps to this.props (or similarly, after the update, componentDidUpdate(oldProps, oldState) is called (doc))
So it's more react than redux that makes available the previous state. This also suggests an other approach that would be a protocol similar to StoreSubscriber that calls a methods similar to componentWillUpdate / componentDidUpdate (ie the state comparison would be made available at the component level instead of within redux store). At the end, it'd be closer than what @agentk  suggested with the state thing handled at the component level instead of within the store. Something like
public protocol StoreSubscriberII: AnyStoreSubscriber {
    #if swift(>=2.2)
    associatedtype StoreSubscriberStateType
    #else
    typealias StoreSubscriberStateType
    #endif

    func stateWillChange(nextState: StoreSubscriberStateType)
}

extension StoreSubscriberII {
    var state: StoreSubscriberStateType?

    public func _newState(state: Any) {
        if let typedState = state as? StoreSubscriberStateType {
            #if swift(>=3)
                stateWillChange(nextState: typedState)
            #else
                stateWillChange(typedState)
            #endif
            self.state = state
        }
    }
}

And finally you could do
class ViewController: UIViewController, StoreSubscriberII {
  func stateWillChange(nextState: nextState) {
    if let state = self.state {
      if (nextState.imageIsLoaded && !state.imageIsLoaded) {
        print("image has just loaded")
      }
    }
  }
}

The steps you outline sound like the view controller would check for state changes/diffs in stateWillChange but not update the view, there; in order to work with the actual data you then have to wait for newState nevertheless, do I get this right? If so, your check would fit into newState just as well it seems.
I like the idea of performing more atomic changes to the view. I'd put a presenter or similar kind of object between store and view; it'd receive newState updates from the store and check for changes. (What @agentk suggested to move into the view model.) Then you can perform changes on view model properties based on the result of the diffing, leaving unchanged values untouched. didSet observers or Reactive Cocoa bindings from properties to view components will do the job.
Here's some code with bind/bindAndFire from Srdan Rasic's post to stay framework agnostic.
extension Presenter {
  func newState(state: AppState) {
    if let previousState = self.previousState {
      if (state.imageIsLoaded && !previousState.imageIsLoaded) {
        self.viewModel.loadingFinished.value = true
      }
    }
  }
}

class ViewModel {
  let loadingFinished: Dynamic<Bool> = Dynamic(false)
}

class ViewController: UIViewController {

  @IBOutlet var loadingIndicatior: LoadingIndicatorView!

  var viewModel: ViewModel! {
    didSet {
      viewModel.loadingFinished.bindAndFire { [weak self] in
        self?.loadingIndicatorView.update(loaded: $0)
      }
    }
  }
}

I think we resolved the original question here. If not, please let me know and we can reopen the issue. Thanks!


[Basic/Darwin] Follow the llvm convention of including additional code with .inc extension, instead of .cpp

@swift-ci smoke test and merge


Add parameter for leak tracking to testWidgets.

Reincarnation of https://github.com/flutter/flutter/pull/137203
Contributes to https://github.com/flutter/flutter/issues/135856
Prerequisite: leak tracker is configured in G3.
Next steps:

Update leaky tests to set leak tracking off (with just added parameter)
Start using the parameter for leak tracking (https://github.com/flutter/flutter/pull/138057)
Replace testWidgetsWithLeakTracking with testWidgets.

Google Testing fails because dependency is not added to G3 build yet. When I try to add it in G3, presubmit fails, because the dependency is not used yet. What is right sequence?
Should I ignore google testing here or add dependency in G3 and ignore unused dependency somehow?

Why does this PR exist? Why is it not enough to just do https://github.com/flutter/flutter/pull/138057/files?

Why does this PR exist? Why is it not enough to just do #138057?

Because, before merging #138057 we need to update all leaking tests to have leak tracking off, that requires this parameter.

Google testing succeeded: https://critique.corp.google.com/cl/582106273

@goderbauer , addressed your comments. Can you lgtm?

Can you please get the PR that removes this hack ready for submission before we submit this so we keep the time this hack exists to a minimum? Presumably, that would be https://github.com/flutter/flutter/pull/138057
Leak tracking has already accumulated a lot of technical debt in our code base, I am wary to add more to that pile.


chore(main): release 0.1.7

Auto generated release
0.1.7 (2024-03-06)
Build System

deps: bump slim/slim from 4.12.0 to 4.13.0 (#23) (d85bb96)
deps: bump symfony/var-dumper from 7.0.3 to 7.0.4 (#22) (07dd8e5)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/jobtrek/php-slim-todo/releases/tag/v0.1.7 :sunflower:


Use modal windows to indicate the beginning and end of each season

A banner will be used to notify users


Implement the quantum part of Shor's algorithm.

Classical part has already been implemented: https://github.com/quantumlib/Cirq/blob/master/examples/shor.py

Assign to me.


Flow run crashed with ValueError: Step has unexpected additional keys: branch, repository, access_token

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Terraform Version
Terraform v1.10.2
on linux_amd64

provider registry.terraform.io/hashicorp/aws v5.80.0
provider registry.terraform.io/prefecthq/prefect v2.12.0

Affected Resource(s)
Prefect deployment
Terraform Configuration Files
resource "prefect_flow" "flow" {
  name         = "flow-test"
  workspace_id = var.prefect_workspace_id
  tags         = ["test-flow"]
}

resource "prefect_deployment" "deployment" {
  name                     = "test-deployment"
  description              = "test deployment"
  workspace_id            = var.prefect_workspace_id
  flow_id                  = prefect_flow.flow.id
  entrypoint               = "flows/flow.py:my-flow"
  tags                     = ["test", "dev"]
  enforce_parameter_schema = false

parameters = jsonencode({
    "parameters_uri" : "s3-path",
    "chunk_size" : 5000
  })

  parameter_openapi_schema = jsonencode({
    "type" : "object",
    "properties" : {
      "parameters_uri" : { "type" : "string" }
      "chunk_size"     : { "type" : "integer" }
    }
  })

  path                = "/flows"
  paused              = false
  pull_steps = [
  {
      type         = "git_clone"
      repository   = "https://my-git-repo.git"
      branch       = "main"
      access_token = var.github_token
  }
  ]
  version             = "v1.1.1"
  work_pool_name      = var.work_pool_name
  work_queue_name     = "default"
}

Debug Output
Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 160, in run_steps
    step_output = await run_step(step, upstream_outputs)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 116, in run_step
    raise ValueError(
_ValueError: Step has unexpected additional keys: branch, repository, access_token_

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 437, in retrieve_flow_then_begin_flow_run
    else await load_flow_from_flow_run(
  File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 100, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/deployments.py", line 307, in load_flow_from_flow_run
    output = await run_steps(deployment.pull_steps)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 188, in run_steps
    raise StepExecutionError(f"Encountered error while running {fqn}") from exc
prefect.deployments.steps.core.StepExecutionError: Encountered error while running access_token

Expected Behavior
The flow run should run without a crash or failure.
Actual Behavior
The flow is crashing with ValueError: Step has unexpected additional keys: branch, repository, access_token. It is failing the pull_steps, is there anything that needs to be added to pull_steps? The documentation has only four fields to pull the flow code in the pull_steps that is type, branch, repository, and access_token https://registry.terraform.io/providers/PrefectHQ/prefect/latest/docs/resources/deployment#type-1
Steps to Reproduce

terraform apply
Run flow from the Prefect UI

Thanks for the report here @tejalal, I'm digging into this and it looks like the pull_steps payload may be malformed in the provider. I'll follow up with any updates/PRs.

That was exactly it, good eye. Already working on a PR, should be up soon 👍🏼

Hi @mitchnielsen, is it ready to use in 2.12.0 or a new version is to be released.


Open of hashes with relative path fails

nsrlsvr currently requires the path of the hashes file to be absolute. I opt to get and test the absolute path before daemonizing also see: https://github.com/rjhansen/nsrlsvr/issues/11
Also some suggestion on improving the error messaging:
https://github.com/rjhansen/nsrlsvr/blob/master/src/main.cc#L88
  if (not infile) {
    log(LogLevel::ALERT, "couldn't open hashes file " + hashes_location);
    exit(EXIT_FAILURE);
  }

Which results in:
./src/nsrlsvr -p 9120 -f ./hashes.txt
daemon started
couldn't open hashes file ./hashes.txt

But this provides me no insight into why the file could not be opened. Also IMO not infile is less clear what it is checking than the equivalent not infile.is_open().
I opt to change this to (or equiv):
  if (not infile.is_open()) {
    log(LogLevel::ALERT, "couldn't open hashes file " + hashes_location + " with error: " + strerror(errno));
    exit(EXIT_FAILURE);
  }

Which results in:
./src/nsrlsvr -p 9120 -f ./hashes.txt
daemon started
couldn't open hashes file ./hashes.txt with error: No such file or directory

Only one request per ticket, please.  :)
Relative paths: this is the sort of thing that's easy to do badly, but which can be done correctly.  There are extremely good reasons to not allow daemons to use relative paths, but the path could be converted to absolute with realpath().  This one is a good idea, and I'll see about implementing it.
Better error reporting: there is no guarantee the C++ runtime will implement streams with C I/O, meaning we cannot rely on errno.  Better error reporting needs to happen but not like this.  I'll think about how to do it.
Code clarity: this is straight from Bjarne Stroustrup's style recommendations.  Not changing it.

Progress report: relative paths works for me now.  Need to kick on it for a while more before I check it into -master.  Expect by the end of the week.
Better error reporting: some improvement.  Still working on figuring out the clearest way to handle this.  Expect by end of the week.

Sounds good, thanks

Fix has been checked into git, but not well-tested.  Use -master at your own risk, as always.  :)

thx, will try and let you know if I run into things

Closed after three weeks.  I'm assuming the fix is acceptable.  :)


Improve error message when template tag is skipped

Version
2.6.11
Reproduction link
https://codesandbox.io/s/currying-shadow-inm2w?file=/src/App.vue
Steps to reproduce

Create a .vue file with template, with 2 elements inside and a script tag.
Remove the wrapping <template></template> tag.
It sparks a confusing error:

Module parse failed: Unexpected token (7:0)
File was processed with these loaders:
 * ./node_modules/vuetify-loader/lib/loader.js
 * ./node_modules/cache-loader/dist/cjs.js
 * ./node_modules/vue-loader/lib/index.js
You may need an additional loader to handle the result of these loaders.
|
|
> <v-layout wrap>
|     <v-flex xs4>
|         <v-select

What is expected?
The error should say that template tag was skipped.
What is actually happening?
Current error message isn't very helpful.

In such a case on codesandbox the error message is actually helpful:
Failed to mount component: template or render function not defined.
but it's different when compiled via yarn serve.
I've wasted many minutes trying to figure out the reason behind this error, as my IDE (WebStorm) doesn't highlight lack of template as an error. With a more helpful error message developers could save time spent on debugging.

Besides the fact that it doesn't make sense to remove the template tag once it has been added, this warning would conflict for users adding custom blocks and components with no template tag. If anything, this should be picked up at an IDE level not at Vue or vue-loader level

Of course it doesn't make sense to remove this tag, but mistakes happen. It doesn't have to literally say "template not defined", just make it more helpful - codesandbox is doing it right, vue-cli isn't.


Add GPS location

Allow users to use their location instead of writing down address

Hello. Can I take this one? I would like to get some experience with PWA's.

@billy-le Hi! Sure! For the UI I was thinking something like a end adornment for the location text field using the GpsFixed icon.
Thanks for the help and happy hacktoberfest!

Thank you, great work!
Closed in #19


[DEFECT] import (discovery) of RAVEN contrib seems to be compromised

Thank you for the defect report

[X] I am using the latest version of RAVEN.
[X] I have read the Wiki.
[X] I have created a minimum, reproducible example
that demonstrates the defect.

Defect Description
I did a fresh installation of RAVEN and the execution of tests cases result in an import error:
 ... successfully imported "ExamplePlugin" ...
Traceback (most recent call last):
  File "/Users/alfoa/projects/raven/framework/Driver.py", line 31, in <module>
    from Simulation import Simulation
  File "/Users/alfoa/projects/raven/framework/Simulation.py", line 28, in <module>
    import Steps
  File "/Users/alfoa/projects/raven/framework/Steps/__init__.py", line 22, in <module>
    from .Factory import factory
  File "/Users/alfoa/projects/raven/framework/Steps/Factory.py", line 24, in <module>
    from .SingleRun import SingleRun
  File "/Users/alfoa/projects/raven/framework/Steps/SingleRun.py", line 30, in <module>
    import Models
  File "/Users/alfoa/projects/raven/framework/Models/__init__.py", line 25, in <module>
    from .Dummy import Dummy
  File "/Users/alfoa/projects/raven/framework/Models/Dummy.py", line 31, in <module>
    from Decorators.Parallelization import Parallel
  File "/Users/alfoa/projects/raven/framework/Decorators/Parallelization.py", line 20, in <module>
    from utils import importerUtils as im
  File "/Users/alfoa/projects/raven/framework/utils/importerUtils.py", line 26, in <module>
    import contrib.lazy.lazy_loader as lazy_loader
ModuleNotFoundError: No module named 'contrib.lazy'

This issue seems to be triggered by the modifications added in https://github.com/idaholab/raven/pull/1685 (even if it does not seem clear why to me)
Steps to Reproduce
Fresh installation of RAVEN.
Expected Behavior
Correctly import RAVEN dependencies in contrib
Screenshots and Input Files
 ... successfully imported "ExamplePlugin" ...
Traceback (most recent call last):
  File "/Users/alfoa/projects/raven/framework/Driver.py", line 31, in <module>
    from Simulation import Simulation
  File "/Users/alfoa/projects/raven/framework/Simulation.py", line 28, in <module>
    import Steps
  File "/Users/alfoa/projects/raven/framework/Steps/__init__.py", line 22, in <module>
    from .Factory import factory
  File "/Users/alfoa/projects/raven/framework/Steps/Factory.py", line 24, in <module>
    from .SingleRun import SingleRun
  File "/Users/alfoa/projects/raven/framework/Steps/SingleRun.py", line 30, in <module>
    import Models
  File "/Users/alfoa/projects/raven/framework/Models/__init__.py", line 25, in <module>
    from .Dummy import Dummy
  File "/Users/alfoa/projects/raven/framework/Models/Dummy.py", line 31, in <module>
    from Decorators.Parallelization import Parallel
  File "/Users/alfoa/projects/raven/framework/Decorators/Parallelization.py", line 20, in <module>
    from utils import importerUtils as im
  File "/Users/alfoa/projects/raven/framework/utils/importerUtils.py", line 26, in <module>
    import contrib.lazy.lazy_loader as lazy_loader
ModuleNotFoundError: No module named 'contrib.lazy'

OS
MacOS
OS Version
No response
Dependency Manager
CONDA
For Change Control Board: Issue Review

[X] Is it tagged with a type: defect or task?
[X] Is it tagged with a priority: critical, normal or minor?
[X] If it will impact requirements or requirements tests, is it tagged with requirements?
[X] If it is a defect, can it cause wrong results for users? If so an email needs to be sent to the users.
[X] Is a rationale provided? (Such as explaining why the improvement is needed or why current code is wrong.)

For Change Control Board: Issue Closure

[ ] If the issue is a defect, is the defect fixed?
[ ] If the issue is a defect, is the defect tested for in the regression test system?  (If not explain why not.)
[ ] If the issue can impact users, has an email to the users group been written (the email should specify if the defect impacts stable or master)?
[ ] If the issue is a defect, does it impact the latest release branch? If yes, is there any issue tagged with release (create if needed)?
[ ] If the issue is being closed without a pull request, has an explanation of why it is being closed been provided?

@PaulTalbot-INL FYI.

Hm. It's odd that none of our regressions machines with fresh checkouts are seeing this issue. Is this local to a specialized install?

Hm, is the line in setupCpp() in CustomDriver/DriverUtils.py:
utils.add_path(os.path.join(frameworkDir,'contrib'))
not getting run before the import for Simulation?

Also, https://github.com/idaholab/raven/pull/1784 will likely either fix this, or really mess it up.

it seems that utils.add_path(os.path.join(frameworkDir,'contrib')) is run (I checked before opening the issue) but without the "init" the hierarchy is not correctly captured. Another approach is to modify utils.add_path(os.path.join(frameworkDir,'contrib')) to utils.add_path_recursive(os.path.join(frameworkDir,'contrib')) that worked too.
I do not know why this has been triggered by the  https://github.com/idaholab/raven/pull/1685 (and not before, since the calls have been simply moved and not "modified")...

Hm, https://github.com/idaholab/raven/pull/1773 also changes how raven does paths and went in soon after #1685

New approach works without a need for a fix. Thank you.


Request the addition of 3 new characters

Could you please add inverted 4, inverted 6,  inverted 7  and inverted 9 to the character list - this would then allow a complete inverted number display for use in head-up displays, where the LED numbers are reflected off the vehicle windscreen.
At present, 1 inverts to itself, 2 inverts to 5, 3 inverts to itself, 5 inverts to 2, 8 inverts to itself, and 0 inverts to itself, so you can see what's missing.
For inverted 4, light segments B,C,E & G
For inverted 6, light segments A,B,D,E,F & G
For inverted 7, light segments B,C & D
For inverted 9, light segments A,B,C,D,E & G
TIA

Realised that this won't work as decimal point won't invert, so will have to build display with inverted 7 segment led modules.

My previous comment was wrong. I have reopened this issue. Definitely need these 4 new characters. 3 inverted is E!

Going to close this issue and rewrite it.


Look into adding nzbToMedia to nzbget.

This will make it postprocess right when it done(Event based). Instead of waiting for cron to poll the download folder for new completed downloads.
Provides an efficient way to handle postprocessing for CouchPotatoServer and SickBeard
https://github.com/clinton-hall/nzbToMedia

Already done. nzbToMedia is already installed and working with Sickrage, couchpotato, and mylar.
Closing.

Perfect. I guess I missed that step.


How to determine what a namespace contains

How do we determine what objects still exist in our namespace? We are deleting everything we can find, but we're missing something; and, it's not clear what objects are preventing us from performing the deletion.

@devinbost
you can use bin/pulsar-admin topics list <tenant>/<namespace> to list all the topics within a namespace. if there are topics, you are prevented to delete a non-empty namespace.


feat(friends): be able to search friends

let's say I'm friends with 3 of Sheldon's accounts and he FR's me again. I need to find the old Sheldon accounts (for cleanup) and would like to enter "Shel" into a search bar and find all matching friends.
This would be helpful when finding someone you want to chat with too.

Might want to note that it might be wise to show the short id in the search along with the username to identify the difference between the user profiles.


Testing cursor position for controlled inputs in enzyme

I am working on a controlled input and am explicitly setting the cursor position in componentWillUpdate() & onKeydown, onChange. The behavior seems right but I'm trying to write some tests for this. The selectionStart always seems to be 0 even if I set it explicitly. How do I go about testing it ?
This is what my test looks like:
const component = mount(<FileOne {...properties} />);
    const input = component.find('input');
    console.log('current cursor position:', input.node.selectionStart); // this will be 0
    // set cursor position to 2
    input.node.selectionStart = 2;
    console.log('current cursor position 1:', input.node.selectionStart); // this is 2
    // call onFocusHandler
// even if I use simulate('focus') here it doesn't help
    component.instance().onFocusHandler();
    // cursor position now should still be 3
    console.log('current cursor position 2:', component); // this is always zero. It should be 2 How do I test this?

**FileOne.jsx**
<Wrapper1>
 <Wrapper2 handleKeyDown={this.handleKeyDown} onFocusHandler={this.onFocusHandler}/>
</Wrapper1>

**two.jsx**
<Wrapper3 handleKeyDown={props.handleKeyDown} onFocusHandler={props.onFocusHandler}/>

**wrapper3.jsx**
<input class='abc' onKeyDown={props.handleKeyDown} onFocus={props.onFocusHandler}/>

Also curious to see if this is a bug, or if there's a way to test this?

I'd suggest avoiding use of simulate; it doesn't actually simulate anything. Instead, explicitly invoke the prop you're interested in.
Being able to test text selection requires that you get the DOM node in question, and use the native text selection APIs. I'd be happy to accept a PR to the docs to illustrate how this can be done, if someone's interested, but I think it's outside the scope of enzyme.


Automoderator exploit using special characters

If a punctuation mark like +, ., -, or ! appears at the beginning or end of a banned word, the automoderator will ignore it (not tested with automoderator v2).  Let me know if you need additional info or testing, we can reproduce it pretty reliably

Asked about this on Discord but it got lost to the ether.
https://discord.com/channels/166207328570441728/384011387132706816/764354443932073994

I have this exact problem. I was originally using YAGPDB for its ability to set a custom slow mode duration, but when I found out that it also had a banned word feature, I jumped at the opportunity to get rid of another redundant bot (It was better anti spam, if you are wondering), so it is a shame that I have to get it back.

Anything around a banned word makes YAGPDB ignore it, not just "!" or "?".

This the expected behavior. Amod works on exact match, to prevent false positives.
For example, if you ban the word "sun", should the world "disunity" also be flagged? In most cases, no.
You guys can always use amodv2's "msg matches regex" trigger. You just type whatever you want (let's say egg), and everytime these 3 letters appear in a msg, the bot will be triggered, but be aware that you are going to end up with a lot of false positives.

That makes sense, though simple punctuation like a full stop or a comma throwing it off is a problem- they do not even have to try to get a banned word through, they can just add an exclamation mark to the end of a word and it can just slip through.
About the message matches regex thing. Is there a way to quickly input a long list of words for the regex? I can only make it only react to one word at a time, and creating another rule just for another word seems inefficient an unnecessary. Advice would be appreciated!

(word1|word2|word3|phrase 1) for example for multiple words and phrases. I think it is better to join the support server for help with further details.

Thanks for the help and for the info- I didn't know there was a support server.


Create ipeindia.txt

added Institute of Public Enterprise

please add my college

@zweizwei please add ipe to the college list


Update vmx_vpfe_config.xml

Variable for interface model type was set to virtio. Updated to '{{ interface.type }}'

Nice, thanks for the catch!


refactoring load_data and open_list_archives o archive.py

Addresses #435
I haven't worked through the notebooks yet to see if they are effected by this change.
I couldn't remember if the new Codecov runs the notebooks or not...

I'm getting lots of errors in the notebooks.
I don't think we've added automated testing of the notebooks into our group workflow yet.
But one way to do this is:
pip install nbval
pytest --nbval-lax examples/

I'm getting tons of errors and need to figure out why.

Ok, the notebooks were not passing locally for many unrelated reasons.
Automated tests are now passing here. I request a review.

Great feedback, @npdoty . Thank you. I will try to address as many of these points as I can in the next round of edits.

Codecov Report

Merging #500 (fb72448) into main (0b4794a) will increase coverage by 0.26%.
The diff coverage is 75.16%.

@@            Coverage Diff             @@
##             main     #500      +/-   ##
==========================================
+ Coverage   74.82%   75.09%   +0.26%
==========================================
  Files          21       22       +1
  Lines        3019     3047      +28
==========================================
+ Hits         2259     2288      +29
+ Misses        760      759       -1

Flag
Coverage Δ

unittests
75.09% <75.16%> (+0.26%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

bigbang/mailman.py
61.42% <20.00%> (-1.22%)
:arrow_down:

bigbang/parse.py
48.24% <28.12%> (-7.86%)
:arrow_down:

bigbang/archive.py
71.01% <91.07%> (+7.14%)
:arrow_up:

tests/unit/test_archive.py
95.74% <95.74%> (ø)

bigbang/w3crawl.py
91.20% <100.00%> (ø)

tests/unit/test_mailman.py
97.70% <100.00%> (+0.53%)
:arrow_up:

bigbang/listserv.py
79.60% <0.00%> (ø)

bigbang/analysis/listserv.py
64.35% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0b4794a...fb72448. Read the comment docs.

@npdoty I've address those issues which I could easily cover in the scope of the original #435.
It seems like this is bringing up many issues of broader scope concerning BigBang's architecture.
I'd recommend not letting these broader issues block this PR, which is at least an incremental improvement.


Update install.md

Update install.md to fix docker bind directory of project with /tmp/ .Files may be deleted by system
in /tmp/ yet.

这样直接改掉的话，这两句命令就变成不是直接可用的了，我更倾向是对这个 /tmp 参数进行解释

Update install.md to fix docker bind directory of project with /tmp/ .Files may be deleted by system
in /tmp/ yet.

倾向于使用 /tmp, 命令可以直接跑. 跑完之后想改目录, 用户可以直接把整个项目 mv/cp 过去


`default_nettype wire

Him
can you please explain this pull request please. I personally never use this default nettype wire anti-feature and I would prefer to add a resetall statement, which IMO is missing.
Damien


Environment Detector detects thunder storms when just raining.

Description
The isThunder() function of the Environment Detector block returns true when it's raining but there is no thunder storm.
I am using the Feed The Beast mod pack called "FTB Presents Direwolf20 1.18" which uses version 1.18.2-0.7.16b of the mod but I have manually changed to 1.18.2-0.7.19r (not included in the version selection drop box below) to see if the bug persists, which it does.
Steps to reproduce

Place a computer beside an environment detector block.
In the computer write a Lua script to detect the state of the weather:

local detector = peripheral.find("environmentDetector") -- Finds the peripheral if one is connected
if detector == nil then error("environmentDetector not found!") end

term.clear()

while true do
  local timer = os.startTimer(1)
  local event, id
  repeat
    event, id = os.pullEvent("timer")
  until id == timer

  term.setCursorPos(1,1)
  if detector.isRaining() then
    if detector.isThunder() then
      term.write("A Thunder Storm!    ")
    else
      term.write("It's raining        ")
    end
  else
    term.write("The weather is clear")
  end
end

Run the script.
In Minecraft control prompt type: weather clear
In the computer you will see it says the weather is clear.
Set the weather to rain with the Minecraft command: weather rain
In the computer you will see it says there is a thunder storm!

Multiplayer?
Yes
Version
1.18.2-0.7.19r
Minecraft, Forge and maybe other related mods versions
Forge 40.1.69 Minecraft 1.18.2
Screenshots or Videos
No response
Crashlog/log
No response

Only fixed in 1.19, need a fix in 1.18.2 too.


SPARK-7889 Jobs progress of apps on complete page of HistoryServer shows uncompleted

This patch pulls all the application cache logic out of the HistoryServer and into its own class with callbacks for operations (get &c). This makes unit tests straightforward with a small bit of mocking of SparkUI.

Merged build triggered.

Merged build started.

Test build #35453 has started for   PR 6935 at commit 9dad9d1.

Test build #35453 has finished for   PR 6935 at commit 9dad9d1.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class CacheEntry(ui: SparkUI, completed: Boolean, timestamp: Long);
class PCAModel(JavaVectorTransformer):
class PCA(object):

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #35454 has started for   PR 6935 at commit d96f29a.

Test build #35454 has finished for   PR 6935 at commit d96f29a.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class CacheEntry(ui: SparkUI, completed: Boolean, timestamp: Long);
class Module(object):
class PCAModel(JavaVectorTransformer):
class PCA(object):

Merged build finished. Test FAILed.

CacheEntry is unintentional public class; will fix. I have no idea where the others came from

style check. Not enough spaces, by the look of things
[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala:2: Header does not match expected text
[error] (core/compile:scalastyle) errors exist
[error] Total time: 9 s, completed Jun 22, 2015 7:49:06 AM
[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/core/src/test/scala/org/apache/spark/deploy/history/ApplicationCacheSuite.scala:2: Header does not match expected text
[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/core/src/test/scala/org/apache/spark/deploy/history/ApplicationCacheSuite.scala:172:4: Insert a space after the start of the comment
[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/core/src/test/scala/org/apache/spark/deploy/history/ApplicationCacheSuite.scala:187:4: Insert a space after the start of the comment
[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/core/src/test/scala/org/apache/spark/deploy/history/ApplicationCacheSuite.scala:199:4: Insert a space after the start of the comment
[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/core/src/test/scala/org/apache/spark/deploy/history/ApplicationCacheSuite.scala:60:71: No space before token =
[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/core/src/test/scala/org/apache/spark/deploy/history/ApplicationCacheSuite.scala:68:59: No space before token =
[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/core/src/test/scala/org/apache/spark/deploy/history/ApplicationCacheSuite.scala:105:21: No space after token =
[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/core/src/test/scala/org/apache/spark/deploy/history/ApplicationCacheSuite.scala:115:56: No space after token =
[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/core/src/test/scala/org/apache/spark/deploy/history/ApplicationCacheSuite.scala:216:40: No space after token ,
[error] (core/test:scalastyle) errors exist
[error] Total time: 6 s, completed Jun 22, 2015 7:49:24 AM
[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala:2: Header does not match expected text
[error] (core/compile:scalastyle) errors exist
[error] Total time: 9 s, completed Jun 22, 2015 7:49:45 AM
[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/core/src/test/scala/org/apache/spark/deploy/history/ApplicationCacheSuite.scala:2: Header does not match expected text
[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/core/src/test/scala/org/apache/spark/deploy/history/ApplicationCacheSuite.scala:172:4: Insert a space after the start of the comment
[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/core/src/test/scala/org/apache/spark/deploy/history/ApplicationCacheSuite.scala:187:4: Insert a space after the start of the comment
[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/core/src/test/scala/org/apache/spark/deploy/history/ApplicationCacheSuite.scala:199:4: Insert a space after the start of the comment
[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/core/src/test/scala/org/apache/spark/deploy/history/ApplicationCacheSuite.scala:60:71: No space before token =
[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/core/src/test/scala/org/apache/spark/deploy/history/ApplicationCacheSuite.scala:68:59: No space before token =
[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/core/src/test/scala/org/apache/spark/deploy/history/ApplicationCacheSuite.scala:105:21: No space after token =
[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/core/src/test/scala/org/apache/spark/deploy/history/ApplicationCacheSuite.scala:115:56: No space after token =
[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/core/src/test/scala/org/apache/spark/deploy/history/ApplicationCacheSuite.scala:216:40: No space after token ,

@steveloughran, I think you always put too many space in some place.

I should add that I did think about actually using the last-updated information to decide whether to refresh or not, and decided simply having a timeout was a good enough step #1

@steveloughran  Have you tested it? I don't think it's ok.
yeah, you realized the refreshing of incompleted apps. But I think second time sending "/history/appid" request , it will not goes into /history handler, but goes to history/appid handler, so it will not call the refresh code and won't refresh.
/cc @squito

NAVER - http://www.naver.com/
<EMAIL_ADDRESS>님께 보내신 메일 <Re: [spark] SPARK-7889 Jobs progress of apps on complete page of HistoryServer shows uncompleted (#6935)> 이 다음과 같은 이유로 전송 실패했습니다.

받는 사람이 회원님의 메일을 수신차단 하였습니다.

I probably tested different bits of it ... I verified that updated apps were coming in, but no, not a full functional test of the Web UI, as that would take a lot of preamble to set up those tests. I do have all the code to set up such tests in SPARK-1537, but was trying to keep thing separate... if we get that in then this would be easy to add as a new test case

@steveloughran  @XuTingjun  sorry I finally took a closer look here.  So I tried running a manual test on this patch and https://github.com/apache/spark/pull/6545, and it seems to me that neither one solves the problem.  (Or perhaps I am misunderstanding the issue.)
Here's the steps I took:

for this patch, modify the refresh time to 1 second just for testing
run the history server within an sbt session

build/sbt
project core
re-start
<choose option for HistoryServer>

In another window, fire up a spark-shell with event logging enabled, and run a few jobs (and leave the shell open)

bin/spark-shell --conf "spark.eventLog.enabled=true"
val d = sc.parallelize(1 to 100)
d.count()
d.count()

View the app UI in the history server at localhost:18080, and also view the apps own UI at localhost:4040
go back to the shell, run another d.count() job
take another look at the UI, both for the app itself (which updates) and via the history server (which does not update, regardless of how long I wait.)

@XuTingjun it sounded like you had done some manual testing on https://github.com/apache/spark/pull/6545 and believe that it should work.  Is there something wrong that I'm doing with my test?
Unless I'm testing the wrong thing, I think this demonstrates the need to write a unit test which basically does the same thing, as its hard to verify the complete correctness here.  I'm writing that test case now.

If you want to do tests, grab the code

WebsiteIntegrationSuite
AbstractTestsWithHistoryServices
*IncompleteSparkUISuite

the tests themselves aren't that complex, but the underlying runner bits to bring up web pages, run probes once they are up, etc, are a lot of foundational pain.

sorry I didn't see that comment about the tests ... I wrote something up which mimics the existing UISeleniumSuite.
From 2dddcd036a4cacf31d1ff5f15417fcdbcfdbc22e Mon Sep 17 00:00:00 2001
From: Imran Rashid<EMAIL_ADDRESS>Date: Fri, 26 Jun 2015 14:16:16 -0500
Subject: [PATCH] failing test case

---
 .../spark/deploy/history/HistoryServerSuite.scala  | 74 +++++++++++++++++++++-
 1 file changed, 72 insertions(+), 2 deletions(-)

diff --git a/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala b/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala
index e5b5e1b..a288490 100644
--- a/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala
+++ b/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala
@@ -21,14 +21,24 @@ import java.net.{HttpURLConnection, URL}
 import java.util.zip.ZipInputStream
 import javax.servlet.http.{HttpServletRequest, HttpServletResponse}

+import scala.concurrent.duration._
+import scala.language.postfixOps
+
 import com.google.common.base.Charsets
 import com.google.common.io.{ByteStreams, Files}
 import org.apache.commons.io.{FileUtils, IOUtils}
+import org.json4s.JsonAST.JArray
+import org.json4s.jackson.JsonMethods._
 import org.mockito.Mockito.when
+import org.openqa.selenium.WebDriver
+import org.openqa.selenium.htmlunit.HtmlUnitDriver
+import org.scalatest.concurrent.Eventually
+import org.scalatest.selenium.WebBrowser
 import org.scalatest.{BeforeAndAfter, Matchers}
 import org.scalatest.mock.MockitoSugar

-import org.apache.spark.{JsonTestUtils, SecurityManager, SparkConf, SparkFunSuite}
+import org.apache.spark.util.Utils
+import org.apache.spark._
 import org.apache.spark.ui.SparkUI

 /**
@@ -43,7 +53,7 @@ import org.apache.spark.ui.SparkUI
  * are considered part of Spark's public api.
  */
 class HistoryServerSuite extends SparkFunSuite with BeforeAndAfter with Matchers with MockitoSugar
-  with JsonTestUtils {
+  with JsonTestUtils with Eventually with WebBrowser {

   private val logDir = new File("src/test/resources/spark-events")
   private val expRoot = new File("src/test/resources/HistoryServerExpectations/")
@@ -264,6 +274,66 @@ class HistoryServerSuite extends SparkFunSuite with BeforeAndAfter with Matchers
     justHrefs should contain(link)
   }

+  test("incomplete apps get refreshed") {
+
+    implicit val webDriver: WebDriver = new HtmlUnitDriver
+    implicit val formats = org.json4s.DefaultFormats
+
+    val testDir = Utils.createTempDir()
+    testDir.deleteOnExit()
+
+    val myConf = new SparkConf()
+      .set("spark.history.fs.logDirectory", testDir.getAbsolutePath)
+      .set("spark.eventLog.dir", testDir.getAbsolutePath)
+      .set("spark.history.fs.update.interval", "1")
+      .set("spark.eventLog.enabled", "true")
+      .set("spark.history.cache.interval", "1")
+      .remove("spark.testing")
+    val provider = new FsHistoryProvider(myConf)
+    val securityManager = new SecurityManager(myConf)
+
+    val server = new HistoryServer(myConf, provider, securityManager, 18080)
+    val sc = new SparkContext("local", "test", myConf)
+    server.initialize()
+    server.bind()
+    val port = server.boundPort
+    try {
+      val d = sc.parallelize(1 to 10)
+      d.count()
+      val appId = eventually(timeout(20 seconds), interval(100 milliseconds)) {
+        val json = getContentAndCode("applications", port)._2.get
+        val apps = parse(json).asInstanceOf[JArray].arr
+        apps.size should be (1)
+        (apps(0) \ "id").extract[String]
+      }
+
+      def getNumJobs(suffix: String): Int = {
+        go to (s"http://localhost:$port/history/$appId$suffix")
+        findAll(cssSelector("tbody tr")).toIndexedSeq.size
+      }
+
+      getNumJobs("") should be (1)
+      d.count()
+      eventually(timeout(10 seconds), interval(100 milliseconds)) {
+        // this fails with patch https://github.com/apache/spark/pull/6935 as well, I'm not sure why
+        getNumJobs("") should be (2)
+      }
+
+      getNumJobs("/jobs") should be (2)
+      // Here is where we still have an error.  /jobs is handled by another servlet,
+      // not controlled by the app cache.  We need some way for each of those servlets
+      // to know that they can get expired, so they can be detached and then the reloaded
+      // one can get re-attached
+      d.count()
+      getNumJobs("/jobs") should be (3)
+
+    } finally {
+      sc.stop()
+      server.stop()
+    }
+
+  }
+
   def getContentAndCode(path: String, port: Int = port): (Int, Option[String], Option[String]) = {
     HistoryServerSuite.getContentAndCode(new URL(s"http://localhost:$port/api/v1/$path"))
   }
--
2.2.2

patch https://github.com/apache/spark/pull/6545 is closer to working (for some reason the cache does not seem to be getting refreshed in this PR, maybe my test is setup wrong).  But I do think the cache with a refresh interval is probably a better way to go, rather than reloading the data on every single request.
I have a rough understanding of what is going wrong when we make requests to http://localhost:18080/history/<appId>/jobs -- after the first time, that is handled by a completely separate servlet, as @XuTingjun has pointed out, which is never detached.  We need a way to wrap each of the servlets for the UI so they know if they need to refresh.
Also I just realized, the same problem applies to the rest api at http://localhost:18080/api/v1/applications/, but that should be much simpler to handle, its just one servlet.  We should add a test case and fix that too, though.

@squito, I am agree with what you said. First about my patch#6545, it will refresh only on request http://localhost:18080/history/<appId>

@steveloughran do you think you can pick this up with the test case?  I think we have a little better understanding of the problem.  I do think your approach is in the right direction.

yeah, I think this patch also can't refresh. The place where detache the handlers is not right.
I think we need to thinking when and where to detache the handlers.

I like the selenium test; it could be combined with the provider i wrote which lets us programmatically create our own history, so add changes we can look for.

(BTW, I'm not going to be looking at this for a couple of weeks; focusing on a (big) patch

Test build #46534 has started for PR 6935 at commit b8deeee.

Test build #46534 has finished for PR 6935 at commit b8deeee.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n  * case class CacheEntry(ui: SparkUI, completed: Boolean, timestamp: Long);\n

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46534/
Test FAILed.

Merged build finished. Test FAILed.

Test build #46535 has started for PR 6935 at commit 6e48da6.

Test build #46535 has finished for PR 6935 at commit 6e48da6.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n  * case class CacheEntry(ui: SparkUI, completed: Boolean, timestamp: Long);\n

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46535/
Test FAILed.

Merged build finished. Test FAILed.

Test build #46537 has started for PR 6935 at commit fe9504f.

Test build #46537 has finished for PR 6935 at commit fe9504f.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46537/
Test FAILed.

As discussed on #6545, I think we could actually switch to live UI playback, provided the number of cached UIs was kept down.
What must be done here is keeping a limit on the number of threads doing playback; a pool will be needed equal to the size of the loaded UIs, or, if the provider starts them,  the number of cached UIs must be limited and when removed from the cache, the thread has to be returned to the pool.
This could not only handle incomplete applications, it should be able to offer a more responsive load of completed ones -the UI could be registered while event playback was in progress.
Proposal

Supplement getAppUI(appId: String, attemptId: Option[String]): Option[SparkUI] with one to start Spark UI playback; some  case class returned with the UI, playback info (in progress), and a closure to call when releasing the cached entry
when loading a complete attempt, the async playback would be executed until the playback finished, then the close() operation called.
when loading an in progress app attempt, the async playback would continue until one of: app completes, app release from cache, history server halted (for in-VM tests)

Test build #46602 has started for PR 6935 at commit d2008d5.

Test build #46602 has finished for PR 6935 at commit d2008d5.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46602/
Test FAILed.

Merged build finished. Test FAILed.

Test build #46635 has started for PR 6935 at commit 8ba44a2.

This is a converged patch which

Splits out all caching into ApplicationCache, for isolation and testing.
Now asks the history server (which then asks the HistoryProvider) to see if an app has changed. If so, it triggers a refresh.
Includes the metrics counting requests and timing probe/getAppUI calls, so when a descendant of #9571 goes in, the cache is instrumented. Those counters are used in the tests to check internal cache state BTW, not without immediate use.
The test there is test at the cache level, with a stub implementation of the operations the history server has to do. Its incomplete (need one to verify that on cache eviction the UIs are detached).

There's nothing for spark UI tests (there's the template above there), or the REST API

Test build #46635 has finished for PR 6935 at commit 8ba44a2.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46635/
Test FAILed.

@squito —I'm adding your test case to this patch. It's never worked, has it?

@steveloughran correct, that test case never passed on either patch.  Thanks for picking this back up, it would definitely be nice to get this working.

I'm having a go, though I'm feeling somewhat defeated

OK, here's the latest revision
lots of tests on the cache itself -all working. This includes cnew hecks on eviction.
merged in the @squito test with a bit extra to make sure the saved histories are up to date.
The tests aren't working: those GET operations under app/id just aren't triggering GET calls yet.
I think a filter is going to be needed on those bits of the web UI: when a GET/HEAD comes in under an app, the cache should be hit to see if it is up to date. If there's a new version, then the background detach/attach should get in there before the request is serviced, or it is updated in some other way.

Test build #46707 has started for PR 6935 at commit fdfdcd5.

Test build #46707 has finished for PR 6935 at commit fdfdcd5.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46707/
Test FAILed.

Test build #46708 has started for PR 6935 at commit f75e39f.

Test build #46708 has finished for PR 6935 at commit f75e39f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46708/
Test FAILed.

The test failing is the web UI one; after hitting the web UI there's a check on the cache metrics. The fact that lookup.count == 1 means the 2nd GET didn't trigger a cache lookup
org.apache.spark.deploy.history.HistoryServerSuite.incomplete apps get refreshed

Failing for the past 1 build (Since Failed#46708 )
Took 12 sec.
Error Message

The code passed to eventually never returned normally. Attempted 82 times over 10.060489942 seconds. Last failure message: Wrong lookupCount value - expected 2 but got 1 in metrics lookup.count = 1 lookup.failure.count = 0 eviction.count = 0 load.count = 1 update.probe.count = 0 update.triggered.count = 0 .
Stacktrace

sbt.ForkMain$ForkError: The code passed to eventually never returned normally. Attempted 82 times over 10.060489942 seconds. Last failure message: Wrong lookupCount value - expected 2 but got 1 in metrics
lookup.count = 1
lookup.failure.count = 0
eviction.count = 0
load.count = 1
update.probe.count = 0
update.triggered.count = 0

Thinking about this some more, how about instead of trying to do something clever with the REST UIs, web filters and stuff, we just have a background thread that probes the current set of cached incomplete apps and evicts them on a regular basis if they've changed.
Something that scans every 5 minutes with a nice simple policy

Unchanged since loaded: leave
Changed since loaded but loaded within a certain minimum "recently" time: leave (stops UI being viewed from being continually reset)
Changed since loaded outside of "recently" threshold: evict (& detach). Do not attempt to reload until UI interaction.

This design would mean that incomplete apps that were changing would be evicted, and, if users will still looking at them, be reloaded. If they weren't being viewed, there would be no replay cost.

Updated patch. This does add a web UI filter, one which is being attached to incomplete apps, and triggering checks of the underlying history, the goal being when an updated account is reached, issue a 302 and then detach and reattach the UI. Ideally, that will mean that on the next request the newly loaded UI is picked up.
The test is failing. Currently because the fs history provider isn't picking up the updated work. More investigations are needed. First step: adding some sleeps in the test to guarantee the updated file has an updated timestamp

Test build #46971 has started for PR 6935 at commit 3c6c5df.

Test build #46971 has finished for PR 6935 at commit 3c6c5df.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46971/
Test FAILed.

Test build #46985 has started for PR 6935 at commit ec7c581.

... now it's looking like somethings up with the modtime info in the local FS; I'm thinking of tracking the file length as well: a bigger file -> more records

Test build #46996 has started for PR 6935 at commit 6fea55d.

Test build #46985 has finished for PR 6935 at commit ec7c581.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46985/
Test FAILed.

Test build #46996 has finished for PR 6935 at commit 6fea55d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46996/
Test FAILed.

the reason the modtime doesn't change is the log file is kept open; the mod time is set when the output stream is created and not updated as new data is added. The only viable probe for new data is going to be looking for new data.
This probe may need to handle incomplete data; the event log listener does do a {{flush()}} on major events —that is not enough to guarantee that changes are visible across HDFS readers (more precisely, there are no guarantees if/when writes to a file are picked up by existing input streams; they may get an out of date view of the data, or an inconsistent one). Because the input stream is re-opened on the playback, the risk is low —but is it there? If so, the risk has always existed when loading an incomplete app, it's just with one load per run (ignoring cache evictions), there's significantly fewer loads of incomplete apps than if it can be triggered by GETs on a loaded app UI.

This is the next iteration; if you look at the intermittent patches I was trying to track the time the filesize changed, but it (a) made the code complex (b) didn't actually work.
Now there's an atomic long generational counter: every time any attempt is updated, the counter is incremented (i.e. it is unique across all apps -we could make it per-attempt but I don't see what that would gain right now).
there is a scan in the update loop for has filesize changing: this triggers an update of the generation counter but no re-read of the application. This means that the updates are low-cost to track.
As before, the check for app updates on modtime do trigger a re-read; as the EventLoggingLIstener always attempts to set the modtime in its stop() command, the mod time is always updated on any filesystem which supports setTimes(), even if rename() doesn't. That includes POSIX, but still omits the object stores. The fact that rename is copy+delete should handle that implicitly.
#I worry things are over complex; I'll need to diff with master after the update and see if I can trim it a bit.
Comments welcome! now it should be ready for initial reviews

Test build #47165 has started for PR 6935 at commit 9f63116.

Test build #47167 has started for PR 6935 at commit b084c4f.

Test build #47167 has finished for PR 6935 at commit b084c4f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47167/
Test PASSed.

Test build #47165 has finished for PR 6935 at commit 9f63116.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47165/
Test FAILed.

Hi @steveloughran I've done one rough pass over the code, thanks for working on this.  This definitely looks promising, the unit tests seem pretty good so its great that those are passing.  It does seem surprisingly complicated -- I pointed a few confusing places, I know you're still working on it, perhaps some cleanup will improve things.
On some higher-level things: in an earlier comment you mentioned:

This probe may need to handle incomplete data; the event log listener does do a {{flush()}} on major events —that is not enough to guarantee that changes are visible across HDFS readers (more precisely, there are no guarantees if/when writes to a file are picked up by existing input streams; they may get an out of date view of the data, or an inconsistent one). Because the input stream is re-opened on the playback, the risk is low —but is it there?

I don't understand the danger here.  FsHistoryProvider.replay always creates a new InputStream, so it doesn't seem like there is anything to worry about?
On https://github.com/apache/spark/pull/6545 you mentioned the idea of having an incremental replay:

There's one more option to consider here. Why do we actually replay-then-pause in-progress applications? Why not have a thread continually loading them and forwarding events until they complete? In this mode, there's no need to reload the cache, replay events....the UI just stays up to date.

That sounds like a fantastic idea -- but I assume that ended up being much more complicated in the end?
 I agree with the complexity; the initial pass "modtime" just didn't work, with a granularity of a couple of seconds on some filesystems, changes were sometimes not being detected at all, especially with 2 jobs back-to-back. Tracking filesizes did work, but it was still having problems. I spent a lot of time looking at debug level logs comparing timestamps on log messages with those of cache & provider entries, as well as FS values. Run the test at debug level and you'll see the details and diagnostics I had to put in.
one particular deal breaker was that the "when-ness" of metadata updates became visible was potentially trouble. I could sketch out a sequence of operations where the writer updated a file (which got a new modtime & size), but that change didn't immediately become visible. Reader does a read of a file & gets the old time, while recording the time (> modtime) of the the read. Polling back would not pick up the changes, as the now-updated information was appearing with older timestamps. A generation counters of file size changes took away all the ambiguity.
w.r.t the continuous streaming of data, I didn't try that here, I just went for eviction. I'm not confident that it will scale to the really big clusters with many spark jobs running simultaneously, long-lived streaming things, short lived analytics works running behind hive queries, etc, etc.
This cache eviction should scale as

It only looks at jobs that are being actively viewed by users.
the cost of the poll/refresh operation itself is very low cost. A directory scan.
you can set a window for how long an incomplete app can live in the FS before even polling for changes. On a big cluster, you could set it to a couple of hours.

Actually, on the very big clusters we may want to think about disabling eviction entirely. That won't pick up the transition of an incomplete app to a complete one: we may need a way for a provider to detect —in a provider specific manner— whether or not an attempt/app has completed, even without the poll.
Now that EventLogListener.stop() explicitly sets the modtime on files, this should happen on all filesystems. I should just order the test sequence so that the listing checks take place before any GET calls on the attempt UI.
If someone really does want to do incremental playback, I think what we have here would be the foundation for a very slick way of doing it: record the position in the history, and start from there. That Option[Any] could be filled with whatever static information is needed to start the replay, and on a poll the provider could just play the new data in —then return false to say "don't evict this SparkUI". That'd be pretty slick as there'd be no background threads, (cost O(incomplete-apps) & expensive), just a load dependent on user interaction O(users), replay cost O(delta-since-last-update), and whatever static memory gets used up, which is presumably some PSPACE thing, still O(incomplete-apps), but less than the cost of a thread in both terms of memory allocation and CPU load/swap hit, etc.
I'm not going to do that, at least not here. What this patch can do is act as a starting point for this though: on-demand incremental updates on incomplete apps
If someone wanted to take up I promise I'll help review the code

on the incremental playback -- agreed, no need to tackle it here, I was just curious since you had mentioned it earlier.

Test build #47204 has started for PR 6935 at commit 2382aac.

Test build #47204 has finished for PR 6935 at commit 2382aac.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47204/
Test PASSed.

Test build #47271 has started for PR 6935 at commit fd5e282.

Test build #47271 has finished for PR 6935 at commit fd5e282.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47271/
Test PASSed.

FWIW, I've got the SPARK-1537 Yarn history provider hooked up to this in a branch. That uses a version counter on every entity publish made by the client on an attempt; that's in the metadata so all it takes is an enum of recent apps + a version comparison to look for newer values. With the code in this branch merged in I can uncomment the incomplete spark UI test i'd had to skip.

thanks for updating and expanding the comments.  I think I understand the version better now.  As I understand it, in the PR right now, the version increases only if the file size increases.  In fact, it seems that as far as the code here is concerned, you could just replace the version with the file size, and it would have the same effect.  I can see perhaps wanting some more general concept of version in the cache, but couldn't you just use file sizes inside FsHistoryProvider?
Or if version is supposed to cover both timestamp & file size, I think there needs to be a change in logic

hi @steveloughran thanks for the updates.  The only real issue I see now is some inconsistency between whether the checks are time stamp or filesize based (or both).  Perhaps I am missing something, but I think its just some bits of leftover code after you changed logic around through working on it.  I think its just style stuff after that.

Test build #47424 has started for PR 6935 at commit 9b76c44.

Test build #47424 has finished for PR 6935 at commit 9b76c44.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47424/
Test FAILed.

Merged build finished. Test FAILed.

Failing test is pyspark —pretty unlikely to be related
======================================================================
FAIL: test_count_by_value_and_window (__main__.WindowFunctionTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/streaming/tests.py", line 635, in test_count_by_value_and_window
    self._test_func(input, func, expected)
  File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/streaming/tests.py", line 160, in _test_func
    self.assertEqual(expected, result)
AssertionError: Lists differ: [[1], [2], [3], [4], [5], [6], [6], [6], [6], [6]] != [[1], [2], [3], [4], [5], [6]]

First list contains 4 additional elements.
First extra element 6:
[6]

- [[1], [2], [3], [4], [5], [6], [6], [6], [6], [6]]
?                           --------------------

+ [[1], [2], [3], [4], [5], [6]]

----------------------------------------------------------------------
Ran 7 tests in 43.333s

Jenkins, add to whitelist

Jenkins, retest this please

thanks for updating again @steveloughran .  Sorry for the long break in my reviews.  Looks really close, I think I just found one bug with the redirects.  Other than that smaller style stuff.
This has turned out to be a lot more complicated than it seemed initially, thanks for sticking with it.

yeah, I found it surprisingly complex too ... if I known at the start I'd have steered clear. And don't worry about any delay; there are some other spark PRs of mine which are older, and I have been neglecting too many hadoop core patches to have any right to be feel even mildly neglected.
I'll look at the issues.
I also want to extend the tests to the REST API too, including the actual getlogs operation.
I'll have something up by the weekend

NAVER - http://www.naver.com/
<EMAIL_ADDRESS>님께 보내신 메일 <Re: [spark] [SPARK-7889] [CORE] HistoryServer to refresh cache of incomplete applications (#6935)> 이 다음과 같은 이유로 전송 실패했습니다.

받는 사람이 회원님의 메일을 수신차단 하였습니다.

I've updated the patch with the comments, and reworked how the updated probe works, removing the need to have provider-specific state cached, returned in the callback and cast.
Now, when an app UI is loaded, a probe is returned which has it's {{isUpdated()}} predicate evaluated —and if true, a cache refresh triggered. The FS probe looks a filesize; the yarn one will do a version counter, anything else can do there own.
This is a lot cleaner and purer: the only more puritanical option would be to go from an abstract probe class to simply passing a function {{Unit=>Boolean}} around.

hi @steveloughran , I think maybe some commits didn't get pushed?  I noticed a few places where things didn't seem to quite line up to your comments.  (also there is a merge conflict w/ master while you are touching it again.)

I'm going to add a warning note here pointing to HDFS-5478, some versions of HDFS aren't picking up changes immediately. I think this is older versions of Hadoop, not the current 2.6+ line.

@squito —have you had a chance to look at the latest version? I think I've addressed all your issues, and it is building and testing against the current master

jenkins, add to whitelist

hi @steveloughran sorry for another really long delay on my end.  mostly this looks fine, there are some style nits and a couple of comments that need updating.  I also looked into the query param thing -- I played with it a bit more and realized its kind of nuisance to test, but I did write a test on ApplicationCache for it which I'll send to you.
I just had one more concern with the last read-through -- what happens when an app goes from incomplete to complete?  I did some manual testing, and things seem to work.  But, I have a fear that there is some lingering state that isn't getting cleaned up.  I will try to walk through things more carefully but maybe you understand it well enough that you can reassure me (or perhaps you should take another look yourself as well ...)
and of course, there are now merge conflicts which need to be fixed, sorry.

...thx for the feedback. I'm fixing the merge which is now triggering a regression —maybe a race condition in test startup—
apps should go from incomplete -> complete; that's one of the recurrent issues we have today. When the attempt is replayed, it's complete flag is changed and the fs history applications field updated. When the history server asks for list of apps, it gets the updated histories, and so picks up the change

..I should add that it depends on the head attempt on the list being complete; the filter in HistoryServer is very sensitive to ordering. If there's an incomplete history older than a complete one, the app is considered incomplete

rebased against master; switch from scan of HTML view to REST API to enumerate listings of complete/incomplete apps, add @squito's ? arg redirection and test

Jenkins, test this please

Test build #2522 has started for PR 6935 at commit 728b12c.

Test build #2522 has finished for PR 6935 at commit 728b12c.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

ps can you close this one now?


Kill thread more aggressively on SIGINT

Had some issues with Ctrl-C where the exit handler would get stuck in an infinite loop. Not quite sure what's going on, but just killing the thread more aggressively seems to resolve it.
Previous code was mostly in ef064248 so @aksh-at lmk if you have any concerns with this?

I don't remember why I added this, but we're using the _close_loop function explicitly from our hackily patched test event loop: https://github.com/modal-labs/modal/blob/a83758e6b710fbc2522b0a04d5ac90f4bd84d3cf/integration_tests/conftest.py#L39. It might have to do with the pytest issues—I remember there were some warnings at some point about the event loop not being closed.
Anyway it looks like the timeout in join doesn't actually kill the thread, so we might still need some way to close the event loop and kill the thread?


Fix user action processing code.

User actions are packed after bits of InputState packets.
current code tries to read as many packets as it can in CRUDLink<EVENT_FACTORY>::event_for_packet.
Since only Map's InputState packets have additional 'following' packets. the logic should be moved closer to InputState processing part ( maybe by storing ByteStream with contents in InputState packet, and processing bits in void MapInstance::on_input_state(InputState *st)

This has been solved, only thing left is implementing per packet-type serialization code.


[RFC] Towards a release 1.1.0

You may have notices that Spring has introduced the Spring Platform module, which handles many of the version-management, thus you can keep your sanity when developing with spring.
Achieving spring-platform-1.1.2 compatibility with the current sources of light-admin is a bit more difficult than achieving spring-boot-1.2.3 compatibility.
Would you (the users) prefere that this project releases now a 1.1.0, which would work with spring-boot-1.2.3?
And go for spring-platform compatibility in the next release, say 1.1.1 or 1.2.0?
Or would you prefere that this is done in one release (with a delay of a few days or weeks)?
(I'm anyway not sure of what will hapen, just requesting some comments)

Not relevant anymore, as spring-platform compatibility is achieved (except for apache.tiles, but things still work fine)