Error Prone Performance Problems.

Hi guys, I'm working on integration of Error Prone into few Lyft Android/Java projects.
During integration I've faced significant build time increase to a point where it became unusable.
I started investigating, benchmarking and profiling different Error Prone setups.
Here is reproducible, open-source benchmarking & profiling setup I've came up with:
https://github.com/artem-zinnatullin/error-prone-performance
It uses RxJava as a test project and integrates Error Prone with different configurations:

Error Prone as a replacement for JDK's compiler / as javac plugin
JDK 8 / JDK 9
Default configuration / Some checks disabled
etc

Here is short sum of my findings:

Error Prone adds significant overhead to compilation time

JDK 8: up to 3.57x slower
JDK 9: As compiler plugin up to 2.15x slower

Error Prone adds significant memory overhead on JDK 8 with Parallel GC: up to 2.58x more memory used during compilation

For more info, benchmark & profiling results, instructions & scripts to reproduce it on your end please refer to the readme of the project

What seems to cause Error Prone overhead?
According to method call profiling, Error Prone spends significant amount of time on this subset of checks:

PrivateSecurityContractProtoAccess.matchMethodInvocation 2.8% - 663 ms (Protobuf)
ImmutableModification.matchMethodInvocation 2.4% - 551 ms (Guava)
ShouldHaveEvenArgs.matchMethodInvocation 0.8% - 195 ms (Truth)
CollectionIncompatibleType.matchMethodInvocation 0.6% - 142 ms (JDK)
NamedParameterChecker.matchMethodInvocation 0.3% - 62,342 µs (JDK)

While these checks might be relevant to many of the Google's projects, only 2 of them are relevant to the test RxJava project.
Profiling Lyft projects where I tried to intgrate Error Prone gives similar results: lots of time is spent on checks that are not relevant to the codebase in the first place.
Potential improvements
Laziness
I hope it's possible to optimize these checks by checking compilation classpath and excluding irrelevant checks from analysis. For example, whole category of Guava checks can be excluded if Guava is not found in compilation classpath.
Parallelism
I hope it's possible to parallelize AST analysis since Error Prone doesn't modify it by default.
Profiling suggests that compilation step where Error Prone does its work only uses single thread.

Tested on Error Prone 2.2.0.

P.S. I'm not trying to blame Error Prone developers, I'm here to help :) I just want to use it in our setup 😸

Thank you for the thorough investigation and for providing an easy way to reproduce your results!
Some thoughts:

Internally we only run the errors as part of the builds; we show warnings in code review, which is less latency sensitive.  This means that if a warning has performance problems, we probably won't notice.  Can you try disabling the warnings and seeing what the performance looks like?
Re: Parallelization.  This initially seemed feasible because Error Prone checks are stateless and don't modify the AST.  However, they do invoke symbol resolution, and javac is not thread safe, so this caused lots of problems.  You might be able to work around this by wrapping calls to the symbol table and type info with locks, but that is probably a substantial amount of work.
The idea to check the compilation classpath and disable irrelevant checks is interesting.  However, at Google compilation classpaths are huge (10k+ jars) and just walking them takes significant time.  Do external builds also have giant classpaths, or is this a Googlism?

There is a benchmark configuration where I disable some irrelevant and expensive checks that popped up after profiling graph1, graph2, it saves some ~200ms (which is already a lot since original compile time is ~4100ms), but that's just few checks: https://github.com/artem-zinnatullin/error-prone-performance/commit/98df505802e7c9f0928970e57dc2210fbba5035d.
Ideally I'd add a benchmark configuration where I disable irrelevant categories of checks, but that option is not supported yet https://github.com/google/error-prone/issues/648#issuecomment-310760961, maybe if I have time I can do it in my fork.
I can however add a configuration with all checks disabled if you want to get a baseline overhead of Error Prone (which I suppose will be very small)
I see, well that's unfortunate. I guess keeping modules small and building them in parallel already partially solves the issue. But experimenting with concurrent access to symbol table would't hurt :)
Hm, that sounds strange to me tbh. Wasn't the idea of using monorepo + build tool (Bazel/Blaze) that's optimized for large amounts of modules to get modules with small set of dependencies? That's certainly the case in our Android monorepo. However Error Prone is generic tool, so it should work on large classpaths as well, I guess we need to benchmark that and compare what's better.

@artem-zinnatullin I think the more common configuration is to only run error prone for continuous integration. While I'm sure it could be made faster I'm not sure it is ever going to get close enough to native javac where a human can wait for the result and not be annoyed.

2.48x slower than JDK 8 javac

It'd be interested to see how that compares to JDK 9. Error Prone is using the JDK 9 javac, so it's possible you're seeing a performance regression between the JDK 8 and 9 in addition to the overhead from Error Prone. There shouldn't be appreciable overhead with all of the checks disabled.

@codylerum At Google we do run Error Prone during development as well as during the continuous build, in fact that's the intended workflow.
  If we are, then that points to a bug that we should fix. @artem-zinnatullin, do you have a comparison against JDK 9 javac?

@eaftan we do! Baseline (no Error Prone) benchmarks & profiling made on JDK 8 and 9 show difference between Oracle JDK's 8 and 9 javac (see README to get idea if JDK 8/9 javac itself is slower/faster and comparing JDK 9 baseline to Error Prone with all checks disabled gets us another piece of comparison.
Oracle JDK 8 javac vs JDK 9 javac

JDK 8:

Median build time: 4.173 sec
Max "Used size" memory: 0.89 GB

JDK 9:

Median build time: 4.918 sec, 1.17x slower 🔻
Max "Used size" memory: 1.14 GB, 1.28x memory used 🔻

So, Oracle JDK 9 javac itself is slower than 8 and eats more memory (memory is tricky tho) while compiling RxJava 2.x.
Oracle JDK 9 javac vs Error Prone 2.2.0 all checks disabled

Oracle JDK 9:

Median build time: 4.918 sec
Max "Used size" memory: 1.14 GB

Error Prone 2.2.0 all checks disabled (ran on Oracle JDK 8):

Median build time: 10.358 sec, 2.14x slower 🔻
Max "Used size" memory: N/A (haven't measured yet)

Error Prone 2.2.0 with all checks disabled is 2.14x slower than Oracle JDK 9 javac.
So yeah, it looks like an issue to me. Unless there is a major difference between running Error Prone on JDK 8 vs 9 (which I can't reproduce atm because of Oracle JDK 9 EOL, see below).

@codylerum according to "Building Useful Program Analysis Tools Using an Extensible Java Compiler" paper that describes ideas behind Error Prone (it's pretty old though, 2012), building checks inside compiler is the whole point of integrating it smoothly into both local development and CI builds. It also claims that integrating it into compiler lets reuse and save time on AST parsing.
That paper also mentions 9% execution time overhead and 0.95% memory overhead, again, it was in 2012, but current numbers are so far from that that it raises concern no matter if you run it locally or on CI.

@eaftan you can try to estimate how much time and money can be saved by optimizing Error Prone since it's used as compiler for most of Google's Java code afaik and it's a default compiler in Bazel.
My understanding is that getting it closer to at least 2x build time overhead can save millions per year on developer productivity and CI resources at scale of Google.
Maybe that can help prioritize this issue :) (not that I want to push you, I just have a ticket to integrate NullAway blocked by Error Prone performance hehe)

// Haven't done it for Java 9+ with error-prone as javac plugin yet, Java 9 EOL broke setup artem-zinnatullin/error-prone-performance#2

I'm planning to integrate OpenJDK 10 and maybe 9 instead of Oracle JDK 9.
There were some issues with OpenJDK 9 and Gradle & gradle-grofiler https://github.com/gradle/gradle/pull/4954, it'll take some time for me to update setup and add benchmark results.
Probably I should switch whole setup to OpenJDK only, no ETA on that though.

@artem-zinnatullin Good to know. My dev enviroment is primarily eclipse so not possible for me.

There are a couple of checks that -XepDisableAllChecks doesn't disable (RestrictedApiChecker, CompileTimeConstant) which explains why there is some performance overhead relative to stock JDK 9. We're paying a fixed cost to scanning the entire AST regardless of how many checks run.
I collected some measurements using your rxjava example and a batch invocation of javac without using gradle, e.g.:
javac -processorpath error_prone_ant.jar -Xplugin:ErrorProne \
-XDcompilePolicy=byfile -d /tmp/classes -cp reactive-streams-1.0.2.jar \
-sourcepath : $(find src/main -name *.java)

The numbers aren't good, but I'm also not seeing 2x overhead with -XepDisableAllChecks. I wonder if there's a difference in the way gradle sets up the compilation with Error Prone enabled that's having an effect.
As @eaftan mentioned we don't enable warnings for interactive builds (and we should provide configuration that does that), so the 1.6x with warnings disabled is closer to the configuration we actually use. There are also some optimizations in Bazel that move some of the time spent in Error Prone off the critical path of builds.
Anyway, there's clearly work that needs to be done here. Thanks for bringing this up!

Experiment
Wall Time (s)
Normalized

JDK 9 + -XDcompilePolicy=by_file
7.471 ± 0.163
1.00

Error Prone + -XepDisableAllChecks
8.441 ± 0.222
1.13

Error Prone (warnings disabled)
12.208 ± 0.193
1.63

Error Prone (default settings)
15.252 ± 0.172
2.04

@cushon heh, I was going to point you that

batch invocation of javac

is not the best benchmark because you start a JVM for javac on every iteration and not warming up classes. You can actually see that my benchmarks show very similar numbers to yours on the first warm-up.
But that actually pointed me to something else, in my benchmarks you can see that Error Prone setup with gradle-errorprone-plugin is barely getting improvements after warm-ups and I figured that Gradle plugin messed up Error Prone class loading so they were not using JIT-ted versions between runs.
I've patched the plugin to cache class loader and that alone was able to gain back about 166% ✅ of the overhead comparing to Oracle JDK 8 javac. PR https://github.com/tbroyer/gradle-errorprone-plugin/pull/68
Benchmark:

It'll take some time to verify patch, review and merge but that's already a huge win for Gradle users with 0 changes to Error Prone itself which is nice.

I've briefly reviewed another Gradle plugin gradle-errorprone-javacplugin-plugin that integrates Error Prone as javac plugin, and quick look doesn't reveal similar class loading issue.
However patched JDK 8 setup now outperforms JDK 9 setup with Error Prone as javac plugin by around 36%, so I think there is something wrong too.
Anyway it's still far from 9% execution time overhead, so I think we should continue looking for a ways to optimize Error Prone performance.
To isolate from Gradle problems I think you should continue directly invoking javac, just maybe build a small JVM app to class load javac/error-prone and be able to warm it up properly before measurements.
Unfortunately I have tons of other tasks, so I'll be replying here on best effort basis, hope that's enough information to dig into that on your end!

@eaftan You mentioned that you only run errors as a part of the build. Is there an easy way to configure this (besides listing all patterns manually)? It seems that #785 is open for a while now

There are a couple of checks that -XepDisableAllChecks doesn't disable (RestrictedApiChecker, CompileTimeConstant)
WWAT?


package.json "main" should just be JS entry point

The main field in package.json is set up to be an array.  However, this should just be the JavaScript file so that tools like Browserify can know the module's entry point.
It should probably be just src/tipso.min.js.  I'm happy to send a PR if you agree.

Note: I'm not sure how this works with bower.json, so it might be 100% ok there.

Yes you can send a PR, that would be great.
I have little experience with npm, so If you can also test the package I would be very grateful :)
Thanks :+1:

Merged and updated :)
Thank you :+1:


Fix CanvasItem notification thread guard

Fixes https://github.com/godotengine/godot/issues/80412

Seems like simple fix, most other nodes like Node2D and Node3D also use ERR_THREAD_GUARD instead of ERR_MAIN_THREAD_GUARD in their notification callback. Looks like CanvasItem should also use ERR_THREAD_GUARD unless there is some specific reason why CanvasItem can't be used with subthread groups.

It looks to me like this was done like this on purpose, not as a typo. See 0a9f72d5a80c8957ef5172f546c9076089862cef where both Control and CanvasItem were made to use ERR_MAIN_THREAD_GUARD, with exceptions made explicit:
@@ -363,6 +379,7 @@ void CanvasItem::_window_visibility_changed() {
 }

 void CanvasItem::queue_redraw() {
+       ERR_THREAD_GUARD; // Calling from thread is safe.
        if (!is_inside_tree()) {
                return;
        }

So this should be discussed with @reduz.

Given that functions like Node::add_child() use ERR_THREAD_GUARD, it seems that the design involves the ability to deal with the scene tree from threaded processing. It should be fine as long as the nodes involved that need to update something belong in the same group.
Therefore, my strong guess is that ERR_THREAD_GUARD is be fine here. Still, I'd like to be very safe and hear @reduz here.

Maybe this is a problem of granularity. CanvasItem::_notification() deals with stuff meant for the main thread (like visibility) and other that can be used from threads (like scene tree management). The safe thing was to use the most restrictive thread guard, but a possible better approach would be to use different kinds of guards depending on the specific notification; e.g., use ERR_THREAD_GUARD for NOTIFICATION_ENTER/EXIT_TREE, and ERR_MAIN_THREAD_GUARD for NOTIFICATION_VISIBILITY_CHANGED.

@akien-mga @bitsawer This is intended, GUI code (control derived) can't use thread groups. I wrote and explained this in detail in the guidelines regarding thread groups (they should probably be made into documentation I guess).
@RandomShaper The guard you are mentioned in Node::add_child seems to be a mistake, the actual check is the line above. Same as in remove_child and a few others there. I think I added those before ERR_MAIN_THREAD_GUARD existed and forgot, so should probably be cleaned up.

Maybe the get_configuration_warning should be updated in Control to detect a thread group has been assigned and return an error.

I'm just making sure, but as Node2D inherits from CanvasItem, doesn't the current behavior basically mean that CanvasItem, Node2D or anything that inherits from them can't be used with thread groups? Currently, attaching a script with _process() causes a errors when attached to Node2D but not with Node3D if used with subthread groups.
I'm also wondering because Node2D has less strict _notification() thread guard (ERR_THREAD_GUARD) than its superclass CanvasItem (which uses ERR_MAIN_THREAD_GUARD), which feels weird. Control has ERR_MAIN_THREAD_GUARD which does make sense.

I am sorry, I misread the fix. This is perfect.

Are you sure that's the right type of guard for every possible notification? Quoting myself:

[...] a possible better approach would be to use different kinds of guards depending on the specific notification; e.g., use ERR_THREAD_GUARD for NOTIFICATION_ENTER/EXIT_TREE, and ERR_MAIN_THREAD_GUARD for NOTIFICATION_VISIBILITY_CHANGED.

Just to be sure.

Reply from production chat:
<reduz> what pedro suggests makes sense but should probably done more fine grained in another PR I think
<Akien> Well I think bitsawer could do it if there's consensus that it's relevant. If Pedro is right, this PR might introduce issues where e.g. NOTIFICATION_VISIBLE is no longer safe for Controls.

So your call guys :)

I think this PR is still worth merging, because currently using any 2D nodes with scripts using _process() is not really possible at all with thread groups (or at least easily), even if you know all threading rules. This PR at least makes it possible to do something until a better solution is implemented.
I also think that the best long-term solution will be to analyze each case and add more fine-grained access checks as already mentioned by RandomShaper. Ideally, the _notification() callbacks might not even need any access guard macros at the base level, each data access could in theory have its own check to make sure they have optimal granularity. Of course, this might be a pain to implement in practice, so some generalization might be unavoidable.

I quite don't see why not apply what I'm suggesting right in this PR and so have both the functionality and safety benefits. For me, that'd be the ground on top of which we could build something better. That said, I wouldn't oppose to it being merged, but I may do a follow-up PR with my take... 😀

Oh, I kind of missed that you were suggesting some additions to this PR, makes sense if you want to add something here. I think a more extensive follow up to add or tweak thread guards with some more thought is a good idea anyway, but I can add stuff here if you want to, feel free to hold my hand here.
I guess you for example suggested ERR_THREAD_GUARD for CanvasItem NOTIFICATION_ENTER/EXIT_TREE, but doesn't basically all scene tree related manipulation require ERR_MAIN_THREAD_GUARD? I'm not all that familiar with this system yet, so you might be better at figuring out the level of checks needed.

Scene tree manipulation methods (e.g., Node::add_child()) are guarded by ERR_THREAD_GUARD, so they are not meant to be used during threaded processing (by design; maybe technically "overcomable," but would surely make the implementation more difficult). The same holds for visibility manipulation. That's why I think this kind of guard should be used for the corresponding notifications, while one allowing threaded processing should be used in those notifications that can be received as a result of legal operations during threaded processing.
However, I've just realized there's more to it. Neither CanvasItem nor Node2D have non-read-only methods that can be used during threaded processing. As a result, we'd indeed put a ERR_THREAD_GUARD in their notification callback and call it a day. That would of course bring back the issue you found. The problem is that they can be getting notifications that are harmless during threaded processing since they are just ignoring them. Because of that, my current take is to use ERR_THREAD_GUARD on each case of the switch, but not in the scope of the function to let other notifications flow without interruption. If another class down in the hierarchy happens to handle one of those, it's up to that class to decide the safety level and therefore decorate the relevant case with the appropriate guard.

So what's the consensus? The merge window for this kind of "risky" change for 4.2 will soon close as we enter the beta phase and try to play it safe.

Summarizing my take:
My current take is to use ERR_THREAD_GUARD on each case of the switch, but not in the scope of the function, to let other notifications flow without interruption, plus moving to a per-switch scheme as well in any other existing notification callbacks using thread guards.
Do @bitsawer and @reduz agree?

Adding ERR_THREAD_GUARD to each CanvasItem::_notification() switch case and removing the current ERR_MAIN_THREAD_GUARD from the base function scope makes sense to me. We can probably fine tune them later if we need to.
Should I do the same with Node2D::_notification() or should we leave it alone for now?

I can go through all existing _notification() callbacks that have existing thread guards and do the same. Fortunately, there doesn't seem to be too many of them yet, looks its basically only CanvasItem, Node2D and Node3D that currently have these thread checks in their notification callback.

Oh, you've already done. You're my hero!
But I think the right kind of guard would be ERR_MAIN_THREAD_GUARD since all the scene tree management and visibility changing functionality is not meant to be dealt with from threads. NOTIFICATION_TRANSFORM_CHANGED is the only one that I believe warrants ERR_THREAD_GUARD.

Updated most guards to use ERR_MAIN_THREAD_GUARD and pushed changes, except for Node3D NOTIFICATION_TRANSFORM_CHANGED. I played a bit safe not to change the existing logic much, but I guess we might as well go all in. What do you think about CanvasItem NOTIFICATION_VISIBILITY_CHANGED, in theory ERR_THREAD_GUARD might also be enough for it?

I think it's perfect now.

What do you think about CanvasItem NOTIFICATION_VISIBILITY_CHANGED, in theory ERR_THREAD_GUARD might also be enough for it?

I think so. CanvasItem::set_visibility is guarded by ERR_MAIN_THREAD_GUARD, which hints that visibility is not meant to be dealt with during threaded processing. It would be hard to enable such a thing, and it's probably not worth it. After all, if you really need to touch some node's visibility during threaded processing you can always use call/set_thread_safe() for it to happen when it's safe afterwards.

Thanks!


docs: update data-config.md

Fix https://github.com/sehilyi/goscan/pull/90#discussion_r1165982500

that looks good, example file names could be be renamed to reflect the preferred extensions tsv/json.
Current documentation:
CNVs:

Drivers:

Thanks! Let me update the files and their names on the docs


Check for more required fields

PR Description
So for these handlers, I believe these fields should be required. As a rule of thumb, if envelope.get(FIELD) is called in the handler, FIELD should be required, right? For one instance, MessageHandler, all of the specific message handlers (NodesHandler, PingHandler, etc) seem to assume a valid session.

Will have to dig into this one more deeply. The risk is that we wind up silently ignoring an event which makes it very hard to debug what went wrong. Currently we'd throw an exception that would get logged so would see that something is wrong.
I really dislike the whole pipeline and filtering pattern in use though so need to really think through how this fits together.


The color did not display well when inactivate the terminal window

I improved the scheme solarized to support the terminal mode by referring to https://github.com/vim-airline/vim-airline-themes/pull/225 and https://github.com/vim-airline/vim-airline-themes/pull/235. After I fixed the issue, all works fine except that when inactivate the terminal window, the "bash name field" show a black line as shown in the actual screenshot below.
 It could be that you have the order of overwriting operations wrong or that you are copying a reference to a map somewhere. For reference you can have a look how I set up mine, because that one works: https://github.com/kmoschcau/nvim-config/blob/master/autoload/airline/themes/material.vim

my VIM colorscheme is changed by airline to default, but the main scheme keep, airline_theme is default.  I run :colorscheme, return default.....
When checking the init step, I find the airline load after colorscheme mycolor.vim
I have to recolorcheme  my color.
Plug manager: Plug-vim

my VIM colorscheme is changed by airline to default, but the main scheme keep, airline_theme is default.  I run :colorscheme, return default.....
When checking the init step, I find the airline load after colorscheme mycolor.vim
I have to recolorcheme  my color.
Plug manager: Plug-vim

I had a similar problem to yours while using the gruvbox colorscheme and managed to fix it by using the patch described in vim-airline/vim-airline-themes#180, maybe it will help you as well.

TL;DR: The thing with OP's patch is, airline does not use the added terminal_inactive entry of a theme's palette. The entries (modes) that matter are terminal, visual_modified, normal_modified, inactive and inactive_modified. Adding proper airline_term entry to these dictionaries could be a better solution.
Explained:
According to documentation in airline's default theme dark.vim, a vim buffer's mode is one of normal | insert | replace | visual | terminal | inactive. Airline maps these mode names to clusters of highlight groups (which are defined as items in dictionary palette), and applies corresponding set of colors in different buffer(window) modes.
For a vim terminal window, when the job running inside it receives user's keystrokes, airline recognizes the window as in terminal mode. Once Ctrl-W N is pressed and the window goes to Terminal-Normal mode (:h t_CTRL-W_N), airline treats it as in normal_modified mode - an override/derivative of normal, having its own key in palette. If we begin visual selecting, the terminal window is in visual_modified mode. And finally if the cursor moves to another buffer, the terminal window becomes inactive. There seems no "terminal_modified" mode in airline, so adding such a key in palette does NOT help.
And then comes the "bash name field", corresponding to the code line OP has commented out to make things work. The line sits in a function which "builds" the statusline section by section, and the line itself adds a section displaying the job's name, employing highlight group airline_term. Once commented out, this section is never created, thus not displaying the color defined by airline_term, no matter which mode the window is in. Note as OP's screenshot suggests, the job's name does not get displayed either.
So to fix the green-on-black display, the key is to get airline_term highlight group appropriately defined for every relevant mode in an airline theme's palette. Unfortunately, most airline themes have not yet been updated to cover this feature, which means airline_term's color is usually poorly defined and relies much on some "fallback" mechanism (see vim-airline-themes#224). E.g. the solarized theme has neither airline_term highlight group nor settings for terminal mode, resulting in OP's first screenshot (plus the normal-mode-like looking while one can "inserts" into terminal window).
The vimrc patch in vim-airline/vim-airline-themes#180 does work for most cases, yet on my computer the terminal window's inactive mode still differs from an ordinary buffer, refusing to turn grey...
Another straightforward way, apparently, is to manually add proper airline_term key to relevant modes' dictionaries. For solarized the patch below should make terminal window's statusline behavior almost identical to an ordinary buffer:
function! airline#themes#solarized#refresh()
  ...
  " Fix terminal window
  " For 'terminal' mode, derive from 'insert' mode
  let g:airline#themes#solarized#palette.terminal = airline#themes#generate_color_map(
        \ [s:I1[0].g, s:I1[1].g, s:I1[0].t, s:I1[1].t, s:I1[2]],
        \ [s:I2[0].g, s:I2[1].g, s:I2[0].t, s:I2[1].t, s:I2[2]],
        \ [s:I3[0].g, s:I3[1].g, s:I3[0].t, s:I3[1].t, s:I3[2]])
  " Add 'airline_term' highlight group
  "  Treat it as an equivalent of 'airline_c'
  let g:airline#themes#solarized#palette.terminal.airline_term =
      \ g:airline#themes#solarized#palette.terminal.airline_c
  " 'Terminal-Normal' as 'normal_modified'
  let g:airline#themes#solarized#palette.normal_modified.airline_term =
      \ g:airline#themes#solarized#palette.normal_modified.airline_c
  " 'Visual' as 'visual_modified'
  let g:airline#themes#solarized#palette.visual_modified.airline_term =
      \ g:airline#themes#solarized#palette.visual_modified.airline_c
  " 'inactive' is somewhat awkward, involving both 'inactive' & 'inactive_modified'
  let g:airline#themes#solarized#palette.inactive_modified.airline_term =
      \ g:airline#themes#solarized#palette.inactive_modified.airline_c
  let g:airline#themes#solarized#palette.inactive.airline_term =
      \ g:airline#themes#solarized#palette.inactive.airline_c
endfunction

One should be able to fix an arbitrary airline theme's terminal window statusline headache in a similar manner.


Fix TestS3Offload

Motivation
s3 mock service doesn't need credential. so we will fail to load credentials on s3 integration tests.
that is expected. However there is a regression when introducing GCS support in #2151:
https://github.com/apache/incubator-pulsar/pull/2151/files#diff-9160ffc9df7afcf42a8335f214393c6dR188
Changes
Don't throw exception when failed to load aws credentials

rerun java8 tests

rerun java8 tests


API server fail validate algorithmEnv  from: " fieldRef   - configMapKeyRef   - resourceFieldRef   - secretKeyRef"

HKube micro-service
api server
Describe the bug
API server fail validate algorithmEnv  from: " fieldRef   - configMapKeyRef   - resourceFieldRef   - secretKeyRef"
thos option should be added to the swagger
getting error
configMapKeyRef is invalid, only fieldRef,configMapKeyRef,resourceFieldRef,secretKeyRef
{
    "name": "env-test",
    "cpu": 1,
    "gpu": 0,
    "mem": "256Mi",
    "minHotWorkers": 0,
    "algorithmImage": "hkube/algorunner",
    "type": "Image",
    "options": {
        "debug": false,
        "pending": false
    },
    "algorithmEnv": {
        "FFF": {
            "configMapKeyRef": {
                "name": ",m-cm",
                "key": "cm_env"
            }
        }
    }
}
**Expected behavior**
A clear and concise description of what you expected to happen.

**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Additional context**
Add any other context about the problem here.

tested on
systemVersion: "v2.0.55",
fullSystemVersion: "v2.0.55-1610028130079",

tested on
systemVersion: "v2.0.55",
fullSystemVersion: "v2.0.55-1610028130079",


Shutil same file error

Describe the bug
Shutil same file error
To Reproduce
Log input audio at beginning of every epoch with
wandb.log({"Input audio": [wandb.Audio(inputs[0].cpu().squeeze().numpy(), sample_rate=train_dataset.get_audio_rate())]})

Expected behavior
Logged without issues.
Screenshots

Operating System

OS: Ubuntu 20.04
Browser chrome
Wandb Version 0.10.11

Additional context
Audio is displayed properly on wandb. However, code crashes with this error after 2 epochs. Replicated multiple times on different machines. Handling the exception during the log does not seem to work for some reason.

Hey @miraodasilva we'll need to get a fix in for the underlying issue, but until then you can fix this by not logging the same audio multiple times.  You could log these inputs once in their own media panel and then if you want to associate predictions or other media to them you can log a wandb.Table with different columns at each step.

Hey @miraodasilva we are digging into this now. Two questions:

Do you mind logging your audio outside of the array? (ie. wandb.log({"Input Audio": wandb.Audio(...)}))
Do you mind sharing some context around this call? The error seems to imply that the same audio file is being logged to the same key for the same training step. Are you perhaps making this same call a few lines prior with commit=False?

Hi! Sure, no problem. However I am only logging once per epoch and I am shuffling the dataset so it should not be the same file. I'm not sure what's going on honestly. I will get back to you if I have any more info.

Hey @miraodasilva we are digging into this now. Two questions:

Do you mind logging your audio outside of the array? (ie. wandb.log({"Input Audio": wandb.Audio(...)}))
Do you mind sharing some context around this call? The error seems to imply that the same audio file is being logged to the same key for the same training step. Are you perhaps making this same call a few lines prior with commit=False?

Actually, I am investigating this as we speak. The last training step should have commit=False to allow for further validation logs within the same step. If I am accidentally doing this in the penultimate step rather than the last, that could certainly be an issue. The concrete line was
wandb.log(logs, commit=True if i < (len(loader) - 1) else False)

This seems like a possible lead (I am not sure how your loop is constructed, but the condition shared may be suspect). By chance, does your logs object contain an Input Audio key?

It does, yes. The concrete line is
logs.update({"Input audio": [wandb.Audio(inputs[0].cpu().squeeze().numpy(), sample_rate=train_dataset.get_audio_rate())]})

Ok, just so we are on the same page, this is how I understand the logic to be:
for i in range(len(loader)):
   logs = {}
   # ... other code
   logs.update({"Input audio": [wandb.Audio(...)]})
   # ... other code
   wandb.log(logs, commit=True if i < (len(loader) - 1) else False)

# then some additional logging?

If you don't mind sharing explicitly the key code blocks of interest, I think we can get to the bottom of this.

I think a rather large problem is the fact that I was accidentally initializing the logs to an empty dict outside the training loop, so logs were accumulated for the entire loop.

So basically the logs = {} was above the first line, rather than on the second line, as you show.

That would mean I am accumulating a huge log throughout the entire epoch. However it would contain only one audio given that I add the audio to the log only on iteration 0.

Do you mind sharing your code snippet? I have a few hypotheses and it might be easier to discuss your concrete implementation.

Unfortunately I can't, as this is a project still in development and that would imply releasing the main training loop.

One thing I should mention, below I am Saving a video with the same audio:
# Visualize inputs
                    logs.update(
                        {
                            "Input audio": [
                                wandb.Audio(
                                    inputs[0].cpu().squeeze().numpy(), sample_rate=train_dataset.get_audio_rate()
                                )
                            ]
                        }
                    )
                    print("logged audio.")
                    sample_path = save_av_sample(
                        vinputs[0],
                        train_dataset.get_video_rate(),
                        audio=inputs[0],
                        audio_rate=train_dataset.get_audio_rate(),
                    )
                    logs.update({"Input video": wandb.Video(sample_path, fps=25, format="mp4")})

Even some basic pseudo code with random data that reproduces this issue would be really helpful.

Ok here is a tear down of what's going on. The loop is something along these lines:
for e in range(epochs):
    # Source of the insanity is the line below. It should be inside the inner loop, not this one
    logs = {}
    for i in range(len(loader)):
        # ... other code
        logs.update({"Input audio": [wandb.Audio(...)],"Input video": [wandb.Video(...)]})
        # ... other code
        wandb.log(logs, commit=True if i < (len(loader) - 1) else False)

    #commit is true here, as default, so epoch AND last iter metrics are logged.
    wand.log("Epoch": e)

Consequence:
Same audio and video are logged every single iteration. By printing the log items every time this is clear

So it is logging the same video and audio over and over and over again. However, each one is recorded in the files on wandb. The level of insanity became clear when I checked the files and saw that there were 10k+ audio files under media/audio. I'm not sure if the same is happening with video, but the video folder does not appear (maybe because too many files cant be displayed).
Source of error
The run crashes arbitrarily after around 1 or 2 epochs. Clearly it doesn't mind posting the exact same file over and over again. I would suggest that the bug has to do with logging too many audio files, to a point where the name/id is repeated. Frankly I'm not sure. Sorry about the lack of initiative but I can't really afford (in terms of time) to devote much more attention to the issue. By moving log = {}, the insanity seems to be solved.

BTW I checked and the commit=False is only true on the last iteration, so this is working as intended.

Thank you for this very detailed breakdown - we appreciate your time helping to characterize the bug. It sounds like you have a workaround by moving the logs initialization call? Is that correct? We will continue to look into this and get back to you with any details.

Yep, insanity in terms of logging thousands of file is fixed by moving that line. Which makes sense.
To reproduce this I would suggest trying to log the same audio and video file 100,000 times or something like that, and see if it gives off a similar bug.
No worries btw, thanks so much for the awesome tool, I think it's really great. And ofc thanks for your time.

Awesome, glad to hear you can continue with your work! Thanks for the kind feedback - we are always here to help. Please let us know if you run into further issues.


Volatility indicator returns 0 values

Describe the bug
I tried to use the volatility indicator but it just returns 0 values. I checked my code if the close column has no value but it has and it's in datatype float so I don't understand why the indicator is giving me 0.0 values.
To Reproduce
I implemented it like this
new_d['volatility'] = ta.volatility(new_d['close'], tf='years')
Expected behavior
this is what I expect to return
https://imgur.com/lopMKFp
Screenshots
If applicable, add screenshots to help explain your problem.
But All I got was this
https://imgur.com/hnAuyrT
https://imgur.com/OBeBLSL
https://imgur.com/ghGb3ea
Thanks in advance for addressing my issue

Hello @GoldenGeese,
Thanks for the info. Unfortunately, I am unable to make headway on this Issue in the near term. Hopefully you or someone with a vested interest can contribute to improve the Performance Metric components until I can get back to expanding the features of the library.
Kind Regards,
KJ


useScript breaking when called in the browser

I checked that Deno is not called on the client side.

Some things to note here:

$fresh/runtime probably should not be imported in this file, since you can use it independently from the fresh setup
useScript should probably not be used in the browser at all; it is importing terser

useScript must fail if run in client side. Its a server side only thing to do.
Maybe you can give a warning and don't run it on client side instead.


ContractClass types responses

-rel #634
getClassAt & getClassByHash returns a ContractClassResponse
abi property is a json object
sierra_program is always uncompressed
program (cairo0) is always compressed (currently rpc returns a compressed program while sequencer returns it uncompressed)
-remove unnecessary compress/decompress on sierra_program
-modified cairo1.tests to run on rpc & sequencer
-great performance improve for formatSpaces function

@notV4l now we can rebase to develop :)


SAM transform is only applied by cfn-lint if not declared as a list

cfn-lint version: (cfn-lint --version)
0.22.0
Description of issue.
cfn-lint works as expected if the SAM transform is specified like this:
Transform: AWS::Serverless-2016-10-31

but if multiple transforms are needed (or just declared as a list) like:
Transform:
  - AWS::Serverless-2016-10-31

the template is not transformed before the validation occurs and that causes an assortment of false positives.
Please provide as much information as possible:
Working Example:
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Globals:
  Function:
    Timeout: 3
    Runtime: nodejs8.10
    Environment:
      Variables:
        VARIABLE: VALUE

Resources:
  Function:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/
      Handler: index.handler
      Events:
        API:
          Type: Api
          Properties:
            Path: /
            Method: GET

Example that yells about E1001 Top level item Globals isn't valid:
AWSTemplateFormatVersion: '2010-09-09'
Transform:
  - AWS::Serverless-2016-10-31
Globals:
  Function:
    Timeout: 3
    Runtime: nodejs8.10
    Environment:
      Variables:
        VARIABLE: VALUE

Resources:
  Function:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/
      Handler: index.handler
      Events:
        API:
          Type: Api
          Properties:
            Path: /
            Method: GET

Cfn-lint uses the CloudFormation Resource Specifications as the base to do validation. These files are included as part of the application version. Please update to the latest version of cfn-lint or update the spec files manually (cfn-lint -u)

Thanks for submitting this.  Working on it.

@kddejong
Was the decision to not do the SAM transform at all if multiple transforms were specified just because it didn't seem like the template would be in a knowable state without mystery transforms  being applied?
The specific use case i had was a transform running before after SAM to paper over a limitation in it ( https://github.com/awslabs/serverless-application-model/issues/717 ):
AWSTemplateFormatVersion: 2010-09-09
Transform:
  - PreSamTransformV1
  - AWS::Serverless-2016-10-31
  - PatchSamTransformV1

So for that specifically it would definitely work better to always run the SAM transform if there and i imagine that would have a better chance of working in the long run than the almost guaranteed errors that would be thrown by not doing the transform.
Are there plans to allow for custom transform hooks?

Yea you are probably right there.  You want to submit a PR?

Sure! I'll try to churn that out this afternoon!


[NFC] Adjust SILFunctionType layout to use TrailingObjects methods.

This makes the implementation cleaner, as well as fixes a bug in the numTrailingObjects<SILResultInfo>() method (which was unused).
(Review from any one person will do.)

@swift-ci please test

Build failed
Swift Test Linux Platform
Git Sha - ed93d85058b68f79e1d3d3e89e668d06f22625b6

Failure seems to be due to another commit which got bundled in for testing.
@swift-ci please clean test Linux platform

Accidentally wrote numTrailingObjects(CanType) instead of numTrailingObjects(OverloadToken<CanType>). Oops, fixed.
@swift-ci please test and merge

@swift-ci please test

Build failed
Swift Test OS X Platform
Git Sha - ed93d85058b68f79e1d3d3e89e668d06f22625b6

Build failed
Swift Test Linux Platform
Git Sha - ed93d85058b68f79e1d3d3e89e668d06f22625b6

@swift-ci please test macOS platform

Build failed
Swift Test OS X Platform
Git Sha - e52b7226f7856d3948af15ad60235fbe6ab01881

@swift-ci please test and merge

@swift-ci please test

Build failed
Swift Test Linux Platform
Git Sha - e52b7226f7856d3948af15ad60235fbe6ab01881

Build failed
Swift Test OS X Platform
Git Sha - e52b7226f7856d3948af15ad60235fbe6ab01881


Cards for orgs should not refer to money raised

An organization is not a profile type for fundraising. When you click through on it, there's nothing visible about this money being raised. Why is it showing money raised?

To be more specific, this is about the card we show in the search: https://github.com/opencollective/opencollective-frontend/blob/3d3a732da606bf6e7b6f3de3659fd29fc4435ebd/components/search-page/SearchCollectiveCard.js. Using totalAmountReceived for an organization that can't receive contribution doesn't make much sense; for them, we should try to display the total amount contributed there, but not the amount received.
Profile: https://opencollective.com/center-for-open-science
Search page: https://opencollective.com/search?q=center+open+science&type=ORGANIZATION

To be more specific, this is about the card we show in the search: https://github.com/opencollective/opencollective-frontend/blob/3d3a732da606bf6e7b6f3de3659fd29fc4435ebd/components/search-page/SearchCollectiveCard.js. Using totalAmountReceived for an organization that can't receive contribution doesn't make much sense; for them, we should try to display the total amount contributed there, but not the amount received.
Profile: https://opencollective.com/center-for-open-science Search page: https://opencollective.com/search?q=center+open+science&type=ORGANIZATION

@Betree : When you mention total amount contributed, do you mean the total money spent? I've opened a PR with a potential fix for this let me know how it looks. I used the phrase Money Spent since the stat used is in sync with that wording.


Fix sdba_encode_cf bug

Pull Request Checklist:

[ ] This PR addresses an already opened issue (for bug fixes / features)

This PR fixes #xyz

[x] Tests for the changes have been added (for bug fixes / features)

[ ] (If applicable) Documentation has been added / updated (for bug fixes / features)

[x] CHANGES.rst has been updated (with summary of main changes)

[x] Link to issue (:issue:number) and pull request (:pull:number) has been added

What kind of change does this PR introduce?
In xarray#7494 the _contains_cftime_datetimes function was restricted to Variable objects. We were passing a DataArray to xr.conventions.encode_cf_variable , which was passing it (assuming a variable) to that function. Since the PR, this raises an error.
In a way, this is my error because the encode_cf_variable clearly documents that its input should be a Variable, although there were not annotations at the time I implemented that line.
Moreover, we were not testing this code... It depended on the sdba_encode_cf option being activated, but no test was activating it.
This PR passes Variable instances were it should and enables the option in one of the test where it makes sense.
Does this PR introduce a breaking change?
No.
Other information:
Fixes xscen#175.

Merge when ready!


[Query] Max Instruction Size for riscv64

Hello, I have a query. I am a developer for Tizen SDK for Visual Studio. For supporting debugging of riscv64 architecture in VS, we are modifying miengine project. While adding code, we found a variable named MaxInstructionSize for each architecture in miengine code.
Example:
case TargetArchitecture.ARM64:
MaxInstructionSize = 8;
Is64BitArch = true;
break;
What might be the MaxInstructionSize for riscv64?

Disclaimer: I have never looked at RISC-V instruction set before this question, so I am by no means an expert on RISC-V.
Looking at this specification I think it isn't an entirely answerable question at an architectural level since RISC-V provides for arbitrary sized extensions. That said, the purpose of the MaxInstructionSize is just to aid in disassembly. So, if you know the largest instruction that the underlying debugger that MI Engine is talking to can actually disassemble, then you should use that value.


Add Traefik 2 labels + move env files to a specific directory + remove GeoIP directory

Ok so I've made a lot of modifications via this pull-request, I let you judge what should be kept or not.
First, I deleted the "geoip" directory, it doesn't need to be in the project. Then I added the labels to make it work with Traefik 2, this assumes that the user already has a working and well-configured container of Traefik. However, the labels are commented, so no impact for a deployment without Traefik.
I have added the GeoIP volume configuration in the original docker-composer, it is an interesting feature, easy to set up. No error if the container is misconfigured, you better use it.
Finally to avoid confusion, I used the same extension in ".env" for the geoip-conf.env and plausible-conf.env files


Fix creating index with custum Prefixes missing colon (#329)

This would be a break - and the issue here is mostly cosmetic as far as I can tell, so closing this out - thanks for opening though!


PEP 458: Rename Role and Key Terms According to Issue #13

Addresses Issue #13.

Update the figures and text to use the new terms.  Online / offline instances have been renamed to on-pypi / off-pypi, and claimed / unclaimed to pypi-signed / developer-signed.

Expanded the section that discusses the lazy bin walk scheme.

@lvigdor
Updates addressing your comments were made here: https://github.com/vladimir-v-diaz/interoperability-peps/commit/474c2f75d9c64b95d12f1c3f72de9d4bc7c1b509
Thanks for reviewing this PR!


Add child system (initial upload)

OK let’s try again.
This pull request addresses #6568 in MNE.

Added our 125-channel child system to kit/constants.py
layout.py returns ‘KIT-125’
Added KIT-125.lout file for sensor-level plotting

FAO @paulsowman

thx @neurofractal


TypeScript & SubmitSelfPost

I ran into a bug with submitSelfPost for Subreddit. The documentation states to use submitSelfpost (lowercase P) however TypeScript autocorrects to SubmitSelfPost (with uppercase P).

Using submitSelfPost returns an error.
Using submitSelfpost does not return an error.

Thanks, I've gone ahead and fixed that. Until we get a new release simply add a TypeScript ignore comment on it


Update user's group

Hey,
Didnt find a way to update user's group (to add to a group)
$user = new User; $user->group = 'AAA'; $user->save();
Not working. Please advice?
didnt find it in docs and in the code.
Thanks

$user = new User;
$user->group = UserGroup::whereCode('AAA')->first();
$user->save();

We will look at improving the related models docs to make this API more obvious. Thanks

$user->group = UserGroup::whereCode('buyer')->first();   $user->save();
"SQLSTATE[42S22]: Column not found: 1054 Unknown column 'group' in 'field list' (SQL: update userssetgroup= {"id":3,"name":"Buyer","code":"buyer","description":"Buyers group","created_at":"2016-11-20 18:08:51","updated_at":"2016-11-20 18:09:01"},updated_at= 2016-11-25 14:32:42 whereid= 42)" on line 662 of /Applications/MAMP/htdocs/fidgycube.co/vendor/laravel/framework/src/Illuminate/Database/Connection.php
Getting this (

Relating models is probably more like this:
$user->group()->associate( UserGroup::whereCode('buyer')->first() );

The relationship may be called "groups" instead of "group".
$user->groups = UserGroup::whereCode('AAA')->first();

Samuel,
i changed to
$user->groups = UserGroup::whereCode('AAA')->first();
and now getting
"Type error: Argument 1 passed to Illuminate\Database\Grammar::parameterize() must be of the type array, integer given, called in /Applications/MAMP/htdocs/fidgycube.co/vendor/laravel/framework/src/Illuminate/Database/Query/Grammars/Grammar.php on line 340" on line 118 of /Applications/MAMP/htdocs/fidgycube.co/vendor/laravel/framework/src/Illuminate/Database/Grammar.php

@Eoler
Not working. Gives no error, but not saving into groups table

Are you sure a group with code AAA exists? This will protect you if it doesn't:
$user->groups = UserGroup::whereCode('AAA')->first() ?: null;

hey @daftspunk
Mine is $user->group()->associate( UserGroup::whereCode('buyer')->first() );
bayer and it exists.

These all should work:
$group = UserGroup::whereCode('buyer')->first();

$user->group()->associate($group);

$user->group()->add($group);

$user->group = $group;

$user->group = $group->id;

"Call to undefined method October\Rain\Database\QueryBuilder::associate()" on line 2123 of /Applications/MAMP/htdocs/fidgycube.co/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php

$group = UserGroup::whereCode('buyer')->get();
gives me ID
something wrong with relation as far as i see

@bbrandtlv, sorry to say we are stabbing in the dark here. Perhaps you can share your code with us so we can give some more educated advice.

(( sad!
Ok.
`<?php namespace Bbrand\Shop\Components;
use Jiri\JKShop\Components\Basket;
use Session;
use Redirect;
use Mail;
use \RainLab\User\Models\User;
use \RainLab\User\Models\UserGroup;
use Auth;
use Database;
class Basketextend extends Basket
{
public function componentDetails()
{
return [
'name'        => 'basketextend Component',
'description' => 'No description provided yet...'
];
}
public function defineProperties()
{
    $parentProps = parent::defineProperties();
    $properties = array_merge(
        $parentProps,
            [
                'myWrapper' => [
                'title'       => 'Wrapper for content',
                'description' => 'Wrapper for rendered content',
                'default'     => '.popup-content',
                'type'        => 'string'
                ],
                'paypal_business' => [
                    'title'       => 'Paypal Business',
                    'default'     => '.popup-content',
                    'description' => 'Your bussines account',
                    'type'        => 'string'
                ],
                'paypal_return_url' => [
                    'title'       => 'Return URL',
                    'description' => 'Use full url where will be your customer redirect after paid "http://www.example.com/paypal-success"',
                    'type'        => 'string'
                ],
                'paypal_use_sandbox' => [
                    'title'       => 'Use sandbox',
                    'description' => 'For testing only!',
                    'type'        => 'checkbox'
                ],
                'slug' => [
                    'title'       => 'URL Order ID',
                    'description' => 'URL Order ID',
                    'default'     => '{{ :slug }}',
                    'type'        => 'string'
                ],
                'stripe_publishable_key' => [
                    'title'       => 'Publishable Key',
                    'description' => 'For testing you can use: "xxxx"',
                    'type'        => 'string'
                ],
                'stripe_secret_key' => [
                    'title'       => 'Secret Key',
                    'description' => 'For testing you can use: "xxx"',
                    'type'        => 'string'
                ],
                'stripe_return_url' => [
                    'title'       => 'Success URL',
                    'description' => 'Use full url where will be your customer redirect after paid "http://www.example.com/paypal-success"',
                    'type'        => 'string'
                ],

            ]
        );

    return is_array($properties) ? $properties : $parentProps;
}

public function onCheckEmail($email)
{
    return Auth::findUserByLogin($email) ? 1 : 0;
}

public function onBasket(){
    $data = [];
    $data["basket"] = $this->getSessionBasket();
    $data["jkshopSetting"] = \Jiri\JKShop\Models\Settings::instance();

    return [
        $this->property('myWrapper') => $this->renderPartial('::basket-0', $data)
    ];
}

public function onMyCompleteOrder() {

    $jkshopSetting = \Jiri\JKShop\Models\Settings::instance();

    // create order
    $basket = $this->getSessionBasket();
    $order = new \Jiri\JKShop\Models\Order();
    $order->createFromBasket($basket);
    $order->save();
    // $this->page['slug'] = $order->id;
    $order = \Jiri\JKShop\Models\Order::find($order->id);
    $this->page["order"] = $order;

    // paypal details
    $this->page["paypal_business"] = $this->property('paypal_business');
    $this->page["paypal_currency_code"] = $order->paymentGateway->gateway_currency;
    $this->page["paypal_return_url"] = $this->property('paypal_return_url');
    $this->page["paypal_use_sandbox"] = $this->property('paypal_use_sandbox');
    $this->page["jkshopSetting"] = $jkshopSetting;

    //<EMAIL_ADDRESS>'acme.blog::mail.message', $params);

    if ( !($this->onCheckEmail($order->contact_email)) ){
        $user = new User;
        $user = Auth::register([
            'name' => $order->ds_first_name,
            'email' => $order->contact_email,
            'password' => 'changeme',
            'password_confirmation' => 'changeme',
            'username' => $order->contact_email,
            'jkshop_ds_first_name' => $order->ds_first_name,
            'jkshop_ds_last_name' => $order->ds_last_name,
            'jkshop_ds_address' => $order->ds_address,
            'jkshop_ds_address_2' => $order->ds_address_2,
            'jkshop_ds_postcode' => $order->ds_postcode,
            'jkshop_ds_city' => $order->ds_city,
            'jkshop_ds_county' => $order->ds_county,

            'jkshop_is_first_name' => ($order->is_first_name) ? : $order->ds_first_name,
            'jkshop_is_last_name' => ($order->is_last_name) ?  : $order->ds_last_name,
            'jkshop_is_address' => $order->is_address,
            'jkshop_is_address_2' => $order->is_address_2,
            'jkshop_is_postcode' => $order->is_postcode,
            'jkshop_is_city' => $order->is_city,
            'jkshop_is_county' => $order->is_county,
            'jkshop_contact_email' => $order->contact_email,
            'jkshop_contact_phone' => $order->contact_phone,
        ]);

        $group = UserGroup::whereCode('buyer')->get();
        // $user->associate($group);
        // $user->group()->add($group);
        $user->save();

    }

    $data = ['name' => $order->ds_first_name .' '. $order->ds_last_name, 'email' => $order->contact_email, 'orderid' => $order->id];

    Mail::send('bbrand.shop::mail.neworder', $data, function($message) use ($data){

        $message->bcc( $data['email'], 'Alexei Kositsin' );
        $message->to( $data['email'], $data['name'] );
        $message->from<EMAIL_ADDRESS>'FidgyCube.co' );
        $message->subject( 'FidgyCube.co order' );

    });

    // clear basket
    Session::forget('jkshop-basket');

    if ($order->payment_gateway_id == 1)
    return [
        $this->property('myWrapper') => $this->renderPartial('::basket-4-thanks-paypal')
    ];
    else if
    ($order->payment_gateway_id == 2){
        $this->page["stripe_publishable_key"] = $this->property('stripe_publishable_key');
        $this->page["stripe_secret_key"] = $this->property('stripe_secret_key');
        $this->page["stripe_currency_code"] = $order->paymentGateway->gateway_currency;
        $this->page["stripe_return_url"] = $this->property('stripe_return_url');
        return [
            $this->property('myWrapper') => $this->renderPartial('::custom-strupe-checkout')
        ];
    }
}

 public function onSubmitPay() {
    //  return 'fuck';

    $stripe = [
      'secret_key'      => $this->property('stripe_secret_key'),
      'publishable_key' => $this->property('stripe_publishable_key')
    ];
    \Stripe\Stripe::setApiKey($stripe['secret_key']);

    // save IN to events
    $eventLog = new \System\Models\EventLog();
    $data = array();
    $data["name"] = "Stripe Response";
    $data["post"] = post();
    $eventLog->message = json_encode($data);
    $eventLog->save();

    $order_id = post('order_id');
    // get order
    $order = \Jiri\JKShop\Models\Order::find( $order_id );

    if ($order != null) {
        $token  = post('stripeToken');
        $customer = \Stripe\Customer::create(array(
            'email' => $order->contact_email,
            'card'  => $token
        ));

        $charge = \Stripe\Charge::create(array(
            'customer' => $customer->id,
            'amount'   => ($order->total_price + $order->shipping_price)*100,
            'currency' => $order->paymentGateway->gateway_currency
        ));

        // check response
        $order->orderstatus = $order->paymentGateway->orderStatusAfter;
        $order->paid_date = \Carbon\Carbon::now();
        $order->paid_detail = $charge;
        $order->save();

        $data = ['name' => $order->ds_first_name . ' ' . $order->ds_last_name , 'email' => $order->contact_email, 'orderid' => $order->id];

        Mail::send('bbrand.shop::mail.payment', $data, function($message) use ($data) {

<EMAIL_ADDRESS>'Alexei Kositsin');
            $message->to( $data['email'], $data['name'] );
<EMAIL_ADDRESS>'FidgyCube.co');
            $message->subject('FidgyCube.co order');

        });

        return 'dsd';
        // return Redirect::to($this->property('stripe_return_url'));   //Redirect::to("/");
    }

}

}
`
using JIRI shop plugin.

Ok, I see the issue:
This code returns a COLLECTION (an array of groups):
// Do not use get()
$group = UserGroup::whereCode('buyer')->get();

This code should be used to return a SINGLE group:
// Use first()
$group = UserGroup::whereCode('buyer')->first();

The relationship name is not called GROUP:
// Do not use group()
$user->group()->associate(...);

This code should reference GROUPS:
$user->group()->associate(...);

As before, here is the complete working code
$group = UserGroup::whereCode('buyer')->first();

$user->group()->associate($group);

$user->group()->add($group);

$user->group = $group;

$user->group = $group->id;

( no
code still not working.
you wrote that I should reference to groups (but wrote group)
I changed it to $user->groups()->associate($group);
and now it cannot find associate() method.
I know that I took much of your time, but i cannot find a way to make this working.
i can go hardcore and simply update user_groups table with ids, but i'm sure you didnt ment for this.
ill try to dig more... hopefully will find it.
Thanks

@daftspunk
Hey
This one is working:
`            $group = array(UserGroup::whereCode('buyer')->first());
        $user->groups()->attach($group[0]->id);

        $user->save($group);`

But i'm dont like this solution. (
Hope this will help to somebody.

Thanks, we have removed attach() and associate() from the documentation to prevent this kind of confusion. In all cases add() should be used instead.

!!!!!!! FINALLY!
`$user->groups()->add(UserGroup::whereCode('buyer')->first());
        $user->save($group);
        `

This one is good!!!!
Thanks again!


Component paths being duplicated and containing "/.." in detected components

We noticed in some pipelines running CG that the Component Detection task was sending us weird paths for components that contained /../../.. either in the start of the path or in the middle, when the paths did not actually contain the "/.." in the repositories.
These paths were also duplicates of other paths detected for the component, with the only difference being the additional "/.." in the path.
For example:
--- Component: ---
github.com/ghodss/yaml v1.0.0 - Go
--- Found at: ---
/s/src/1/2/3
/../../../../someotherpath
/../../../../s/src/1/2/3
/someotherpath

AB#2109910

Pending follow up from linwen on build which repros this issue.

@linwen-h I was unable to reproduce the issue locally using the repository you mentioned - I was only seeing the non-relative paths (without ../).
I pushed a potential fix to cobya/RelativeDupe if you have a local repro.


Failure to create existing directory

We currently observe an issue with setup-foam
Download packages to cache
  mkdir: cannot create directory ‘.apt-install/packages’: File exists
  Error: Process completed with exit code 1.

See
https://github.com/exasim-project/NeoFOAM/actions/runs/11880294674/job/33103383610?pr=81
Do you have an idea what the issue is?

@greole Thanks for the report! I've transferred the issue here as I believe this package is the culprit. I'll be releasing a fix ASAP.

The issue should be fixed by #42 here, which is already released. Let me know if that doesn't do it.


Makefile Arm SSE fix

Currently the Makefile for librealsense does not work on the Jetson TK1. This patch checks to see if SSE is available on the CPU before enabling the -msse flags. The TK1 is an ARM processor, which does not have SSE instructions, and the compiler chokes when it sees a SSE flag enabled.

Thanks a lot Jim (@jetsonhacks) ! I wanted to fix that for the longest time but didn't have access to a TK1. I will look at the request later.
Also, could you please confirm that the latest version of this fork works on TK1? I updated to the original repo a couple of days ago and somebody told me that it stopped working for them on the Odroid board. It still works on my TX1.
Thanks again!

Hi Mehran,
In the request, I tried to look to see if SSE is available on the
processor instead of examining the architecture.
The latest fork does not work with the TK1, the RGB stream is black. I
haven't been able to figure out why yet. There could be a conflict with
the image patch, but I haven't explored enough to know.
I have  recompiled a TK1 Linux kernel with the patches in the
librealsense library on the TK1 on an older version, V4L2 seems to work ok.
I've been playing with workarounds to get the camera running without the
plug/replug between program invocations:
https://github.com/jetsonhacks/setupR200-Jetson
The issue appears to be that the video streams halt after being used the
first time. The workaround tries to clear the halts before use. A call
to r200JetsonSetup() or videoStreamsClearHalts() at the beginning of a
program should get everything lined up and ready to go.
This generally clears the -EPIPE error that I'm sure you've seen.
However it doesn't appear to be 100% effective, sometimes the video
stalls on the streams and the streams timeout. Usually running the
program a few times will clear this and everything will work normally
again.  The camera does not appear to shut down correctly when a program
is terminated (at the very least the streams stall and the red light
stays on). I don't know where this occurs in the usb stack, on the
camera, drivers, or kernel. I've spent way too long looking at it, I
think to investigate further takes a USB hardware analyzer and a better
debugging setup than I have.
I also converted the Qt Creator files over to use the UVC backend:
https://github.com/jetsonhacks/R200-UVC-Examples
In the examples folder there is sample which uses the workaround
mentioned above. Eventually I'm hoping to consolidate all of this.
Looking through the changes to your code base, I think one thing to
consider is submitting a pull request to the Intel repository with your
SSE workaround patches. That way you don't have to end up maintaining
the entire mess that they're creating there.
Best,
Jim
On 6/2/16 11:09 AM, Mehran Maghoumi wrote:

Thanks a lot Jim (@jetsonhacks https://github.com/jetsonhacks) ! I
wanted to fix that for the longest time but didn't have access to a
TK1. I will look at the request later.
Also, could you please confirm that the latest version of this fork
works on TK1? I updated to the original repo a couple of days ago and
somebody told me that it stopped working for them on the Odroid board.
It still works on my TX1.
Thanks again!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Maghoumi/librealsense/pull/2#issuecomment-223374474,
or mute the thread
https://github.com/notifications/unsubscribe/AIDJzeUTMg9aH0VK6usxqVWKOgt8utHLks5qHxxegaJpZM4Is03r.

Two more points of information:

There have been changes to the GLFW library recently that break everything on the Jetson TK1. The symptom is that the app is started, and a graphics window does not open, sometimes with the machine going into a hard lock. The latest version that the Jetson TK1 works on is glfw 3.1.2, the current revision appears to be 3.2. It could be that other machines have this issue too.

The script that loads the glfw library in the librealsense scripts folder loads the 'latest' tag, which equates to 3.2 (or greater going forward).

Reverting the last change to src/image.cpp (removing the patch for the UDOO fix) fixes the black RGB stream on the Jetson TK1. There is also a conflict marked against the Intel repository master in that file.

@jetsonhacks Could you link to the repository which is worked! I received a black image as well.

OK thanks for the responses @jetsonhacks, I will look into this soon.
@AutoModelCar You could revert to a commit before I updated everything to reflect the latest changes in the original repository.

I am working on SSE2NEON, I could compile librealsense using neon, and RS_FORMAT_Y8 works good, but RS_FORMAT_RGB8 still does not work correctly.
https://github.com/ZahraBoroujeni/librealsense

OK, I've been able to reproduce the problem on my Jetson TX1. I'm now getting the hard freezes. I am currently looking into it and it may take a couple of days. I apologize for any inconveniences.
@jetsonhacks I will incorporate this pull request after I've fixed the problem with the sensor not working at all.

Looking through the changes to your code base, I think one thing to consider is submitting a pull request to the Intel repository with your SSE workaround patches. That way you don't have to end up maintaining the entire code base that they're creating there.
Unfortunately, that is not possible currently as I'm 99% sure the PR will get rejected due to the ARM support not being robust at all. I will plan to send them a PR once I've found a reliable ARM NEON implementation.

@ZahraBoroujeni That's great news! I will look into your repository when I find some time on my hands.

I pushed a new version and incorporated @jetsonhacks fixes.
@ZahraBoroujeni @jetsonhacks @AutoModelCar Let me know if the latest version fixes your issues.

I can confirm that we can see the RGB/IR/Depth images now.
My Platform and situation:

Odroid xu4
patched  ubuntu 14.04 (add new uvcvideo.ko)
sr300
make librealsense using "make BACKEND=LIBUVC && make install"
30 Hz
ROS indigo

Situation: The second run works. I do not need to plug/unplug the camera, I need to run the program twice!
and Always the error below occurs after first run (odd runs):
[ERROR] [1356999361.466719947]: RealSense Camera - Error calling rs_create_context ( api_version:4 ):
libusb_bulk_transfer(...) returned LIBUSB_ERROR_TIMEOUT

[FATAL] [1356999361.635365197]: Service call failed!
[SR300_nodelet_manager-3] process has died [pid 14572, exit code 1, cmd /opt/ros/indigo/lib/nodelet/nodelet manager __name:=SR300_nodelet_manager __log:=/root/.ros/log/39a0b54c-53a6-11e2-9cb6-7cdd908e0595/SR300_nodelet_manager-3.log].
log file: /root/.ros/log/39a0b54c-53a6-11e2-9cb6-7cdd908e0595/SR300_nodelet_manager-3*.log
[camera_nodelet-4] process has died [pid 14585, exit code 255, cmd /opt/ros/indigo/lib/nodelet/nodelet load realsense_camera/RealsenseNodelet SR300_nodelet_manager rgb/image_raw:=app/camera/rgb/image_raw camera/depth/image_raw:=depth/image_raw camera/color/image_raw:=rgb/image_raw camera/infrared1/image_raw:=ir/image_raw camera/infrared2/image_raw:=ir2/image_raw camera/depth/points:=depth/points camera/depth/camera_info:=depth/camera_info camera/color/camera_info:=rgb/camera_info camera/infrared1/camera_info:=ir/camera_info camera/infrared2/camera_info:=ir2/camera_info __name:=camera_nodelet __log:=/root/.ros/log/39a0b54c-53a6-11e2-9cb6-7cdd908e0595/camera_nodelet-4.log].
log file: /root/.ros/log/39a0b54c-53a6-11e2-9cb6-7cdd908e0595/camera_nodelet-4*.log

@ZahraBoroujeni well glad to hear that at least it works most of the times! I'd blade that on the fact that your board is not "officially" supported. Heck, people are still having problems with "supported" boards too (you can tell by looking at the issues reported in the original repository). Also, RealSense sensors are all still dev-kits and are not consumer-ready yet, so issues are expected.
Looking forward to a working ARM NEON version : )
By the way, what did you use to patch your kernel? Do you have a how-to guide for that or something?

@Maghoumi this works with the Jetson TK1 and R200 camera.

@Maghoumi  sorry for the late answer. To Patch the kernel for odroid,

git clone the linux kernel https://github.com/hardkernel/linux

apply patch, if patch does not work, you can manually change three files inside drivers/media/usb/uvc/ (uvc_driver.c uvcvideo.h videodev2.h) based on the patch file, you should add some lines to the three files, and then you can make a new patch for yourself for future.

because it is arm processor, therefore we should cross compile the linux kernel, use old-config  and make a new config : make ARCH=arm CROSS_COMPILE=path/to/compiler/bin/arm-linux-gnueabihf- oldconfig

go to uvc folder: cd drivers/media/usb/uvc

make -C /path/to/linux M=$PWD ARCH=arm CROSS_COMPILE=/path/to/compiler/bin/arm-linux-gnueabihf- modules

Now you have replace uvcvideo.ko file in your arm board with new one! you can copy the file to lib/modules/uname -r/kernel/drivers/media/usb/uvc/

@ZahraBoroujeni Thanks.
The problem is that Jetson boards (the TX1 at least) are still using kernel v3.10! I will soon check and see if the patch can still be applied to that kernel on the TX1 (as mentioned in the beginning of the post, TK1 seems to be working fine).

@Maghoumi
I built patch files that have been tested on the TK1 for the UVC kernel module, available at:
https://github.com/jetsonhacks/installLibrealsense
They are located in the UVCKernelPatches directory. Note that on the TK1 I switched uvcvideo to be a module instead of built-in.
I have not looked at bringing this code over to the TX1, but I think that the sources for both kernels for uvcvideo are the same, so it should work.

The arm.patch in jetsonhacks' repo works for me, maybe consider adding some macros for arm devices in src/uvc-v4l2.cpp?
I'm using

Odroid XU4
Ubuntu 15.04 (kernel 3.10.82-56 with new uvcvideo.ko)
Intel R200
latest version librealsense with V4L2 backend

When running programs for the first time, there might arise error like

RealSense error calling rs_create_context(api_version:5):
UVCIOC_CTRL_QUERY:UVC_SET_CUR error 5, Input/output error

dmesg is like

[ 1994.530289] [c0] uvcvideo: Failed to query (SET_CUR) UVC control 1 on uni.
[ 2331.871794] [c0] uvcvideo: Failed to set UVC commit control : -32 (exp. 34).
[ 2331.877472] [c0] uvcvideo 4-1.2:1.1: resume error -5
[ 2331.884556] [c0] uvcvideo: Failed to set UVC commit control : -32 (exp. 34).
[ 2331.890215] [c0] uvcvideo 4-1.2:1.3: resume error -5

replug the device seems to solve that.
The frame rate of cpp-alignimages is 3 to 4 HZ and takes up 144% cpu resource, is it because I'm using R200 instead of SR300?
Thanks to all of you guys!

It seems it does not patched correctly! Because it does not find UVC commit control!
You can try the image from the linke below! it is reported that the image works also for R200. But I did not test it personally.
https://github.com/AutoModelCar/AutoModelCarWiki/wiki/Flashing-the-eMMC

Thanks for your help ;-) @ZahraBoroujeni, it turns out that it's a missing OpenGL library causing the cpp-alignimages and other examples run at low frame rates, I fixed it with cv::imshow and now it works at 30HZ.
I also recompiled the kernel to make the xhci-hcd to be an external module, which enables me to reload the module instead of physically replug the device when the mentioned errors occur. This trick seems to only work on the V4L2 backend library.


⚠ Avatar API Endpoint has degraded performance

In 0a5cf97, Avatar API Endpoint (https://avatar.roblox.com/v1/avatar-rules) experienced degraded performance:

HTTP code: 200
Response time: 4331 ms

Resolved: Avatar API Endpoint performance has improved in 87054b1.


Outage issues not being created (seeing an API request return 404)

Describe the bug
Issues are not being created when outages happen.
To Reproduce
Steps to reproduce the behavior:

Have an outage
Check your issues page and actions page

Expected behavior
I expect an issue to be created as intended.

Additional context
I noticed that in the actions log, there's an API request which returns a 404 (see an example on my status page). It looks like this is what's preventing the issue from being created, or at least part of the issue.

I have the same problem when getting a 403 error.
The status page said it was down, but no issue created.


Is any option to send button message

Is any option to send button message and list message 🙂

no


gevent + requests blocks when using socks

I use python(2.7.6),gevnet(1.1.2),requests(2.11.1) to make http requests concurrent, and it works good. But when I add socks proxy to requests, it blocks.
This is my code:
import time
import requests
import logging
import click

import gevent
from gevent import monkey
monkey.patch_all()

FORMAT = '%(asctime)-15s %(message)s'
logging.basicConfig(format=FORMAT)
logger = logging.getLogger('test')

#socks proxy
user = MY_SOCKS_PROXY_USERNAME
password = MY_SOCKS_PROXY_PASSWORD
host = MY_SOCKS_PROXY_HOST
port = MY_SOCKS_PROXY_PORT
proxies = {
        'http': 'socks5://{0}:{1}@{2}:{3}'.format(user, password, host, port),
        'https': 'socks5://{0}:{1}@{2}:{3}'.format(user, password, host, port),
}
url = 'https://www.youtube.com/user/NBA'

def fetch_url(i,with_proxy):
    while True:
        logger.warning('thread %s fetch url'%i)
        try:
            if with_proxy:
                res = requests.get(url,proxies=proxies, timeout=5)
            else:
                res = requests.get(url, timeout=5)
        except Exception as e:
            logger.error(str(e))
            continue

        logger.warning(res.status_code)

def do_other_thing():
    while True:
        logger.warning('do other thing...')
        time.sleep(1)

@click.command()
@click.option('--with_proxy/--without_proxy',help='if use proxy', default=True)
def run(with_proxy):
    if with_proxy:
        logger.warning('with proxy......')
    else:
        logger.warning('without proxy......')
    ts = []
    ts.append(gevent.spawn(do_other_thing))
    for i in xrange(3):
        ts.append(gevent.spawn(fetch_url,i,with_proxy))
    gevent.joinall(ts)

if __name__=='__main__':
    run()

These pictures show the result.
run with proxy
run without proxy

With proxy, do_other_thing will blocks before fetch_url done.
Without proxy， it works good. (timeout error occurs, because of the GFW)

Can anyone help me solving this problem? Thanks very much!

My best guess is that you are monkeypatching too late. When using gevent you must monkeypatch first, before importing anything else. Doing it later runs the risk of breaking anything that has done a from import from the things that gevent monkeypatches.
Can you try moving your import and monkeypatch to the very top of your file before doing anything else?

Yeah,  ' moving your import and monkeypatch to the very top of your file ' solved my problem! Thank you very much!
This suggestion also solves my project's problem.
I hava 3 files called A,B,C, and run sequence is A->B->C.  I use gevent in B, so I add monkeypatch in B, but i blocks with requests+socks. Now I add monkeypatch at the very top of A, it works great now! Hope this will help others!
Thank you again!

Happy to help!


 Make sure you are using the la
test version; type  youtube-dl -U  to update. Be sure to call youtube-dl with th
e --verbose flag and include its complete output.',)
[debug] Invoking downloader on 'https://r4---sn-8xgp1vo-q4fe.googlevideo.com/vid
eoplayback?id=bc9b5fed1fe855ac&itag=135&source=youtube&requiressl=yes&pcm2cms=ye
s&mm=31&mv=m&ms=au&mn=sn-8xgp1vo-q4fe&pl=21&ratebypass=yes&mime=video/mp4&gir=ye
s&clen=18837139&lmt=1434107319160983&dur=277.143&upn=asv5m_WinBs&sver=3&key=dg_y
t0&fexp=901816,9407574,9407664,9408142,9408420,9408710,9413032,9414875,9415436,9
415559,9416126,9416214,9416262,9416361,9416729,9417193&mt=1437136340&signature=3
CC66F0FE9267C1B310FF0073258ECDB43940D35.35676E403AA90C4B6A0AEC63346CB5FAA5209D09
&ip=<IP_ADDRESS>&ipbits=0&expire=1437158022&sparams=ip,ipbits,expire,id,itag,so
urce,requiressl,pcm2cms,mm,mv,ms,mn,pl,ratebypass,mime,gir,clen,lmt,dur'
[download] Destination: 38 Special - Hold On Loosely-vJtf7R_oVaw.f135.mp4
[download] 100% of 17.96MiB in 00:04
[debug] Invoking downloader on 'https://r4---sn-8xgp1vo-q4fe.googlevideo.com/vid
eoplayback?id=bc9b5fed1fe855ac&itag=141&source=youtube&requiressl=yes&pcm2cms=ye
s&mm=31&mv=m&ms=au&mn=sn-8xgp1vo-q4fe&pl=21&ratebypass=yes&mime=audio/mp4&gir=ye
s&clen=8838340&lmt=1432527372607136&dur=277.199&upn=asv5m_WinBs&sver=3&key=dg_yt
0&fexp=901816,9407574,9407664,9408142,9408420,9408710,9413032,9414875,9415436,94
15559,9416126,9416214,9416262,9416361,9416729,9417193&mt=1437136340&signature=0E
8E99FE7D9BABE520E411F1241EC4A7786D2739.5F5F40850461AD14DE3516390135ACC05C11BE80&
ip=<IP_ADDRESS>&ipbits=0&expire=1437158022&sparams=ip,ipbits,expire,id,itag,sou
rce,requiressl,pcm2cms,mm,mv,ms,mn,pl,ratebypass,mime,gir,clen,lmt,dur'
[download] Destination: 38 Special - Hold On Loosely-vJtf7R_oVaw.f141.m4a
[download] 100% of 8.43MiB in 00:02
[ffmpeg] Merging formats into "38 Special - Hold On Loosely-vJtf7R_oVaw.mp4"
[debug] ffmpeg command line: ffmpeg -y -i '38 Special - Hold On Loosely-vJtf7R_o
Vaw.f135.mp4' -i '38 Special - Hold On Loosely-vJtf7R_oVaw.f141.m4a' -c copy -ma
p 0:v:0 -map 1:a:0 '38 Special - Hold On Loosely-vJtf7R_oVaw.temp.mp4'
Deleting original file 38 Special - Hold On Loosely-vJtf7R_oVaw.f135.mp4 (pass -
k to keep)
Deleting original file 38 Special - Hold On Loosely-vJtf7R_oVaw.f141.m4a (pass -
k to keep)

Duplicate of #5118


dp.Select not displaying correctly with 10 selects on a page

System Information

OS:
Python version: 3.9
Python environment: pip
Using jupyter: false
Datapane version: ?

Bug / Issue
I have a page with 10 select blocks. A drop down lists the 10 blocks correctly, all the 10 blocks are selectable. So far so good.
When you select a block from the dropdown, it displays but the block selected is not shown correctly in the drop dowm, so you don't know which block you are looking at.
https://datapane.com/reports/VkBWjZA/ifsc-world-cup-event-statistics/
go to the country results world cups tab and try the dropdown, the text in the dropdown doesn't change when you pick from the list.
This does work ok on other tabs with a dropdown with <10 entries.

Thanks for the issue @paulhighams. Adding @Buroni who looks after this bit of the codebase.

thanks, thats great.

Hey @paulhighams!
We'll keep the issue open until the release, that way anyone searching for the same answer will be in the loop!

@shahinrostami I am experiencing this as well.  Is there a version I should target to attempt to resolve this?

@shahinrostami I am experiencing this as well. Is there a version I should target to attempt to resolve this?

Hello @gelliot-os !
We're pushing a new release soon which resolves this issue - we've kept it open in the meantime.

Hello @paulhighams ! Happy to say this made it into our recent release and should be resolved. I checked the awesome report here https://datapane.com/reports/VkBWjZA/ifsc-world-cup-event-statistics/ and it appears to be working. Please let us know if there's anything else!


requesting option: show parent link in dropdown

once the screen is reduced to small/medium view, the menu's parent items click through to display the sub menu items . But what if one want to allow the user to access the parent menu item's link?
i know it would be possible to make a category without a link, but this is problematic in many cms.. and
from a usability point-of-view there should be an option to display the parent item's link in the dropdown (especially for mobile devices)

Look at the dropdown options, this is already included

o sorry, didnt know that, cant find it in the options, which one is it??
     inDuration: 300,
     outDuration: 225,
     constrain_width: false, // Does not change width of dropdown to that of the activator
     hover: false, // Activate on click
     alignment: 'left', // Aligns dropdown to left or right edge (works with constrain_width)
     gutter: 0, // Spacing from edge
     belowOrigin: false // Displays dropdown below the button
   }```

belowOrigin, but if you want a parent link, just add one in the html markup in the dropdown.


pull request for e2e test at 01/22/2017 08:26:17

:white_check_mark: Validation status: passed
For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Add pulseaudio

While waiting for https://github.com/conan-io/conan-center-index/pull/10152

It was fixed in conan-center


[TV4] HTTP Error 503: Service Unavailable

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I remove or skip any mandatory* field

Checklist

[X] I'm reporting a broken site
[X] I've verified that I'm running yt-dlp version 2022.11.11 (update instructions) or later (specify commit)
[X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue
[X] I've read about sharing account credentials and I'm willing to share it if required

Region
Sweden
Provide a description that is worded well enough to be understood
www.tv4play.se not working:
Unable to download JSON metadata: HTTP Error 503: Service Unavailable
Provide verbose output that clearly demonstrates the problem

[X] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[X] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
[debug] Command-line config: ['https://www.tv4play.se/program/next-level-chef-us/del-5-s%C3%A4song-1/13772514', '-vU']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.11.11 [8b64402] (pip)
[debug] Python 3.10.6 (CPython x86_64 64bit) - Linux-5.15.0-52-generic-x86_64-with-glibc2.35 (OpenSSL 3.0.2 15 Mar 2022, glibc 2.35)
[debug] exe versions: ffmpeg 4.4.2 (setts), ffprobe 4.4.2, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.11.0, brotli-1.0.9, certifi-2020.06.20, mutagen-1.46.0, pyxattr-0.7.2, secretstorage-3.3.1, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Loaded 1723 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2022.11.11, Current version: 2022.11.11
yt-dlp is up to date (2022.11.11)
[debug] Using fake IP <IP_ADDRESS> (SE) as X-Forwarded-For
[debug] [TV4] Extracting URL: https://www.tv4play.se/program/next-level-chef-us/del-5-s%C3%A4song-1/13772514
[TV4] 13772514: Downloading video info JSON
ERROR: [TV4] 13772514: Unable to download JSON metadata: HTTP Error 503: Service Unavailable (caused by <HTTPError 503: 'Service Unavailable'>); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "/home/user/.local/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 674, in extract
    ie_result = self._real_extract(url)
  File "/home/user/.local/lib/python3.10/site-packages/yt_dlp/extractor/tv4.py", line 75, in _real_extract
    info = self._download_json(
  File "/home/user/.local/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 1034, in download_content
    res = getattr(self, download_handle.__name__)(url_or_request, video_id, **kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 998, in download_handle
    res = self._download_webpage_handle(
  File "/home/user/.local/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 868, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
  File "/home/user/.local/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 825, in _request_webpage
    raise ExtractorError(errmsg, cause=err)

  File "/home/user/.local/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 807, in _request_webpage
    return self._downloader.urlopen(self._create_request(url_or_request, data, headers, query))
  File "/home/user/.local/lib/python3.10/site-packages/yt_dlp/YoutubeDL.py", line 3692, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.10/urllib/request.py", line 525, in open
    response = meth(req, response)
  File "/usr/lib/python3.10/urllib/request.py", line 634, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.10/urllib/request.py", line 563, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.10/urllib/request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 503: Service Unavailable

I got the same issue. My guess it has to do with the fact that you have to login using a free account on tv4play now. This seems to have happened just recently.

I don't get any login prompt, but the provided link appears to be DRM protected

Should be the same issue as this: https://github.com/spaam/svtplay-dl/issues/1509
Fix: https://github.com/spaam/svtplay-dl/commit/8950be8eec917b3bf206c3d32d6df2a6fe4797be

And with my little knowledge I don't think it is DRM-protected.

Why do you say so?

Since It is possible to download with https://github.com/spaam/svtplay-dl/ with the latest fix and that program does not "decode" DRM as far as I know.
But maybe I have misunderstood that.

Fix: spaam/svtplay-dl@8950be8

Ah, my bad. I did not notice this part in previous comment 😓
Does this work with cookies?
diff --git a/yt_dlp/extractor/tv4.py b/yt_dlp/extractor/tv4.py
index 1378a6f57..22ee7e2f3 100644
--- a/yt_dlp/extractor/tv4.py
+++ b/yt_dlp/extractor/tv4.py
@@ -73,12 +73,14 @@ def _real_extract(self, url):
         video_id = self._match_id(url)

         info = self._download_json(
-            'https://playback-api.b17g.net/asset/%s' % video_id,
+            f'https://playback2.a2d.tv/play/{video_id}',
             video_id, 'Downloading video info JSON', query={
                 'service': 'tv4',
                 'device': 'browser',
                 'protocol': 'hls,dash',
                 'drm': 'widevine',
+                'browser': 'GoogleChrome',
+                'capabilities': 'live-drm-adstitch-2,expired_assets',
             })['metadata']

         title = info['title']

It does not work for me with fix, with or without cookies.
Latest svtplay-dl does not work either.
502 Bad Gateway. Does it work for anyone else?

So it might work now. The 502 was for the manifest_url still pointing to the b17g domain.

Fix: spaam/svtplay-dl@8950be8

Ah, my bad. I did not notice this part in previous comment 😓
Does this work with cookies?
diff --git a/yt_dlp/extractor/tv4.py b/yt_dlp/extractor/tv4.py
index 1378a6f57..22ee7e2f3 100644
--- a/yt_dlp/extractor/tv4.py
+++ b/yt_dlp/extractor/tv4.py
@@ -73,12 +73,14 @@ def _real_extract(self, url):
         video_id = self._match_id(url)

         info = self._download_json(
-            'https://playback-api.b17g.net/asset/%s' % video_id,
+            f'https://playback2.a2d.tv/play/{video_id}',
             video_id, 'Downloading video info JSON', query={
                 'service': 'tv4',
                 'device': 'browser',
                 'protocol': 'hls,dash',
                 'drm': 'widevine',
+                'browser': 'GoogleChrome',
+                'capabilities': 'live-drm-adstitch-2,expired_assets',
             })['metadata']

         title = info['title']

PS: Related spaam/svtplay-dl#1509

This fix only changes the info_url. The manifest_url also has the be changed to 'playback2.a2d.tv/play/'. The info_url might be 'playback2.a2d.tv/assets' actually?


Unable to use args -R infinite and --fragment-retries infinite

Hi,
I'm using version:
Installed version: v4.3 -   You are up to date.
Installation type: docker
Docker tag: nightly
Commit hash: 9c0a77c
Build date: 2022-08-02
As soon as I try to use args "-R infinite,,--fragment-retries infinite" y get error, these worked on previous version:
Thanks.
2022-08-07T16:13:45.179Z VERBOSE: Collecting info for download 92ffd873-78f8-43b2-8a6f-bcd917aac3c6
2022-08-07T16:13:45.186Z VERBOSE: youtube-dl args being used: -o,subscriptions/playlists/test/%(title)s.%(ext)s,--write-info-json,--print-json,-f,'(mp4)[height<=720],--write-thumbnail,-R infinite,--fragment-retries infinite,--write-thumbnail,--restrict-filenames,--write-all-thumbnails
2022-08-07T16:13:45.199Z VERBOSE: Collecting info for download 4877162f-afab-411b-9820-25b7599cceb7
2022-08-07T16:13:45.205Z VERBOSE: youtube-dl args being used: -o,subscriptions/playlists/test/%(title)s.%(ext)s,--write-info-json,--print-json,-f,'(mp4)[height<=720],--write-thumbnail,-R infinite,--fragment-retries infinite,--write-thumbnail,--restrict-filenames,--write-all-thumbnails
2022-08-07T16:13:46.158Z VERBOSE: Downloading 4877162f-afab-411b-9820-25b7599cceb7
2022-08-07T16:13:46.161Z VERBOSE: Downloading 92ffd873-78f8-43b2-8a6f-bcd917aac3c6
2022-08-07T16:13:54.733Z ERROR: Usage: youtube-dl [OPTIONS] URL [URL...]
youtube-dl: error: no such option: --fragment-retries infinite
2022-08-07T16:13:54.737Z ERROR: Usage: youtube-dl [OPTIONS] URL [URL...]
youtube-dl: error: no such option: --fragment-retries infinite
2022-08-07T16:13:55.202Z VERBOSE: Disabling checking downloads as none are available.
2022-08-07T16:16:41.084Z VERBOSE: Subscription: getting videos for subscription test with args: --dump-json,-o,"subscriptions/playlists/test/%(title)s.%(ext)s",-ciw,--write-info-json,--print-json,-f,bestaudio,-x,--audio-format,mp3,--write-thumbnail
2022-08-07T16:16:47.903Z VERBOSE: Subscription: finished check for test
2022-08-07T16:16:48.294Z VERBOSE: Collecting info for download 283b54b4-46d0-40d0-a4b6-14cd7ef8d0f3
2022-08-07T16:16:48.314Z VERBOSE: youtube-dl args being used: -o,subscriptions/playlists/test/%(title)s.%(ext)s,--write-info-json,--print-json,--audio-quality,0,-x,--audio-format,mp3,--write-thumbnail,-R infinite,--write-thumbnail,--restrict-filenames,--write-all-thumbnails
2022-08-07T16:16:48.328Z VERBOSE: Collecting info for download df317706-3354-44bd-ad01-57d841916002
2022-08-07T16:16:48.349Z VERBOSE: youtube-dl args being used: -o,subscriptions/playlists/test/%(title)s.%(ext)s,--write-info-json,--print-json,--audio-quality,0,-x,--audio-format,mp3,--write-thumbnail,-R infinite,--write-thumbnail,--restrict-filenames,--write-all-thumbnails
2022-08-07T16:16:49.275Z VERBOSE: Downloading df317706-3354-44bd-ad01-57d841916002
2022-08-07T16:16:49.278Z VERBOSE: Downloading 283b54b4-46d0-40d0-a4b6-14cd7ef8d0f3
2022-08-07T16:16:58.041Z ERROR: Usage: youtube-dl [OPTIONS] URL [URL...]
youtube-dl: error: invalid retry count specified
2022-08-07T16:16:58.244Z ERROR: Usage: youtube-dl [OPTIONS] URL [URL...]
youtube-dl: error: invalid retry count specified
2022-08-07T16:16:58.307Z VERBOSE: Disabling checking downloads as none are available.
2022-08-07T16:17:21.265Z VERBOSE: Running task youtubedl_update_check
2022-08-07T16:17:21.443Z VERBOSE: Finished running task youtubedl_update_check

Post the actual verbose youtube-dl log, or raise the issue with the creator of the Docker image. Make --print-json be omitted, as it overrides --verbose.
I just ran the released yt-dl with youtube-dl -v -f 'best[fps < 50]' -o 'subscriptions/playlists/test/%(title)s.%(ext)s' --write-info-json --print-json --audio-quality 0 -x --audio-format mp3 --write-thumbnail -R infinite --write-thumbnail --restrict-filenames --write-all-thumbnails with no error for the retry count infinite.

I have changed to yt-dlp on the web interface and added again the subscription, following the log file
combined.log
Thanks.

Raise the issue at https://github.com/Tzahi12345/YoutubeDL-Material/issues as you don't have access to the required debug information.


fixing the dev environment by downgrading clean-webpack-plugin

creating an error boundary and also adding StrictMode
having all the components throw errors synchronously so they hit the error boundary
Honestly, I'm not sure I like the idea of this pattern or not. The goal behind it was to get something like the screenshot below for anyone who has an error, since asking people who have never devved/inspected plugins could be a tall task. But I'm not sold on it.
Thoughts?

addressed feedback.

👍


Update links to the connectors repository

The connectors repository has been recently archived (on Jul 13, 2023). Its contents have been moved into the delta repository, under connectors.
Therefore, any link pointing to the archived repository should be updated to point to the new location instead.
We update links from published blog posts as well, as that's probably the right thing to do.
But we leave untouched any link that refers to an issue, a release, a pull request or a branch, as those should still point to the archived repository. Only links that refer to files or directories within the repository are adjusted.

Closes #326

Thanks @manuteleco !


Color Specific Hue/Saturation from Photoshop to iOS

I'm trying to use GPUImage to map this filter. Please note, I need help mapping the color (Blues) specific (note: NOT Master, just Blues) photoshop element to iOS.
Does anyone know how to manipulate a CIFilter or GPUImage class to get the photoshop effect below in iOS?

I don't think there are any filters out of the box that let you manipulate the HSL of one color channel. That said, it would be relatively simple to edit each of the filters to just take one color channel, then add a parameter to (optionally) take that channel in the interface.


Colliding registrations throws some weird errors

  OP_DECOMPOSE(batch_norm);

[W OperatorEntry.cpp:116] Warning: Overriding a previously registered kernel for the same operator and the same dispatch key
  operator: aten::batch_norm(Tensor input, Tensor? weight, Tensor? bias, Tensor? running_mean, Tensor? running_var, bool training, float momentum, float eps, bool cudnn_enabled) -> (Tensor)
    registered at aten/src/ATen/RegisterSchema.cpp:19
  dispatch key: FuncTorchBatched
  previous kernel: registered at ../aten/src/ATen/autocast_mode.cpp:450
       new kernel: registered at /home/chilli/fb/functorch/functorch/csrc/BatchRulesConv.cpp:198 (function registerKernel)

The error is caused by the fact that we're registering two batch norm keys - the error message is very strange, but I'll close this issue for now.


Logistic Regression is Classification

Hey! I noticed you put logistic regression under regression. Despite its name, Logistic Regression is actually used for classification.

Hi @TheHollidayInn,
thanks, yes, it is correct, Logistic Regression is very often used for classification, but not only, which we would like to leave it in this category.
Wikipedia explains it best:

The logistic regression model itself simply models probability of output in terms of input and does not perform statistical classification (it is not a classifier), though it can be used to make a classifier, for instance by choosing a cutoff value and classifying inputs with probability greater than the cutoff as one class, below the cutoff as the other; this is a common way to make a binary classifier.

Azo


Address multiple Done/Delete issues

Allow for Done command although some tasks are already marked as Done
Update user feedback for multiple tasks done/delete

maybe use auto complete for this?