feat: allow label etc params as well as slot for simplicity

Allowing params for label, prefix, suffix, overridden by slots of the same name, instead of only allowing slots, simplifies usage. If this makes sense to you I'm happy to do this for all components it makes sense for.
Removing the requirement for autoComplete param on TextField as this should not be required.

Yeah, I think we should apply this for all components.
But please beware the HTML structure from Shopify, if you wrap span around the label or prefix,... that's different with Shopify, that may cause the css issues.
So you just should wrap it by template (null fragment). E.g: template(v-else) {{ label }}

Thanks. I just didn't understand pug templating enough and it was complaining when I did:
template(v-else
  {{ label }}

There are 2 ways:

template(v-else) {{ label }}
or

template(v-else)
  | {{ label }}


WIP: Small UI improvements Application summary

Summary
Fixes #

Occurred changes and/or fixed issues

Small UI improvements for Epinio Application Summary
TODO:

[ ] pull data for created at,uptime, modified

Areas or cases that should be tested

Areas which could experience regressions

Screenshot/Video

Should we have the envs values hidden and an icon to show the actual values?

Current state:

Opening a new PR: https://github.com/rancher/dashboard/pull/5678


Describe the results you expected:
The sleep process terminates immediately.
Provide additional info you think is important:
When executing the same in bash process, sleep is terminated by Ctrl-C.

Could be duplicated of #2838, but it's closed. And I don't know why. Is it expected behavior? I thought -ti options should make docker processes to behave the same as usual bash processes.

Found another duplicate: #3793

Have you tried
docker run --rm -it alpine sh -c "sleep 5"

So that "sleep" is a process in the shell?

It works. How can I make it a default behavior, so that any CMD would be executed in a shell?

No, it never is a default option, because the command you're running may itself be a shell. You could build your own image using a shell as an "entrypoint", e.g. https://github.com/jfrazelle/dockerfiles/blob/master/fish/Dockerfile

Tried this:
ENTRYPOINT ["sh"]

Didn't work:
$ docker run --rm -it  shell-image ls
sh: can't open 'ls'

Ok, this works:
ENTRYPOINT ["sh"]

Well, it requires quoting arguments, which is not good:
$ docker run --rm -it  shell-image echo test
# empty line

Is there a solution to that?

I thought this is a bug (#21014), but it's a valid behavior. So I need an "sh" option to interpret all arguments as arguments to the first argument-command.

@Vanuan You basically need to setup whatever gets run in the container with signal handlers.
A lot of things, when run as PID1, drop their signal handlers.
You can do something like:
#!/bin/sh
trap "exit 0" SIGINT

sleep 5

I want to be able to use docker as a proxy to dev tools. Some (it not all) dev tools don't have SIGINT handlers. So I need a global solution. For now, only this workaround is suitable:
ENTRYPOINT ["tini", "--"]

Other reasons to use tini as your entrypoint: https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/

That'll go away in 1.11 when we move to using containerD as the execution runtime.

Oh, thanks!

Will Ctrl-C issue go away too?

@Vanuan No, we won't be doing anything to make a pid be not be PID 1, this will be totally up to the what gets started in the container.

IMHO, it's much more common to run program that doesn't expect to be run as PID 1. Vagga creator thinks the same: http://vagga.readthedocs.org/en/latest/pid1mode.html
This issue makes it particularly annoying to use Docker for any development tools.

@Vanuan Docker starts a process in a container for you, it's up to you to decide how it should run.


CreateInStream: Syntax error in the example

The example says:
MyTable.MyBLOBfield.CREATEINSTREAM(MyStream, TextEncoding.WINDOWS)
But the correct syntax (note the "::") is:
MyTable.MyBLOBfield.CreateInStream(MyStream, TextEncoding::Windows);

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 1ac054bd-7f24-6c8e-b822-f0e7fc5e3c67
Version Independent ID: 5bcc6c83-94d8-e3de-dcb3-a5cf5d1bd9b1
Content: CREATEINSTREAM Method (BLOB) - Business Central
Content Source: dev-itpro/developer/methods/devenv-createinstream-method-blob.md
Service: dynamics365-financials
GitHub Login: @SusanneWindfeldPedersen
Microsoft Alias: solsen

Thanks for the feedback. We have fixed the mistake, and an update will be published soon.


Question regarding sin(x)=0.7

Is there a way that AlgebraJS is able to solve equations such as sin(x)=0.7 or similar?  If not, what I need to do on my end, so that the latest version of AlgebraJS can solve it?

There are no plans to support trigonometric operations in the near future.

@nicolewhite How hard would this be to implement? I would think that given JavaScript's functions like Math.sin and Math.asin, this should be fairly trivial, but I don't pretend to understand the workings of this project. Still, I think that this could be a relatively easy thing to check off
Is there a tasklist somewhere describing what is planned?


Add tailoredDeprecationMessage in addition to the common deprecation warning

Description: If the task is deprecated and this message exists in the task.json file, it will be added to the common deprecation warning. Suggested to display this message as a separate warning for better visibility.

/azp run


nixos/zsh: don’t be noisy when scripts are run with -u

Description of changes
When a script specifies the shell option “nounset” as part of the shebang (e.g., via “#!/usr/bin/env -S zsh -u”), our initialization scripts would produce error messages of the form:
__ETC_FOO_SOURCED: parameter not set

These messages could probably be confusing to users when running such scripts. By providing a fall-back in the parameter expansion, we can avoid them.
This patch does not address interactive shell start-up, where such messages may (or may not) be less problematic.

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[ ] Fits CONTRIBUTING.md.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-ready-for-review/3032/1423


Needless fork: github.com/rakyll/pb

New release 2.2.0 bundles github.com/rakyll/pb which appears to be an outdated fork of github.com/cheggaaa/pb. The latter seems to be better maintained.
Please consider using original project.

/cc @yichengq

@onlyjob I need some time to try it out to see whether they perform the same/meet our requirements.

Thank you. At least etcd-2.2.0 builds just fine with github.com/cheggaaa/pb.
IMHO using only well-maintained projects is important -- file a bug if it doesn't meet your requirements but please be very reluctant to integrate (seemingly) unmaintained forks where you can't even open a ticket if something goes wrong...

Thank you. :)

This is still a problem in 2.2.5: https://github.com/coreos/etcd/blob/v2.2.5/tools/v3benchmark/main.go#L24
Please reopen...

@onlyjob Thanks. Reopened. Will fix.

@onlyjob Just fixed https://github.com/coreos/etcd/commit/9f5189cd001593cf304ce45ad319777b2fe884f8

Thank you!


Published actions in this repository are pre-compiled before commit

Currently, the javascript action published in this repo is packaged using ncc build index.js, and the compiled artefact is committed to the repository.
Consider a pattern where ncc is run as a CI step.

https://www.edwardthomson.com/blog/building_an_action.html


[Scroll] RFC: Add scrollWithoutAnimationTo to immediately scroll

This is an API like setting the contentOffset except it is imperative, which is usually what you want unless the scroll view is a controlled component (for that to be realistic, JS would need to drive the scroll physics and specify the contentOffset during each frame without interruptions like the GC or CPU-hungry JS).

I would want to keep boolean args out of the JS API since JS doesn't have named args.

Totally agreed.
This looks good to me

closing in favor of #509 w/suggested changes


Icons & Dropdown seems broken to me - Angular 12

Describe the bug
I'm new to development with angular but also this is my first time using clarity, I created a new Angular project version 12.2.4, then I installed clarity with the command ng add @ clr / angular , then I added in my component app.component.html file a simple header and sidebar and this is what I got.

I couldn't see the icons or use the dropdown in the header section.
Expected behavior
I can use the header dropdown and see the icons.
Versions
Clarity project:

[ ] Clarity Core
[x] Clarity Angular/UI

Clarity version:

[ ] v3.x
[ ] v4.x
[x] v5.x

Framework:

[x] Angular
[ ] React
[ ] Vue
[ ] Other:

Framework version:
Angular 12.2.4
Code:
app.component.html

        <cds-icon shape="vm-bug"></cds-icon>
        Project Clarity

        <button class="nav-text dropdown-toggle" aria-label="open user profile">
<EMAIL_ADDRESS>          <cds-icon shape="angle" direction="down"></cds-icon>
        </button>

          Preferences
          Log out

        <cds-icon shape="home"></cds-icon> Home
        <cds-icon shape="cog"></cds-icon> Settings
        <cds-icon shape="user"></cds-icon> Users
        <cds-icon shape="Folder-open"></cds-icon> Files
        <cds-icon shape="Cloud"></cds-icon> Data
        <cds-icon shape="image-gallery"> Photos</cds-icon> 6
        <cds-icon shape="video-gallery"></cds-icon> Videos

      ...

angular.json
"styles": [
<EMAIL_ADDRESS>      "src/styles.scss"
    ],
    "scripts": []

app.module.ts
import { NgModule } from '@angular/core';
import { BrowserModule } from '@angular/platform-browser';

import { AppRoutingModule } from './app-routing.module';
import { AppComponent } from './app.component';
import { HomeComponent } from './components/home/home.component';
import { AddOrEditProductModalComponent } from './components/add-or-edit-product-modal/add-or-edit-product-modal.component';
import { DeleteProductModalComponent } from './components/delete-product-modal/delete-product-modal.component';
import { ClarityModule } from '@clr/angular';
import { BrowserAnimationsModule } from '@angular/platform-browser/animations';

@NgModule({
  declarations: [
    AppComponent,
    HomeComponent,
    AddOrEditProductModalComponent,
    DeleteProductModalComponent
  ],
  imports: [
    BrowserModule,
    AppRoutingModule,
    ClarityModule,
    BrowserAnimationsModule
  ],
  providers: [],
  bootstrap: [AppComponent]
})
export class AppModule { }

I didn't declare or import anything manually, it was all done after installing Angular 12 and the Clarity, are there things that I should import manually or how to make it work ? Can you help me with this please...

Hi @nabilbejar,
What you are missing is adding the icons you would like to use in the registry.
Here you can see more information how to do that.

Hi @bbogdanov,
Thanks for the link that you gave me, I used these two lines in the file app.module.ts to import some icons and it works fine in my HTML template
import { ClarityIcons, userIcon, homeIcon, vmBugIcon, cogIcon, eyeIcon } from '@cds/core/icon';

ClarityIcons.addIcons(userIcon, homeIcon, vmBugIcon, cogIcon, eyeIcon);

Can you  please or someone else tell me why the dropdown is not functional ? Is something still missing ?

The code you are using is for the HTML/CSS only. It's just styling.
If you want an interactive dropdown in @clr/angular, you have to use the clr-dropdown component.
Docs/examples can be found on the website


Fix crash during sync.

Do not use statefulset number of pods to figure out running ones
for volume resizing, since the statefulset pointer could be nil.
Instead, look at the actual running pods.

:+1:

👍

Coverage decreased (-0.01%) to 11.332% when pulling f474a46b2f5656eee9d9a74b58630f555be83e00 on sync_volumes_before_statefulset into 88d6a7be3ff8c6b3ccd0cfdac5cfbd1aaa7c7f55 on master.


Fix "[Sandbox] Could not enable Mach bootstrap, errno = 1."

Possibly related to WKWebView under Xcode 14 beta?

Investigate when iOS 16 and Xcode 14 is released?

Don't see warning anymore.


error on windows 11 powershell with ps1

this is the error I get on windows 11 powershell running as admin

no changes have been made to the file. I just open with powershell (admin or not) and the window pops up and goes away, when I open in powershell ISE and run the script it gives me these errors instead of just pooping up and going away.

Same here

Thank you @edouardaihua for your answer.
Auto-GPT is evolving at an incredible speed.
I've updated the .ps1 and .sh script to make them more future-proof. You should check the latest release :)
cc. @spart0n654 - I'm closing this issue since it's not releated to the current release


Dont work right.

I've already tried involving it in a safeareaprovider, it doesn't work. I made it work without the offset, but when I click on show and hide several times at the same time, toast crashes and stays open forever.

Hello, you have to wrap your whole app with SafeAreaProvider, I cannot see that in the attached images. If you are not sure how to do it, I can show you an example.

Show me please, then

export default function RootNavigation() {
    const { top } = useSafeAreaInsets();

    return (
        <RootProvider>
            <NavigationContainer />
            <Toastable
                statusMap={{
                    success: 'red'
                    danger: 'yellow'
                    warning: 'green'
                    info: 'blue'
                }}
                offset={top}
                position="top"
            />
        </RootProvider>
    );
}

import { SafeAreaProvider } from 'react-native-safe-area-context';

export const RootProvider = ({ children }: PropsWithChildren) => (
    <SafeAreaProvider>
        { /* your other providers */ }
            {children}
    </SafeAreaProvider>
);

Ok, it works, I tested it with your settings, just a note, it is saving the state and not providing security in the rendering, if the user clicks several times to run the show toast, it saves the state and even if the user stops clicking it keeps spawning on the screen continuously.
Another thing, I tested it within redux saga and it didn't work, is there any way to make it work there?

Can you show me the code? It should work within React context or outside.
It will be shown as much as user clicks the button. What's wrong with this?

I suggest you add a takeLatest, so if the user clicks several times, the component will only execute based on the last one, not causing an overload crash.
About redux, in a common redux, I tested it and it worked in react components, but trying to run it in the saga generator function, it simply wouldn't open. But anyway, I'm not using your lib anymore, thank you.


Register v0.5.2

@JuliaRegistrator register()

Registration pull request created: JuliaRegistries/Test/37
After the above pull request is merged, it is recommended that you create
a tag on this repository for the registered package version:
git tag -a v0.5.1 -m "<description of version>" af1c18b7058d174542ed48af939400e02bd83cd0
git push v0.5.1


Fix some shutdown issues in the web plugin

This PR fixes 2/3 shutdown issues I found in the web plugin; in particular the web plugin now shuts down correctly when run as a separate command when receiving a signal and when detecting a node shutdown.
Note that it does not address the situation where web server is passed as a start command to vast start.

@lava
Does this warrant a changelog entry?


local cluster metadata rpcAddress does not match the frontend endpoint

Hi,
In the new scheduler feature having the rpcAddress setup to the correct endpoint is essential as otherwise schedule workflow fails to run the scheduled jobs as it uses this info to connect to a Frontend. Changing this value post server build is a pain hence it would be better if one is set correctly from the start.
Currently the rpcAddress is set to the Cadence default port where it should be set similar to the way public client is setup in order for the scheduler to connect to the same endpoint with the system worker.
https://github.com/temporalio/helm-charts/blob/e9dc0d001c5df7f1f3672e0eb9c2508e14c723ac/templates/server-configmap.yaml#L156
Thanks,
A

FYI, this shouldn't be an issue any more after server 1.18 (soon to be released). The current cluster rpcAddress will be ignored and it'll use only either publicClient.hostPort or the internal membership service resolver, with whatever TLS you've configured for local frontend connections.

That will be great.
Is this property plays any other role in the system in case some other problem related to it surfaces?

Fixed by #497.


exposition: prepare for release 0.8.0

Prepare for relase of metriken-exposition v0.8.0

Closing w/o merge. Version number is already bumped for release.


'Search' and partial name-matching does not work for library radio stations

What version of Music Assistant has the issue?
2.1.1
What version of the Home Assistant Integration have you got installed?
2024.07.1.b0
Have you tried everything in the Troubleshooting FAQ and reviewed the Open and Closed Issues and Discussions to resolve this yourself?

[X] Yes

The problem
Summary: radio stations in the library (added from any of Tune-in, RadioBrowser or via explicit URLs) don't seem to be findable via 'Search' or playable with partial name matching.
Details

My Radio library setup:

(Grayed out radio stations are from RadioBrowser stations added to library, with the RadioBrowser provider deactivated, see explanation at the end. This is not related to the problem and could all be reproduced without RadioBrowser provider.)
'Radio Nova' was added from Tune-in, explicit URL and from RadioBrowser.
'FM4' was added from Tune-In, and 'oe1-q2a' was added via URL (with the renaming bug mentioned in #2548)

Things that work:

playing (non-grayed-out) radio stations via the UI
playing via action mass.play_media using the full station name:

action: mass.play_media
target:
  entity_id: media_player.vlc_telnet_2
data:
  media_id: radio nova

Things that don't work (but I would expect to work(?)):

finding a (library) radio station via UI MA search:

searching via mass.search:

action: mass.search
data:
  name: Radio Nova

returns emtpy lists (with media-type=radio on or not)

playing mass.play with partial station name:

action: mass.play_media
target:
  entity_id: media_player.vlc_telnet_2
data:
  media_id: fm4

returns an error Failed to perform the action mass.play_media. MediaNotFoundError: Could not resolve ['fm4'] to playable media item
Note that UI search and partial-name action-call mass.search and mass.play_media does actually work with the RadioBrowser  provider, but only on the 'provider-side', not with stuff in the MA library, which is why I had de-activated that provider for the above experiments.
(One problem with RadioBrowser provider-side matching is that its is way to unspecific and will match unintended stations, rather than my library/favorites).
How to reproduce
Turn off or deactivate RadioBrowser provider. Add radio stations to library via Tune-in Radio or explicit radio-stream URL providers.
Try UI Search, run Developer->Actions mass.search with partial or full station name, or mass.play_media with a partial station name.
Music Providers

Tune-In Radio
added radio stream URLs

Search and play works with RadioBrowser, but only provider-side, not with items in the library.
Player Providers
any / irrelevant
Full log output
No response
Additional information
Maybe some similarities with #2188 but overall seems to be something else (?).
What version of Home Assistant Core are your running
2024.8.1
What type of installation are you running?
Home Assistant OS
On what type of hardware are you running?
Generic x86-64 (e.g. Intel NUC)

Well trying to play something with a partial name is not supported
Something else is going on with your search though as you should have returns for all the item types like this:

Playing partial names does work though with library tracks/albums/artists and it also works for Radio stations if RadioBrowser is active. This is also quite important so that voice commands like 'play radio fm4' just work without having to quote the full name in the library.
Your above search results indicate that all stations are found on RadioBrowser though, which also works for me, but not if I have only non-RadioBrowser stations (Tune-In / URL). Also it does not seem to list RadioBrowser stations higher that I have in my library, eg this:

which are 'bad' results in the sense that they are the wrong 'Radio Nova' stations found, even the 'top restult', on the 'RadioBrowser' provider, but not even listing on the first page the one I have added to my library.

It might work trying to play something with a partial media id but there is no guarantee you would get the correct result. Hence why there are options to specify media type, artist name or album names.
It is not possible to search TuneIn. Only TuneIn favourites are imported into the MA library.
The view you are looking at though is the GLOBAL SEARCH. It is designed to search across all providers so you can find items to add to the library. Thus in library items are not prioritised. If you want to search through your in library items you need to go to the relevant view.
mass.search works the same way as above. However, there is an item on the backlog to enable the limiting of this search to library or favourited items. ETA for that can't be given.
So I believe MA is working correctly given the current level of functionality.

mh, I see ... a feature, not a bug :disappointed:   ... thanks for clarifying.
It still seems a bit strange to me that 'search' does not seem to cover the radio library per se, not just for the in-library Tune-in stations, but also the by-hand added URL streams seem to get fully ignored :man_shrugging:
In any case, being able to limit the search to library items and/or favorites might indeed be a good solution for this, given that at least with RadioBrowser this would work.
However, it seems to me that for assistants and automations to work correctly with radio stations, one basically has to tell them what exact radio stations should be considered possible, and one cannot offload this task to MA, despite the stations being in the library/favorites?! ... oh well. (Maybe I still misunderstand ..)

Yes you would have to tell MA which one. Take these as an example. If I said play "Triple M" or "Classic Rock" how could MA know which one I was referring to?

I don't understand why you want to search for the URL stream you have added.

Yes, so if I had only one of those 'Triple M ...' stations in my library or favorites, then it should just work no? Even if I had a few of them in my library, then it could just pick one, as it usually does with music found in the library.
My main motivation was to be able to tell the OpenAI assistant: "play Radio Nova in the kitchen" or "play FM4 ... " etc, for example, even if
a) the station name is not unique in RadioBrowser or
b) if it's only a partial match with the full station name ("ORF Radio FM4" from Tune-in).
as in both cases using my library items would dis-ambiguate what was meant.
So I guess the actual 'missing' features at the moment are:

search over and do partial-match play_media() on Tune-in stations and manually-added URL streams
being able to restrict or preferentially match on library-items/favorites

Right?
But yeh, I'm seeing enough work-arounds to get around this, if this is all intended behavior right now.
I am very new to HA/MA and so my apologies if I misunderstand the details/ideas of how this is actually supposed to work. Thanks for the feedback!

The other problem you have is the AI may guess what you mean. The prompts we have provided don't allow for radio stations.
Having said that I expect that it should work for a service call (action as it is now) so let me ask about that.

Ok an adjustment has been made which should prefer local matches even partial ones. Let’s see how that goes after the next update.

Try 2.1.2 or 2.2.0b4 and see how that goes

Awesome!! :+1:
Updated to version 2.1.2 and now basically everything works: :tada:

global UI search finds radio items in the library (from Tune-In or URLs)
service/action mass.search finds all library radio items as well
service/action mass.play_media finds and plays (incl. partial matched) library radio stations

All this works based on Tune-In and local URL streams, without RadioBrowser provider.
With RadioBrowser the search also covers 'upstream' provider matches, but now gives preference to in-library matches even for partial matches.
Perfect! :)


Update launch_options.md

I have no idea how to use GitHub.
Hope I'm doing this right.

Thank you for your contribution to mastercoms!


Save Schematic

Hello, Nice work but i have one problem.
Cannot serialize graph nodes to the file.
i saw this error:
"GraphNode" is not marked serializable.
But when i marked serializable class and removed read only item,
I have serialized the node but if node have a connection i got a crash whit same problem
Have a idea how to fix this bug?

You'll probably want to write wrapper classes and use Newtonsoft.Json to serialize.  You can find examples (messy but functional) in one of my projects (https://github.com/RichardRanft/T3DConvoEdit) in T3DConvoEditor/Wrappers.  It's a little extra work, but not terrible.
I'm also working from a modified version of Graph - I need to be able to extend the classes in consumer applications so I have "broken" it.

Hi Richard i have fixed whit all issue.
After testing about hour, the conclusion is mark all event whit "Non Seriazable"
Typically .net cannot serialize event. after this fix, work fine


Fix CRS file size check and incorporate it into the Wasm backend

The CRS file size check is only present in the static lib and always results in a download whenever nargo prove or nargo verify is run. This PR changes the check to ensure that it passes if the following g2_data assignment makes sense. Moreover, the check is moved to the CRS logic in common, thus incorporating it into the Wasm backend, and common's CRS code is brought into scope in barretenberg_static_lib to avoid duplicate code.

Yup. Should I just move the tests over to lib.rs, or would you prefer I put them somewhere else?

I think they are fine where they are for now. We can always move them if needed. I'm going to merge this in a moment as I need this fix as well.


performance on oriented images

Thanks for sharing your work.
I wanted to know your thoughts on how will retinaface perform on rotated face images/ face at weird angles?
When i tested it the result is not very good.
Moreover,
How could i use the landmarks information to maybe align the image?
Thanks in advance.

There is no special setting for rotating face in the network. If you want to get good results, you need to add more rotating faces to your training.

Okay, Thanks for the response.

Do you have a pre trained model？@biubug6

Pretrain model and trained model are put in google cloud and baidu cloud. You can refer to Training in readme.md


Update the tag patter in the docs

Proposed changes:

The docs can now point to the 2.0.0a1 2.0.0rc1 version

Status (please check what you already did):

[ ] added some tests for the functionality
[x] updated the documentation
[ ] updated the changelog (please check changelog for instructions)
[x] reformat files using black (please check Readme for instructions)

mhm I think we usually use the 2.0.0a1 and 2.0.0rc1 format (as that is the python default)

Shouldn't it be the semver style?

yes imo it should be semver, but the python community decided it would be a great idea to go a different route when handling pre-release versions...


Release 1.4.0 breaks simple inheritance

Code :
# frozen_string_literal: true

module NetboxExtractor
  module Netbox
    class Object
      prepend MemoWise

      attr_reader :netbox_object

      def initialize(netbox_object)
        @netbox_object = netbox_object
      end

      ###############
      # NETBOX DATA #
      ###############

      def netbox_data
        netbox_object.raw_data!
      end
      memo_wise :netbox_data

      def netbox_tags
        netbox_data.dig('tags').map { |t| t['slug'] }
      end
      memo_wise :netbox_tags
    end

    class Device < Object
    end

    class VirtualMachine < Object
    end
  end
end

Before :
netbox_object = NetboxClientRuby.dcim.devices.first
device = NetboxExtractor::Netbox::Device.new(netbox_object)
puts device.netbox_tags # => ["netbox-synced"]

After :
netbox_object = NetboxClientRuby.dcim.devices.first
device = NetboxExtractor::Netbox::Device.new(netbox_object)
puts device.netbox_tags # => { "id" => 45, "url" => "http://netbox/api/....json", ... }

Here device.netbox_tags returns the call to netbox_data without the filtering part (.dig('tags').map { |t| t['slug'] })

Thanks for reporting this, @n-rodriguez, and so sorry for the problematic behavior! I believe this is the same issue as #251, which we have a fix for and we hope to publish a new version to Rubygems within the next few days.

@n-rodriguez when you have a moment, could you please try version 1.5.0 and hopefully confirm that this issue is fixed?

@n-rodriguez would it be possible to verify that that latest version of MemoWise fixed your issue?
Planning to close this issue in two weeks -- on Tue Feb 8, 2022 -- unless we here back that this issue remains unfixed

I'm assuming this is fixed. @n-rodriguez please let us know if you continue to experience issues.


Updated sonobi docs to support britepool user id module

@JonGoSonobi - is this related to a code PR? I don't see one.

found this was already updated as part of a general update of the userIds field


[SPARK-7683] [PYSPARK] Confusing behavior of fold function of RDD in pyspark

Fix order of arguments that Pyspark RDD.fold passes to its op -  should be (acc, obj) like other implementations.
Obviously, this is a potentially breaking change, so can only happen for 2.x
CC @davies

Test build #49458 has started for PR 10771 at commit 5fe0d43.

Test build #49458 has finished for PR 10771 at commit 5fe0d43.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49458/
Test PASSed.

I think this is the right thing to do for 2.x but would love a sanity check from .. @JoshRosen @davies @mengxr  ?

LGTM, the implementation did not match with docs, I think it's a bug, not behavior change.

Merged to master

Can we add a unit test for this?


incubator-kie-kogito-apps missing file headers

The following kinds of files need to be updated to include the ASL license header:

Makefile
BPMN
DMN
XML
graphqls (if comments are supported)
Properties
SQL
Markdown
Dockerfiles
Significant text files

@porcelli

incubator-kie-kogito-apps-rat-report.txt


hosted version of Arcs is available at https://live.arcs.dev returns empty page :-(

Please let us see this early work, I've been following it since the beginning and some parts still elude me.
Thanks!

Can you tell us something about the system (OS) and browser you are using?
Also, when working, the latest iteration can look blank at startup, because it only displays a little bit of (mostly white!) UI at the bottom of the screen (one needs to mouse-over or tap to make it pop-up).

It's been working across all my devices' browsers for some time now.
Can't wait to see where this and openid connect leads us. :-)


Push live

Fixes #218

Docs Build status updates of commit bf15f00:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

.gitattributes
:white_check_mark:Succeeded

.openpublishing.publish.config.json
:white_check_mark:Succeeded

AdaptiveCards/breadcrumb/toc.yml
:white_check_mark:Succeeded

AdaptiveCards/content/designer.png
:white_check_mark:Succeeded

AdaptiveCards/getting-started/flow.md
:white_check_mark:Succeeded

AdaptiveCards/getting-started/windows.md
:white_check_mark:Succeeded

AdaptiveCards/resources/partners.md
:white_check_mark:Succeeded

AdaptiveCards/sdk/authoring-cards/net.md
:white_check_mark:Succeeded

AdaptiveCards/sdk/designer.md
:white_check_mark:Succeeded

AdaptiveCards/sdk/rendering-cards/javascript/fabric.md
:white_check_mark:Succeeded

AdaptiveCards/sdk/rendering-cards/javascript/getting-started.md
:white_check_mark:Succeeded

AdaptiveCards/sdk/rendering-cards/react-native/getting-started.md
:white_check_mark:Succeeded

AdaptiveCards/sdk/rendering-cards/toc.yml
:white_check_mark:Succeeded

AdaptiveCards/toc.yml
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Allow for passing null / undefined into a typed, none required prop

If a prop is declared with type validation but required:false, the validation will fail when passing either null or undefined as the value.
I think that the type validation should only occur on none required props if an actual value (not null or undefined) has been passed. I believe this is similar to issue #1070.
Here's a JSBin demonstrating this.

Shouldn't the prop check for a default value first? It would keep it more in line with what happens if you omit the prop completely.
Here's another JSBin showing the different ways to pass a non value prop through. I'm thinking it should fall back to the default value in all cases.

I have thought about this before, and didn't implement it because user may explicitly want to pass down undefined - but now I think it is appropriate to map it to the semantics of JS function calls so that default value would be used if the passed down prop is undefined.

See 04116fc

Thanks. I was going to suggest a nullable prop option for allowing nulls to be passed down in the  case of none required props. In this case, the default would only apply if the prop was completely absent.

@yyx990803 I'm a bit confused by the commit, it says
`if (raw === null) {
// initialize absent prop

   initProp(vm, prop, getDefault(vm, options))

   initProp(vm, prop, undefined)`

but I can't see any code for handling undefined, and the commit actually says that if it is null, then it's set to default.
when is undefined passed as a prop handled?

// check default value
if (value === undefined) {
value = getPropDefaultValue(vm, prop, key);
// since the default value is a fresh copy,
// make sure to observe it.
var prevShouldConvert = observerState.shouldConvert;
observerState.shouldConvert = true;
observe(value);
observerState.shouldConvert = prevShouldConvert;
}
{
assertProp(prop, key, value, vm, absent);
}
I found this bit in vue.js v2 dist, is it around this part that defaults are now handled?
I can't find a comment of
Build a function that applies props to a vm.
anywhere in the current v2 vue.js build


As a developer I'm want to set server settings via options

When using Gitlab CI I want set server settings via options. Example:
-h host -u user -p pass --deploy_path=/var/www
I'm known about option() but it's very difficult to set server settings with it

What's up? No comments? I'm one in the world use both deployer and gitlab?

Try to ask on gitter - may be some can help you there.


.macos high sierra

I find that most of the finder related tweaks no longer work in High Sierra. For example, finder is no longer quitable, and doesnt default open in list view. Is there a fix?

I found that both mentioned configs in this issue worked on High Sierra. But as stated in #801, the Notification Center is no longer disabled.

I know I'm almost a year late to answering this, but it should be noted that after running defaults write com.apple.finder FXPreferredViewStyle -string "Nlsv", folders that were opened previously will still open in icon view because of the .DS_Store file that finder automatically generates to save the view mode a folder is opened in. If you want your previously opened folders to open in list view after setting the FXPreferredViewStyle property, you'll either have to delete the .DS_Store files in each of those directories manually, or you can run sudo find / -name ".DS_Store"  -exec rm {} \; (as per this Stack Exchange answer). Remember that sudo is powerful and potentially dangerous, so be careful, and if you don't know what you're doing, make sure to copy and paste that command so you don't mess something up.


Namespace - Detail

Note: Move container default resource limit into tab and make it look similar to the layout for the edit page. Work on rancher/dashboard#568 before this one.

[ ] Tabs Update

Tab Name
Content Notes

Container Default Resource Limit
Make the component/layout inside the tab match the Edit Namespace page

Labels
Confirm it uses the ResourceTabs component

Annotations
Confirm it uses the ResourceTabs component

@lvuch can you do a mock for this one?

https://xd.adobe.com/view/16abab82-98ff-46d0-59c7-2261b36a56fb-609a/

The label should be milli CPUs for CPU.


🎁: Workorder area add field and filter

Expected outcome

[ ] Replace title in table with description
[ ] Add type of work as filter

@ingridhovden


Updated jshrink to the 1.0 line of code.

JShrink has reached a stable version, and has had many bug fixes and optimizations. This update allows munee to take advantage of these improvements.

Thank you!


fix(/components/sidebar): remove Sidebar.Collapse parent Tooltip on child hover

Changes

[x] refactor(/components/dropdown): remove extraneous @ts-expect-error
[x] fix(/components/sidebar): remove SidebarCollapse parent Tooltip o…

Description
In Sidebars with a Sidebar.Collapse, when you hover over a child Sidebar.Item,
the parent's Tooltip is still displayed, erroneously.
Now only the child's Tooltip comes up when you hover over that child.
This new behavior causes a different problem where the parent's Tooltip isn't
visible at all, even when you hover over the parent button. To solve that, we just
add an HTML title attribute to the button.
Fixes #429
Type of change
Please delete options that are not relevant.

[x] Bug fix (non-breaking change which fixes an issue)

Breaking changes
N/A
How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

[x] Test A: Navigate to the Sidebar section in Storybook. Select the Multi-level dropdown example. Expand the collapse. Set the Sidebar to collapsed. Hover over the child item inside the collapse to observe the new, correct behavior.

Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[x] Any dependent changes have been merged and published in downstream modules
[x] I have checked my code and corrected any misspellings

Weird, I'm not sure why but this issue with my refactor commit is coming up in the CI, but it currently blocks me from running yarn build:lib if I don't remove the @ts-expect-error from my local machine. I'll just remove that for now.


Some small changes and fixes

Applied some clippy suggestions and made the interpreter code more readable.
Also a bug fix for some ops where the emulator would report too many operands when there was in fact no operands.

Thanks, I kinda forgot clippy was a thing :P. also I like that #operand solution


create an "allowing mode" (or similar) -- for quickly migrating data with default mappings and reference data

No extra information on this - closing


Buffer-create throw error

Example,
something:buffer-create.

also,
something:buffer-value.

something:buffer-create is not a good example. According to the documentation, it has to go together with parentheses, even if those are empty.
Opening a ticket for something:buffer-value, since parentheses are optional.
Registered: https://github.com/usagi-coffee/tree-sitter-abl/issues/101

Fixed with version 0.0.45 of tree-sitter-abl


Add BOLT_SKIP_GEN env var for faster commands

Changes

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

Current dependencies on/for this PR:

main

PR #310

PR #311

PR #312   👈

PR #313

This stack of pull requests is managed by Graphite.


[CI] Verify Linked Issue Action bot is noisy

🐛 Bug

the bot (verifying if pr contains issue number) is noisy as observed in #10012.
To Reproduce
Submit a PR w/o any issue number.
Expected behavior
Bot comments only one time.
Additional context
#9575 tried to fix this but it seems it didn't?
cc: @Borda

Hey @akihironitta,
We will implement an improved version of the bot after v1.5. The spam is still a good thing to avoid un-tracked work.
Best,
T.C

Closed in #11338.


[Bug Report] table slot append outside table element

Element UI version
2.0.7
OS/Browsers version
Chrome / Version 63.0.3239.84
Vue version
2.5.8
Reproduction Link
https://jsfiddle.net/L7z6nntx/
Steps to reproduce
Run the JSFiddle and you'll notice the columns do not align.
Prior to version 2.0.0, the rows/columns would be within the table element.
What is Expected?
I expect the rows/columns to be within the table element
The following link demonstrates the appropriate results using version 1.3.5
https://jsfiddle.net/L7z6nntx/1/
What is actually happening?
The rows/columns are outside of table element losing the formatting and alignment

append slot is not designed for more table rows. It's suitable for other components, like an infinite loading indicator. This is intended and a breaking change of 2.0.


Video doesn't start or stops after a few seconds in Chrome

Description
Video doesn't start or stops after a few seconds on Chrome.
Chrome version: 58.0.3029.110 (64-bit) Mac.
Steps to reproduce the issue:

Publish local stream with Chrome
Subscribe local stream with Chrome
Wait few seconds

Describe the results you received:
Video stream ends, audio keeps going.
Observed behavior on Erizo Agent right after video stops working:
[erizo-XXX] date 15:43:59,947  - WARN [0x...] rtp.PliPacerHandler - id:<PHONE_NUMBER>70561200,  message: Timed out waiting for a keyframe
[erizo-XXX] date 15:44:02,948  - WARN [0x...] rtp.PliPacerHandler - id:<PHONE_NUMBER>70561200,  message: Timed out waiting for a keyframe
[erizo-XXX] date 15:44:05,949  - WARN [0x...] rtp.PliPacerHandler - id:<PHONE_NUMBER>70561200,  message: Timed out waiting for a keyframe
[erizo-XXX] date 15:44:08,949  - WARN [0x...] rtp.PliPacerHandler - id:<PHONE_NUMBER>70561200,  message: Timed out waiting for a keyframe
[erizo-XXX] date 15:44:10,066  - DEBUG [0x...] WebRtcConnection - id:<PHONE_NUMBER>70561200,  unknownSSRC:<PHONE_NUMBER>, localVideoSSRC:<PHONE_NUMBER>, localAudioSSRC:<PHONE_NUMBER>
[erizo-XXX] date 15:44:11,481  - DEBUG [0x...] WebRtcConnection - id:<PHONE_NUMBER>70561200,  unknownSSRC:<PHONE_NUMBER>, localVideoSSRC:<PHONE_NUMBER>, localAudioSSRC:<PHONE_NUMBER>
[erizo-XXX] date 15:44:11,750  - DEBUG [0x...] WebRtcConnection - id:<PHONE_NUMBER>70561200,  unknownSSRC:<PHONE_NUMBER>, localVideoSSRC:<PHONE_NUMBER>, localAudioSSRC:<PHONE_NUMBER>
Describe the results you expected:
Receive video/audio normally.
Licode commit/release where the issue is happening
Licode version: 90355f6 - May 11 2017
Additional environment details (Local, AWS, Docker, etc.):
AWS, Docker, Coturn.

Not valid.


[SPARK-23146][K8S] Support client mode.

What changes were proposed in this pull request?
Support client mode for the Kubernetes scheduler.
Client mode works more or less identically to cluster mode. However, in client mode, the Spark Context needs to be manually bootstrapped with certain properties which would have otherwise been set up by spark-submit in cluster mode. Specifically:

If the user doesn't provide a driver pod name, we don't add an owner reference. This is for usage when the driver is not running in a pod in the cluster. In such a case, the driver can only provide a best effort to clean up the executors when the driver exits, but cleaning up the resources is not guaranteed. The executor JVMs should exit if the driver JVM exits, but the pods will still remain in the cluster in a COMPLETED or FAILED state.
The user must provide a host (spark.driver.host) and port (spark.driver.port) that the executors can connect to. When using spark-submit in cluster mode, spark-submit generates the headless service automatically; in client mode, the user is responsible for setting up their own connectivity.

We also change the authentication configuration prefixes for client mode.
How was this patch tested?
Adding an integration test to exercise client mode support.

TODO - finish verifying the integration test, and docs.

test this please

Test build #92869 has started for PR 21748 at commit 19618aa.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/857/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/858/
Test PASSed.

Test build #92869 has finished for PR 21748 at commit 19618aa.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92869/
Test PASSed.

retest this please

Test build #92873 has started for PR 21748 at commit 19618aa.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/861/

Test build #92875 has started for PR 21748 at commit 94ed1cc.

Test build #92873 has finished for PR 21748 at commit 19618aa.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92873/
Test PASSed.

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/861/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/861/
Test FAILed.

Test build #92878 has started for PR 21748 at commit 560993e.

Test build #92875 has finished for PR 21748 at commit 94ed1cc.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92875/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/862/
Test FAILed.

Test build #92879 has started for PR 21748 at commit e961fd3.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/863/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/865/
Test FAILed.

Test build #92880 has started for PR 21748 at commit a00561f.

Test build #92878 has finished for PR 21748 at commit 560993e.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92878/
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/866/
Test FAILed.

Merged build finished. Test FAILed.

Test build #92879 has finished for PR 21748 at commit e961fd3.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92879/
Test FAILed.

Test build #92881 has started for PR 21748 at commit a2609b0.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/867/
Test FAILed.

Test build #92881 has finished for PR 21748 at commit a2609b0.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92881/
Test FAILed.

Test build #92882 has finished for PR 21748 at commit 97f1284.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92882/
Test FAILed.

retest this please

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92883/
Test FAILed.

Test build #92884 has started for PR 21748 at commit 97f1284.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/868/
Test FAILed.

Test build #92887 has started for PR 21748 at commit 2205220.

Test build #92884 has finished for PR 21748 at commit 97f1284.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92884/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/870/
Test FAILed.

Test build #92887 has finished for PR 21748 at commit 2205220.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class JavaSummarizerExample
trait ComplexTypeMergingExpression extends Expression
sealed trait MultipleWatermarkPolicy
case class WatermarkTracker(policy: MultipleWatermarkPolicy) extends Logging

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92887/
Test FAILed.

Test build #92892 has started for PR 21748 at commit 64cc39b.

i'm going to kill the ubuntu build and reboot the worker.  i'll retrigger when it's back.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/874/
Test FAILed.

Test build #92892 has finished for PR 21748 at commit 64cc39b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92892/
Test PASSed.

test this please

Test build #92893 has started for PR 21748 at commit 64cc39b.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/875/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/875/
Test FAILed.

Test build #92895 has started for PR 21748 at commit 0f6ad73.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/877/

Test build #92895 has finished for PR 21748 at commit 0f6ad73.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92895/
Test PASSed.

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/877/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/877/
Test FAILed.

test this please

Test build #92898 has started for PR 21748 at commit 0f6ad73.

Test build #92900 has started for PR 21748 at commit 88a9d7f.

Test build #92898 has finished for PR 21748 at commit 0f6ad73.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92898/
Test PASSed.

Test build #92900 has finished for PR 21748 at commit 88a9d7f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92900/
Test PASSed.

retest this please

Test build #92902 has started for PR 21748 at commit 88a9d7f.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/881/

Test build #92902 has finished for PR 21748 at commit 88a9d7f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92902/
Test PASSed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/883/
Test FAILed.

test this please

Test build #92959 has started for PR 21748 at commit 88a9d7f.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/916/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/916/

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/916/
Test FAILed.

Merged build finished. Test FAILed.

Test build #92959 has finished for PR 21748 at commit 88a9d7f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92959/
Test PASSed.

test this please

Test build #92977 has started for PR 21748 at commit 88a9d7f.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/931/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/931/

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/931/
Test FAILed.

Merged build finished. Test FAILed.

test this please

Test build #92979 has started for PR 21748 at commit 88a9d7f.

Test build #92977 has finished for PR 21748 at commit 88a9d7f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92977/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/932/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/932/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/932/
Test FAILed.

Test build #92979 has finished for PR 21748 at commit 88a9d7f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92979/
Test PASSed.

Merged build finished. Test PASSed.

test this please

Test build #92982 has started for PR 21748 at commit 88a9d7f.

Can we remove this as a part of this PR? https://github.com/apache/spark/blob/master/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh#L87 - it seems to set a client mode by default..

@mrow4a you still need this to run in client mode. Basically the way the docker image entrypoint works is that it assumes it runs in cluster mode. What I imagine client mode doing would be using the same image but using the K8s APi to change the command. See e.g. the new integration test that was written.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/934/

Test build #92982 has finished for PR 21748 at commit 88a9d7f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92982/
Test PASSed.

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/934/

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/934/
Test FAILed.

Merged build finished. Test FAILed.

Test build #92985 has started for PR 21748 at commit 5785ce7.

Test build #92985 has finished for PR 21748 at commit 5785ce7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92985/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/937/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/937/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/937/
Test FAILed.

Test build #92988 has started for PR 21748 at commit 75db063.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/939/

@mccheah Is this ready for review?

Test build #92988 has finished for PR 21748 at commit 75db063.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92988/
Test PASSed.

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/939/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/939/
Test FAILed.

Integration test isn't ready for review but the main code is.

Test build #92990 has started for PR 21748 at commit 846f093.

Test build #92991 has started for PR 21748 at commit 69bf2a4.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/940/

Test build #92990 has finished for PR 21748 at commit 846f093.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92990/
Test PASSed.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/940/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/940/
Test PASSed.

Test build #92991 has finished for PR 21748 at commit 69bf2a4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92991/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/941/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/941/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/941/
Test PASSed.

Ok, integration test is green. Probably should add some docs though - following up in a moment.

Test build #92994 has started for PR 21748 at commit ee5c267.

Ok documentation is done, this is fully ready for review now.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/944/

Test build #92994 has finished for PR 21748 at commit ee5c267.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92994/
Test PASSed.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/944/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/944/
Test PASSed.

Test build #93124 has started for PR 21748 at commit bd102b3.

Comments are addressed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/1017/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/1017/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1017/
Test FAILed.

Test build #93124 has finished for PR 21748 at commit bd102b3.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class AvroDeserializer(rootAvroType: Schema, rootCatalystType: DataType)
 sealed trait CatalystDataUpdater
 final class RowUpdater(row: InternalRow) extends CatalystDataUpdater
 final class ArrayDataUpdater(array: ArrayData) extends CatalystDataUpdater
  class SerializableConfiguration(@transient var value: Configuration)
class AvroSerializer(rootCatalystType: DataType, rootAvroType: Schema, nullable: Boolean)
  case class SchemaType(dataType: DataType, nullable: Boolean)
class IncompatibleSchemaException(msg: String, ex: Throwable = null) extends Exception(msg, ex)
class SerializableSchema(@transient var value: Schema)
 implicit class AvroDataFrameWriter[T](writer: DataFrameWriter[T])
 implicit class AvroDataFrameReader(reader: DataFrameReader)
class KMeansModel (@Since(\"1.0.0\") val clusterCenters: Array[Vector],
case class Least(children: Seq[Expression]) extends ComplexTypeMergingExpression
case class Greatest(children: Seq[Expression]) extends ComplexTypeMergingExpression
case class Size(child: Expression) extends UnaryExpression with ExpectsInputTypes
case class MapConcat(children: Seq[Expression]) extends ComplexTypeMergingExpression
case class Concat(children: Seq[Expression]) extends ComplexTypeMergingExpression
abstract class ArraySetLike extends BinaryArrayExpressionWithImplicitCast
case class ArrayUnion(left: Expression, right: Expression) extends ArraySetLike
case class Coalesce(children: Seq[Expression]) extends ComplexTypeMergingExpression

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93124/
Test PASSed.

Test build #93133 has started for PR 21748 at commit 37537c6.

Test build #93134 has started for PR 21748 at commit 0db7a79.

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/1022/

Merged build finished. Test FAILed.

Test build #93134 has finished for PR 21748 at commit 0db7a79.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93134/
Test PASSed.

test this please

Test build #93139 has started for PR 21748 at commit 0db7a79.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/1027/

Test build #93139 has finished for PR 21748 at commit 0db7a79.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93139/
Test PASSed.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/1027/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1027/
Test PASSed.

Filed https://issues.apache.org/jira/browse/SPARK-24825 to track the test problem.

thanks.  and yes, i did indeed need to run mvn install to get this green.
On Mon, Jul 16, 2018 at 10:31 PM, mccheah<EMAIL_ADDRESS>wrote:

Filed https://issues.apache.org/jira/browse/SPARK-24825 to track the test
problem.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/apache/spark/pull/21748#issuecomment-405401503, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABiDrJpozJWsvbln1QbHVcYb41YSPqjLks5uHRRSgaJpZM4VLmBR
.

@mccheah Tried this PR in client-mode In-Cluster on minikube v0.25.2: Exectuors are started but directly removed. As the start/remove is so fast, I can hardly see logs (and the logs I have seen don't show any stacktrace). Maybe something in my env?
The config I have for the client mode is:
# DRIVER_POD_NAME=$HOSTNAME
  --conf spark.kubernetes.driver.pod.name="$DRIVER_POD_NAME" \
  --conf spark.driver.host="$DRIVER_POD_NAME" \
  --conf spark.driver.port=7077 \
  --conf spark.driver.blockManager.port=10000 \

The driver log is:
2018-07-18 14:29:43 INFO  SparkContext:54 - Created broadcast 0 from broadcast at DAGScheduler.scala:1039
2018-07-18 14:29:43 INFO  DAGScheduler:54 - Submitting 10 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9))
2018-07-18 14:29:43 INFO  TaskSchedulerImpl:54 - Adding task set 0.0 with 10 tasks
2018-07-18 14:29:45 INFO  ExecutorPodsAllocator:54 - Going to request 1 executors from Kubernetes.
2018-07-18 14:29:46 INFO  BlockManagerMasterEndpoint:54 - Trying to remove executor 5 from BlockManagerMaster.
2018-07-18 14:29:46 INFO  BlockManagerMaster:54 - Removal of executor 5 requested
2018-07-18 14:29:46 INFO  KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint:54 - Asked to remove non-existent executor 5
2018-07-18 14:29:52 INFO  BlockManagerMaster:54 - Removal of executor 6 requested
2018-07-18 14:29:52 INFO  KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint:54 - Asked to remove non-existent executor 6
2018-07-18 14:29:52 INFO  BlockManagerMasterEndpoint:54 - Trying to remove executor 6 from BlockManagerMaster.
2018-07-18 14:29:52 INFO  ExecutorPodsAllocator:54 - Going to request 1 executors from Kubernetes.
2018-07-18 14:29:55 INFO  BlockManagerMaster:54 - Removal of executor 7 requested
2018-07-18 14:29:55 INFO  KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint:54 - Asked to remove non-existent executor 7

My guess is that the executor is failing to contact the driver - I don't think you can use the pod name as the hostname in minikube. Can you create a headless service to bind your pod to a stable hostname and try again?

As @mccheah mentioned, you probably need a headless service that exposes port 7077 and 10000 to connect to the driver using the driver pod name. I think using the driver pod name should work given the service.

Thx @mccheah  and @liyinan926. It now works with an headless service using spark.driver.host=spark-driver-service and spark.kubernetes.driver.pod.name=spark-pod
Two more questions:

I still have an issue with Out-Cluster: Executors are also killed on start. Is there something I should configure to make it work?
With the PR I had developer on the fork (long time ago), I didn't have to create by myself the headless service. I have looked at the difference between fork and apache, but don't see where it comes from. In short, as a user, it would be better if I hadn't to create by myself the headless service, especially if we consider an notebook environment where a WEB Server running in a pod would launch multiple REPL (in client mode), I see practical issue to manage the assignment of the exposed ports (you can not assign a port which is already assigned). Thoughts?

For info, the specs of the headless service which is needed to run In-Cluster.
apiVersion: v1
kind: Service
metadata:
  name: spark-driver-service
spec:
  clusterIP: None
  ports:
  - port: 7077
    name: spark-driver-port
  - port: 10000
    name: spark-driver-blockmanager-block
  selector:
    run: spark-pod

Test build #93244 has started for PR 21748 at commit d286de1.

@echarles then you'd probably want more information such as the logs of the executors, though I'd imagine that one would have trouble getting those given that the executor exits so quickly. But maybe a kubectl get pods -n <namespace> <some labels that match your executors to get a bit more information as to what is going on.

Also make sure your driver can actually allocate the port specified by spark.driver.port?

Test build #93243 has finished for PR 21748 at commit d286de1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1101/
Test PASSed.

Can you point in the fork where the submission client is create the headless service? (just to help me understand the internals).

https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/DriverServiceFeatureStep.scala

Can you point in the fork where the submission client is create the headless service? (just to help me understand the internals)

Btw If we stick to this manual approach, the need for the manual headless service should be documented.

@echarles we create the headless service in spark-submit as part of spark-submit only for cluster mode: https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/DriverServiceFeatureStep.scala#L69
Note that we only invoke any of the feature steps and the entry point of KubernetesClientApplication if we run in cluster mode. If we run in client mode, we enter directly into the user's main class, or, the user is in a process that just created a Spark context from scratch with the right master URL (i.e. new SparkContext(k8s://my-master:8443)). If you wanted to create the headless service in client mode, you'd have to do it when instantiating the KubernetesClusterSchedulerBackend somewhere, probably before creating the ExecutorPodsAllocator so that you'd set spark.driver.host and spark.driver.port properly when telling the created executors where to find the driver via the --driver-url argument.
I've deferred implementing this code path. The documentation for using a headless service is only suggested, but not mentioned as a hard requirement: https://github.com/apache/spark/pull/21748/files#diff-b5527f236b253e0d9f5db5164bdb43e9R131. I didn't put this as a hard requirement because I could imagine some users wanting to not specifically use a headless service for this; perhaps they want to use a full Service object and share that service object with ports to be exposed for other reachable endpoints their pod exposes.

@mccheah @liyinan926 it is now working on my env in Out-Cluster. I was failing because I forgot to remove the spark.kubernetes.driver.pod.name props. In general, configuration is tedious and we have to fully document with examples the different cases to be sure user don't hit issues.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/1102/

@mccheah @liyinan926 the code base has largely changed from the fork, but at that time it was working fine without having to manually create any headless service. Not sure why... but sure it was working with a patch that was not touching that part.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/1102/

@echarles there was a change in Spark recently in how the driver self-discovered its hostname by default, if I am not mistaken. Can't recall the exact patch. I remember that change specifically prompting us to set up the headless service for cluster mode.

Sounds fine. How does the documentation look now in that regard?

Sounds fine. How does the documentation look now in that regard?

I think we should add the following: 1) be explicit about the OwnerReference when there's a driver pod, and 2) guidance on the label selector to use on the headless service for the driver pod. 2) is relevant as it's important to make sure the headless only includes the desired driver pod, so including some label(s) in the label selector that's unique to the application is important. We should recommend in the doc that users need to add such label to both the driver pod and to the label selector of the service.

Note that we only invoke any of the feature steps and the entry point of KubernetesClientApplication if we run in cluster mode. If we run in client mode, we enter directly into the user's main class, or, the user is in a process that just created a Spark context from scratch with the right master URL (i.e. new SparkContext(k8s://my-master:8443)). If you wanted to create the headless service in client mode, you'd have to do it when instantiating the KubernetesClusterSchedulerBackend somewhere, probably before creating the ExecutorPodsAllocator so that you'd set spark.driver.host and spark.driver.port properly when telling the created executors where to find the driver via the --driver-url argument.

For Out-Cluster we don't need any headless service, but for In-Cluster the user has to ensure that the driver is accessible via a service (headless or not) - I understand it may be preferable to align on spark other managers and to have coherent behaviour for k8s in/out cluster, but I still believe that we have to make life easy to users and devops.
The notebook case is self explaining : For Zeppelin you may have a single pod (Web Server) that will create multiple client sessions (this is a common supported scenario), Jupyter with Toree may be configured to support multiple REPL (and if it not today, it will come on day). You also have the case of current livy which is a single REST Server managing a pool of REPL.
With current PR, we push to all those 3rd party consumer the responsibility to create services, but all the hard work to assign and maintain ports to avoid collision on the single pod with multiple drivers.
What about having a property that would enable/disable that headless service creation (could be disabled by default) - This is not yet the perfect solution as the user still would have to pass the ports and ensure there is no clash.

PS: Actually, there would even be no issue with the port assignment as Spark knows which ports he will be using, so he can create the headless service with the correct ports for the user.

@echarles I don't think we should be special-casing Kubernetes here as being any different from the other cluster managers. The main point of client mode is that the driver is running locally and we make no assumptions about network connectivity. Deploying notebooks on other cluster managers which have network isolation between the driver and the executors would require the user to manually configure their own connectivity too.
There's another problem in creating the headless service here: when you create a service, you have to back that service with pods that have certain labels; that is, a label selector. However, labels are not unique for a given pod; multiple pods can have the same label key-value pairs. Now, when Spark Submit has explicit control over the pod that is created as well as the service, we make it such that the driver pod has a unique label corresponding to an application id, and then the service can be ensured to be backed by only that driver pod. If we don't control the creation of the driver pod, we can't guarantee that whatever service we create would route to only the driver pod running the application in question.
It is therefore unclear what the label selector would be for the headless service we create automatically. You could allow the user to have some control over the configuration of the headless service. For example, we could have a setting indicating the label selector the headless service should be created with. Or, the documentation can specifically require the driver pod to have a set of labels which are strictly unique to that pod in that namespace. In both cases, I would argue that we end up with the same complexity and mental burden, if not more so, than having the user deploy a headless service that gives their driver pod (and only that driver pod) a stable hostname.

About selecting the pod with labels, another approach I have taken is simply using the name of the driver pod, a bit like I have done with the following deployment (so no need to ensure labels - the ports are the ports assigned by spark that the code can retrieve).

I don't think you can back a service with a selector that's a pod's name, but someone with more knowledge of the Service API might be able to correct me here. I was under the impression one had to use labels. In your example, the service would match any pod with the label key of run being equal to spark-pod, which isn't guaranteed to map to a single unique pod. In spark-submit we set spark-app-id to a unique identifier.

If I compare with yarn-client with all nodes on the same LAN

But if you run a YARN application with the driver not being on the same network, then the user has to set up their own connectivity. In Kubernetes that kind of networking setup happens to come up more often, perhaps, but it's not enough reason to introduce the complexity of it.
Another situation where we want the driver to not be making the headless service is in a world where the driver shouldn't have permission to create service objects, but can have permission to create pod objects. Adding a flag allowing the driver to create the headless service would implicitly change the required permissions of the application. This is more work to document and more for the application writer to consider.

I don't think you can back a service with a selector that's a pod's name, but someone with more knowledge of the Service API might be able to correct me here. I was under the impression one had to use labels.

Yes, the service gets its endpoints by matching its label selector against labels on the pods so it's critical to have matching labels. Another tenable solution is for the driver backend code to get the pod object of the driver and sets the label selector of the service based on the labels on the driver pod.

don't think we should be special-casing Kubernetes here as being any different from the other cluster managers. The main point of client mode is that the driver is running locally and we make no assumptions about network connectivity. Deploying notebooks on other cluster managers which have network isolation between the driver and the executors would require the user to manually configure their own connectivity too.

I would back-step a bit and argue that Kubernetes is unique in this regard because of the need of a service for in-cluster client mode to work and the service is a Kubernetes resource. For this reason, I don't think it makes no sense for us to do a bit more for the users if technically it's not impossible to do so.

Yes, the service gets its endpoints by matching its label selector against labels on the pods so it's critical to have matching labels. Another tenable solution is for the driver backend code to get the pod object of the driver and sets the label selector of the service based on the labels on the driver pod.

The problem is that the driver's labels might not be unique to that driver, which therefore would require the user to assign their own unique labels or for us to patch the driver pod in-place to assign it a unique label. Neither seem ideal.

The problem is that the driver's labels might not be unique to that driver, which therefore would require the user to assign their own unique labels or for us to patch the driver pod in-place to assign it a unique label. Neither seem ideal.

This exact problem still exists if users would have to create the service themselves: they would still need to make sure to use label(s) that can tell the driver/service combination apart from other apps.

Got you points. About labels, right, we could take the road of the code that creates labels on its own pod. To ensure uniqueness, we could use the spark-app-id as key (if it maps the requirement of a label name) and driver as value (let's be imaginative...).
About the permissions, this would have to be documented: I don't see an issue for a manager to give a pod permission to create services - If security prohibits this, up to the administrator to create them before hand....
I just feel that we risk to miss some use cases if we don't ship an option for this (the default behaviour could be the safer with the need to create manually the service)

Got you points. About labels, right, we could take the road of the code that creates labels on its own pod. To ensure uniqueness, we could use the spark-app-id as key (if it maps the requirement of a label name) and driver as value (let's be imaginative...).

Label spark-app-id is only set if spark-submit goes through the steps to create the driver pod so doesn't apply in this case.

Label spark-app-id is only set if spark-submit goes through the steps to create the driver pod so doesn't apply in this case.

In that case, the client process could create its own spark-client-app-id...

In that case, the client process could create its own spark-client-app-id...

Yes, and that's what my point above is about. Regardless of how the driver pod is created and managed, users need to make sure it contains a label that is unique for the pod/service combination to work.

I think taking a step back, it seems unwise more so to be making any assumptions about the location in which a driver is running in client mode. Client mode is simply just saying that the application driver is running "locally", wherever locally is. That is why I suggested also to remove the driver's knowledge of the driver pod name and to remove the owner reference concept entirely. As soon as we start to make assumptions in one place, it may (though not necessarily) encourage us to rely on those assumptions to create more forks and complexities in the code in other places. That's a precedent we likely do not want to set.

Yes, and that's what my point above is about. Regardless of how the driver pod is created and managed, users need to make sure it contains a label that is unique for the pod/service combination to work.

In one case the requirement for the unique label is explicit - the user has created their service and their pod manually and have assigned the labels accordingly. In another case, the user must know to assign the unique label but they would only know to do so from having read the documentation on deploying Spark. Explicit requirements that are well defined by the API are preferable to implicit requirements.

That is why I suggested also to remove the driver's knowledge of the driver pod name and to remove the owner reference concept entirely.

While, not worrying about the driver pod name and owner reference doesn't eliminate the need of a headless service if the driver is indeed running in a pod. So the question is should the k8s scheduler backend be responsible for creating the service or not if the driver is running in a pod. I'm personally leaning towards doing that for the user.

I'm personally leaning towards doing that for the user.

Especially if the user is a data scientist behind his notebook launching a paragraph which is supposed to instanciate a Spark REPL as fast as possible and without having to call his support center and to launch a script. And if a script would be needed, we'd better ship the logic in the core of Spark.

We discussed this offline. After some experimentation, we concluded that it's not actually straightforward to set up the headless service in the Kubernetes scheduler code in client mode, which would be where we'd have to put this code. The problem is that before the scheduler starts up, the driver needs to bind to the host given by spark.driver.host. But if that hostname is tied to a headless service and the headless service does not exist, the hostname bind will fail.
There's a few ways to work around this, but all of them seem risky to put in a first pass at this patch.
Additionally, it's not clear if the scheduler code should be opinionated about configuring network connectivity for the driver. Client mode is just a definition of the driver running in a local process - Spark currently doesn't make any assumptions about what that local process is, whether it's in a Kubernetes pod or not. Contrast this with cluster mode where the driver should know it's running in a pod as submitted by KubernetesClientApplication, and KubernetesClientApplication can know to set up its headless service.
Therefore we are not going to have the driver set up the headless service in this patch. If we decide later on that creating the headless service is the right thing to do, we can introduce that functionality in a separate patch.
I'm going to update the docs and hope to merge this by the end of the day on Monday, July 23. Let us know if there are any additional concerns. Thanks @liyinan926 @echarles.

Test build #93357 has started for PR 21748 at commit 086747e.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/1180/

Test build #93358 has started for PR 21748 at commit d90f753.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/1180/

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1180/
Test PASSed.


Compiling the game (error: Uncaught exception)

Describe your problem here.
I'm not really sure what happened, as far as I was aware the compiling worked beforehand but when I tried compiling tonight I was thrown with this error that I cant seem to find a fix online. Does anyone know how I can fix this? Thanks in advance!

Are you modding a build from source or with Lua?
Source
What is your build target?
Windows x64
Did you edit anything in this build? If so, mention or summarize your changes.
No response

did you add anything in the example_mods folder

update to haxe 4.2.5

I'm positive I have it up to date but I'll double check just in case

no you don't

I'll make sure to update it now, thank you :)

if it worked close the issue

I haven't tested it yet cause I'm not home atm but I will do if it did fix it though :)

so weirdly enough haxe was already installed but I think on another harddrive so my program that was on my main ssd couldnt access it (I assume) so I reinstalled haxe 4.2.5 like you said and did all the haxelib installs all over again and yet I was prompted with the same issue any other suggestions? :(

OK, I think your trying to compile from outside the project folder, not in the root of it, if your trying to run that where there's no project.XML file and no source folder, then there's your problem.
See: https://community.haxe.org/t/error-from-the-command-haxelib-run-lime-test-windows/3010

I usually compile from within this folder and it always works is this where im meant to compile? It odes have the project.xml and everything I thought it needed but still prompts the same error

yeah, and if it still giving you the same error, then I'm out of ideas, your haxe might just be completely screwed up.

I'll try look into it a bit more later but thanks you for the time and effort you took to help :)

FIGURED IT OUT! And so the problem for people who are also having the same issue is that no files are allowed to be edited outside of the mods folder (AT LEAST WHAT IVE GATHERED AND JUST FIGURED OUT RIGHT AT THIS MOMENT DOSENT MEAN ITS CORRECT IM JUST TRYING TO HELP). I simply added my own dataminershaven folder with pictures in it and thats all it took to completely stop me from compiling, so make sure anything you add (i assume add only) is only in the mods folder anything in preload is not takin so lightly by the compiling process clearly.


Possible inconsistency with default config.source value

Hello!
I reached out to support a while back and wanted to share the findings.
Here's the background

We're working on a cross-platform React Native app -- so web and native report to the same RUM app -- and I was trying to figure out the best way to distinguish between >>the native app and web.
Currently there's a "mobile" facet which is great! BUT it includes mobile web -- and I am trying to get just the native app sessions.
There's also "OS" facet, but that includes mobile web results, as well.
So far the best I've been able to figure out is to add -@browser.name:* to the search, to exclude any sessions with a defined browser.
But perhaps I have overlooked something important?
My hope is that we can figure out a way to quickly toggle between native and web platforms -- perhaps a "Platform" facet?

After taking a look at your case, I think that using the source facet can achieve your requirement, would you be able to confirm that?

The source property definitely sounds like what Im looking for -- but it appears as tho android is being grouped as "react-native" [see: screenshot]

I looked around the codebase and it looks like the default is being defined as react-native here
So I guess the actual question is: "why is ios appearing as the source at all?"*
I think the solution is probably just to manually define the source property -- but I thought I'd check here to see if the expected behavior should be to autodetect the platform by default.
Thanks!

Version
Been staying up to date withe release candidates:
    "@datadog/browser-rum": "3.2.0",
    "@datadog/mobile-react-native": "1.0.0-rc8",
    "@datadog/mobile-react-navigation": "1.0.0-rc8",

Hello @leggomuhgreggo. Thanks for opening the issue. It was actually a bug in a native part of our SDK which was fixed recently in https://github.com/DataDog/dd-sdk-ios/pull/832/commits/ae560c22ab8ed75d3c14e483135f20b922a4baca. The fix should be available in our latest React Native SDK version 1.0.0-rc8 release.
Can you please check that you don't have this issue for your app version using 1.0.0-rc8 and it happens only for app versions coming with earlier React Native SDK versions?

Looks good! Thanks for the context.


meminfo: buflen identify the left of buffer

Summary
It is enought to only checking the buflen's avaiable or not. (total size means total copied size)
Impact
Testing

@sdc-g please squash the change into one patch with:
git rebase --interactive HEAD~4
change pick to squash for the last commit
git push -f origin

@sdc-g please squash the change into one patch with: git rebase --interactive HEAD~4 change pick to squash for the last commit git push -f origin

@xiaoxiang781216 thanks, updated.


toolchain: compile newlib with optimizations

Ooops...
I have broken it here:
https://github.com/DragonMinded/libdragon/commit/1c522f2da5928bb5bb79ac76436c2186091fef75#diff-4cdfece525412cfc651464566e6bc1e6a8c333a3d67831a751e8c78f3a82ab69L93-R93
but I had removed CFLAGS="-O2" because CFLAGS was being ignored in the compilation of Newlib (I knew because I had tried to add the -DHAVE_ASSERT_FUNC in CFLAGS and it was being ignored). So I moved to CFLAGS_FOR_TARGET and I assumed that nothing was necessary.
It looks like for unknown (to me) reasons, CFLAGS="-O2" was indeed having an effect.
I've also took the chance to modify the CI to upload all the built ROMs as artifacts, which costs nothing and since we build them, we can as well make them available for download from the CI.

I remember the day we decided to remove that, how did you find out?
PR LGTM.