🪲

Sample Code/Statement If you can, please submit the SQL statement or Python code snippet, or a representative example using the sample datasets.
SELECT * FROM $planets INNER JOIN (SELECT * FROM UNNEST((1, 2, 3)) AS id) AS PID USING(id)

NoOpNode object has no attribute 'columns'

Additional context Add any other context about the problem here, for example what you have done to try to diagnose or workaround the problem.

SELECT * FROM $planets INNER JOIN UNNEST((1, 2, 3)) AS id USING(id)


Test connection works, but pipeline job doesn't send or give error

Your checklist for this issue

[x] Jenkins version

[x] Plugin version

[ ] OS

Description
Jenkins v2.164.2
Plugin v2.23
don't know OS but can find out if important
I've got the plugin working to send to Slack when configuring globally - the "Test Connection" button works fine, it posts. But anything I try from a pipeline does nothing but
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Clone)
[Pipeline] echo
Slack message stub method
	 >>>> [channel:#jenkins, color:good, message:TEST]

in the job console
Code goes like this
// stages/stage/steps for context
    stages {
        stage('Clone') {
            steps {
                slackSend (channel: '#jenkins',
                    color: 'good',
                    message: "TEST")

I can't find any references via google for this "Slack message stub method" thing so I don't know what that means. All I know is I've tried putting this in all kinds of places in various styles (with and without parentheses), without or without a channel, with the token and subdomain specified, without, etc, no luck. I'm at wits end after throwing half a day at this. Nothing in the Jenkins logs either. Help!

You have a space between slackSend and the parentheses.
Also, I'd suggest this indentation.
slackSend(
  channel: '#jenkins',
  color: 'good',
  message: "TEST"
)

@casz I can give it a try, but I've tried single line, various formatting, I figured if it was a formatting error Groovy would yell at me.

Are you using shared libraries? Did someone already use the slackSend symbol?

Can you post the full pipeline / try run it with just the slackSend bit in it and not your other code

I'm having to look up Shared Libraries so I think not, though maybe someone added it. How can I check that?

#!groovy

pipeline {

    agent any

    stages {
        stage('Test') {
            steps {
                slackSend(
                  channel: '#jenkins',
                  color: 'good',
                  message: "TEST"
                )
            }
        }
    }
}

Gives this
First time build. Skipping changelog.
[Pipeline] }
[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Test)
[Pipeline] echo
Slack message stub method
	 >>>> [channel:#jenkins, color:good, message:TEST]
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline

It can only come down to another plugin or shared libraries doing something funky.
Shared libraries can be configured on a folder level or globally and be loaded implicitly.
Please read here on how these could be configured and where they are loaded from:
https://jenkins.io/doc/book/pipeline/shared-libraries/#global-shared-libraries

The echo step gives it away that it sounds like a shared library

Could you share the full build log? if your pipeline does more than one SCM checkout your definitely loading a shared library :cold_sweat:

Thanks for the quick replies folks.
Here's the full build log, some items redacted to be careful (user & private repo name)
Started by user Robert Gerald Porter
16:13:30 Connecting to https://api.github.com using user_redacted/****** (Github Pipeline with CI integration)
Obtained Jenkinsfile from 53e2deff9731d9c7674db3f234c647546e02521e
Running in Durability level: MAX_SURVIVABILITY
Loading library<EMAIL_ADDRESS>No credentials specified
Cloning the remote Git repository
Cloning repository https://github.com/base2Services/ciinabox-pipelines.git
 > git init /var/jenkins_home/workspace/experiment_basic-e2e-jenkinsfile@libs/ciinabox # timeout=10
Fetching upstream changes from https://github.com/base2Services/ciinabox-pipelines.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/base2Services/ciinabox-pipelines.git +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/base2Services/ciinabox-pipelines.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/base2Services/ciinabox-pipelines.git # timeout=10
Fetching upstream changes from https://github.com/base2Services/ciinabox-pipelines.git
 > git fetch --tags --progress https://github.com/base2Services/ciinabox-pipelines.git +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/tags/version-0.1^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/refs/tags/version-0.1^{commit} # timeout=10
Checking out Revision 1900b4b3357515c3a624bdf1463910c17b48f8a1 (refs/tags/version-0.1)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 1900b4b3357515c3a624bdf1463910c17b48f8a1
Commit message: "Added AWS and Docker Helper"
[Pipeline] Start of Pipeline
[Pipeline] node
Running on jenkins-docker-dind-slave in /home/jenkins/workspace/experiment_basic-e2e-jenkinsfile
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Checkout SCM)
[Pipeline] checkout
using credential wxdeploy-rw-github-pipeline
Cloning the remote Git repository
Cloning with configured refspecs honoured and without tags
Fetching without tags
Cloning repository https://github.com/WeeverApps/redacted.git
 > git init /home/jenkins/workspace/experiment_basic-e2e-jenkinsfile # timeout=10
Fetching upstream changes from https://github.com/WeeverApps/redacted.git
 > git --version # timeout=10
using GIT_ASKPASS to set credentials Github Pipeline with CI integration
 > git fetch --no-tags --progress https://github.com/WeeverApps/redacted.git +refs/heads/experiment/basic-e2e-jenkinsfile:refs/remotes/origin/experiment/basic-e2e-jenkinsfile
 > git config remote.origin.url https://github.com/WeeverApps/redacted.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/experiment/basic-e2e-jenkinsfile:refs/remotes/origin/experiment/basic-e2e-jenkinsfile # timeout=10
 > git config remote.origin.url https://github.com/WeeverApps/redacted.git # timeout=10
Fetching upstream changes from https://github.com/WeeverApps/redacted.git
using GIT_ASKPASS to set credentials Github Pipeline with CI integration
 > git fetch --no-tags --progress https://github.com/WeeverApps/redacted.git +refs/heads/experiment/basic-e2e-jenkinsfile:refs/remotes/origin/experiment/basic-e2e-jenkinsfile
Checking out Revision 53e2deff9731d9c7674db3f234c647546e02521e (experiment/basic-e2e-jenkinsfile)
Commit message: "Add another basic Jenkinsfile"
First time build. Skipping changelog.
[Pipeline] }
[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Test)
[Pipeline] echo
Slack message stub method
	 >>>> [channel:#jenkins, color:good, message:TEST]
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 53e2deff9731d9c7674db3f234c647546e02521e
fatal: bad object 1900b4b3357515c3a624bdf1463910c17b48f8a1

GitHub has been notified of this commit’s build result

Finished: SUCCESS

So yeah, looks like it might be ciinabox, a shared lib. The codebase has zero results for slackSend but there are commits a couple months ago referring to removing it. I wonder if we're somehow behind, though it looks like we're pulling from the head. 🤷♂

Yeah this looks like it here. Going to close this and open an issue with those that set up Jenkins for us to find out how we're getting an outdated version of the code even though we pull from master...
https://github.com/base2Services/ciinabox-pipelines/commit/f80419d2cbe26d7312101421e18beecb10f002f0


Symfony\Component\Process\Process::__construct() must be of the type array, string given

Hello I get an error when I try to run the package
Symfony\Component\Process\Process::__construct() must be of the type array, string given, called in  arc\src\Console\InstallArc.php on line 46
 137â–•      * @param int|float|null $timeout The timeout in seconds or null to disable
    138â–•      *
    139â–•      * @throws LogicException When proc_open is not installed
    140â–•      */
  âžœ 141â–•     public function __construct(array $command, string $cwd = null, array $env = null, $input = null, ?float $timeout = 60)
    142â–•     {
    143â–•         if (!\function_exists('proc_open')) {
    144â–•             throw new LogicException('The Process class relies on proc_open, which is not available on your PHP installation.');
    145â–•         }

Strange, looks like there has been a change to the Symfony/Process package. I'll take a look and update.

Strange, looks like there has been a change to the Symfony/Process package. I'll take a look and update.

Thanks for the response and the pull request 😊🙏


feat: Honor .cargo/config.toml

Implements cargo/.config build target and cfg access by using unstable cargo options:

cargo config get to read the target triple out of the config to pass to cargo metadata --filter-platform
cargo rustc --print to read out the rustc_cfgs, this causes us to honor rustflags and the like.

If those commands fail, due to not having a nightly toolchain present for example, they will fall back to invoking rustc directly as we currently do.
I personally think it should be fine to use these unstable options as they are unlikely to change(even if they did it shouldn't be a problem due to the fallback) and don't burden the user if they do not have a nightly toolchain at hand since we fall back to the previous behaviour.
cc #8741
Closes #6604, Closes #5904, Closes #8430, Closes #8480

bors r+


typo in example code

This line fails:
print("CO2:", scd.CO2, "PPM")

it should read:
print("CO2:", scd.eCO2, "PPM")

Hi @DonBower, the field was just changed to CO2 because it's an actual measurement of CO2 instead of estimated. Please make sure you have the latest library version to match the example code.

Hi @DonBower, the field was just changed to CO2 because it's an actual measurement of CO2 instead of estimated. Please make sure you have the latest library version to match the example code.

Interesting.
I ran the sudo pip3 install adafruit-circuitpython-scd30 command about less than an hour before I made the comment.

Interesting.
I ran the sudo pip3 install adafruit-circuitpython-scd30 command about less than an hour before I made the comment.

@DonBower Did you get the example from here? We may have updated here without releasing. (I released 2.0.0 right after replying to you yesterday.)

@DonBower Did you get the example from here? We may have updated here without releasing. (I released 2.0.0 right after replying to you yesterday.)

yea, once i did a pip3 --upgrade, it came back with .CO2, not .eCO2.  I think we just must have passed in the mail.


Add support for multiple NUT servers

New Feature
The PR adds support for multiple NUT servers. It uses a new configuration which is backwards compatible with the original single server configuration format. UPSs are now indexed by an ID, "[ups]@[server]". The UI has been updated to support a new server column in the "UPS Devices" view.
Some additional minor fixes:

Add link to to "UPS Devices" page on the logo to allow navigation back to the root page from an individual UPS device page.
Add support for "Low battery" to status in _get_ups_status()
Fix a couple of isses were some fields, e.g. battery.charge, are not set. This can happen for a dummy/heartbeat UPS, which does not pupulate all values. Use .get() with a default value "--" in these cases.

any update here? Would be cool to add multiple servers.

any update on that ?

You can use this branch, and I would welcome feedback on it but there is nothing I can do about merging it into main unless @rshipp accepts the changes.

I know it is about 1 year :(
so i go ahead and use your repo....

@AdeMiller i found a small bug when you have multiple UPS monitored and one go offline the web interface do not display anything
where i can report it ?


"Bad credentials" bug

I'm triing to update a test file with this github action but I still getting an error.
Is there something wrong in my file ?
result when runing the actions
Run<EMAIL_ADDRESS>  with:
    gistID: 1db057b2a755ff6cfb841d987eeed3ab
    filename: test
    label: Test
    message: 98%
    color: green
result:{"message":"Bad credentials","documentation_url":"https://docs.github.com/rest"}

.yml action file
name: Update language value

on:
  push:
    branches:
    - "*"

jobs:
  updateValue:
    runs-on: ubuntu-latest
    steps:
    - name: Dynamic Badges
      uses<EMAIL_ADDRESS>      with:
        auth: ${{ secrets.GIST_SECRET }}
        gistID: 1db057b2a755ff6cfb841d987eeed3ab
        filename: test
        label: Test
        message: 98%
        color: green

Looks mostly good to me. The only thing that seems a bit weird is that you use the filename "test" while the only file in this gist [1] is called "gistfile1.txt". But maybe you changed this in the meantime? The only other thing I could think of is that the secret is not set correctly. Are you sure you checked the gist scope? And copied the correct secret string?
[1] https://gist.github.com/HydrolienF/1db057b2a755ff6cfb841d987eeed3ab

Has this been resolved on your end? Then I would close this issue...

This is also happening to me.
Following the docs, i set up the gist scope token on my main account. However, the action is running for another group. Is this the reason? And if so, what is the correct way of setting the secret?


[wip] +Migra -liquibase

Opening this for visibility, I intend to use this to do an actual migration in dev before considering it complete. There is some related work in our migrations repo (migra branch) that needs to be updated to work with this as well.

Closing until/if we decide to come back to this.


The zls.json should be configurable from within the extension settings in the editor

For the sole purpose of ease of use.

Yeah, thsi would be nice.
Especially if I reintroduce the workspace specific configuration at some point.

any update?
it's useful, say I work with different zig releases but need to manually modify the zig_lib_path in zls.json. otherwise, the extension complains Unable to open 'std.zig': Unable to read file ...

@fivemoreminix @oopsmonk Your wish has been realized (well half of it anyways, zigtools/zls#520 still needs to land); enjoy!


Unable to run ciao on raspberry pi using docker

Describe the bug
When I try to run ciao on a raspberry pi using docker, I see a linux error. I did not encounter the same problem on OSX
To Reproduce
After installing docker on a raspberry pi and then running ciao with
docker run --name ciao -p 8090:3000 brotandgames/ciao
I see the following error:
standard_init_linuz.go:211 eexec user process caused "exec format error"
Expected behavior
When I ran the same command on OSX, ciao started as requested, and I found it so useful, that I want to run it 24/7 on a rapsberry pi.
Desktop (please complete the following information):
The operating system is Raspbian GNU/Linux, version 10 (buster).
It is running on a raspberry pi 3 model B

@obromios There is currently no Docker build for ciao on Arm.
We are open for PRs.

Try this one
https://hub.docker.com/repository/docker/emaderito/ciao-rpi

Try this one
https://hub.docker.com/repository/docker/emaderito/ciao-rpi

You can build the image on your raspberry. I needed to make some changes before it would build to begin with:
https://github.com/brotandgames/ciao/pull/93

You can build the image on your raspberry. I needed to make some changes before it would build to begin with:
https://github.com/brotandgames/ciao/pull/93

Ciao (> 1.6.13) is now available for arm/v7 and arm64:
https://hub.docker.com/r/brotandgames/ciao/tags


Google Oauth 2.0 fail

from example repository

oauth blocked because need proper user-agent

@thaarrtt what's this screenshot? Are you using the example code: https://github.com/casdoor/casdoor-flutter-example or it's your own app?
What's the mobile device? brand and version and OS version

@thaarrtt what's this screenshot? Are you using the example code: https://github.com/casdoor/casdoor-flutter-example or it's your own app?
What's the mobile device? brand and version and OS version

its from example repository and its generic android emulator with latest sdk

A partial fix is available in https://github.com/casdoor/casdoor-flutter-sdk/pull/38 , the remaining is to work arround the google signin custom scheme limitation to trigger the callback.

@heralight thanks for PR and 1.5.0 has been released: https://pub.dev/packages/casdoor_flutter_sdk/versions/1.5.0
How to fix the remaining part?

@hsluoyz
At the end of the day, this sdk is too much outdated and has a lot of bad practices (no appauth or custom tab) and secrets are not well managed.
prefer to use https://pub.dev/packages/oidc directly, it work with casdoor like:
final userManager = OidcUserManager.lazy(
  discoveryDocumentUri: OidcUtils.getOpenIdConfigWellKnownUri(casdoorIssuer),
  clientCredentials:
      const OidcClientAuthentication.none(clientId: appClientId),
  store: OidcDefaultStore(),
  settings: OidcUserManagerSettings(
    redirectUri: redirectUri,
   extraAuthenticationParameters: {    'organizationName': 'myorg',
    'appName': 'myapp',},
    // the same redirectUri can be used as for post logout too.
   // postLogoutRedirectUri: redirectUri,
    scope: ['openid', 'profile', 'email'],
  ),
);

best regards,

@heralight can you make a PR?


Use Cortana to call a running app does not trigger OnStartAsync, OnInitializeAsync, OnPrelaunchAsync or OnResuming

When the app is already running and I use Cortana to call into the app, I am unable to find the place where I can intercept the IActivatedEventArgs I looked at the OnStartAsync, OnInitializeAsync, OnPrelaunchAsync, OnResuming. Nothing seems to get triggered.
How can I react to this event?

I am currently using with the VoiceCommandActivatedEventArgs. I did not activate the Personal Assistant since I thought that was only for Cortana inside the app

OK - have you looked at the cortana sample?
https://github.com/Windows-XAML/Template10/tree/master/Samples/Cortana

I don't quite follow the your last comment. The app has to be run once to install the voice commands (as I am sure you know). Are you saying that the sample does not support responding to a voicecommand if it is running, only when it is not running? (as a general rule - please be as expansive as possible with your comments, say what you did, what errors, etc.)

Sure, the app runs one to install the
var storageFile =                 await StorageFile.GetFileFromApplicationUriAsync(new Uri("ms-appx:///Assets/AskAlexCommandSet.xml"));             await VoiceCommandDefinitionManager.InstallCommandDefinitionsFromStorageFileAsync(storageFile);
Then I close the app and use Cortana to launch and react to the VoiceCommandActivatedEventArgs
This first scenario works. If the Application is already running and I use Cortana to issue the same VoiceCommandActivatedEventArgs the app comes into focus but nothing else is triggered.

What version of T10 are you running?

Currently on 1.1.8

I'm seeing the same kind of behavior if, the app is already running and that I share to it from a browser, no event is sent through the OnStartAsync.

Would you please use the code in GitHub and test? I believe this was resolved.

Using the code from the repo, I found in the BootStrapper under InternalActivatedAsync that line 216 makes sur that we have an activate or launch mode. the previous state for this call is running. This results _HasOnStartAsync being True because it is not reset and causes OnStartAsync to be bypassed.

Thought, should Template 10 automatically hide the main Window like the way Mail app behaves when we activate the share contract, then once the share is complete show the main window? This would solve both windows navigating to the page, that handles each IActivatedEventArgs Kind. It would also help with managing multiple windows and static references/instances.

I have run into this same issue. Fixed it by removing the  'if(!_HasOnStartAsync)' check on the mentioned line 216. Haven't noticed any major side effect with the change and Cortana commands now work fine.

I added the condition when 'running' is the previous state and found that if the app is running, and that the share contract is triggered, the app shows up un the side windows. When the share is set to completed, both windows close. I have to figure out why.

I wonder if there's a way to get rid of statics by way of using constructor injection. This would make sure that multiple windows have independent states and lifecycles. The service locator pattern/ singleton pattern works great when we don't have to worry about this type of scenario.

Resolved


Allow label sets to be created from meter or Meter

Creating per SIG meeting
Which problem is this PR solving?

#572

Short description of the changes

allow meter.labels and Meter.labels as a convenience

Codecov Report

Merging #576 into master will increase coverage by 3.7%.
The diff coverage is 100%.


OCPVE-220: remove unused certificates

This PR removes the initcontainer self-signed-cert-generator from TopoLVM Controller. This is because the certs are used for TopoLVM webhooks which are not deployed in LVMS. As there is no option to disable this from TopoLVM, we just pass the existing webhook certs from LVMS to make the controller start, but the certs are not used afterward.
Also, this PR removes the unused webhook and certmanager files to simplify the codebase.

/hold

/test all

/unhold

/retest

/retest

/retest

Codecov Report

Merging #413 (8c621ab) into main (9677ed3) will decrease coverage by 0.38%.
Report is 2 commits behind head on main.
The diff coverage is 100.00%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #413      +/-   ##
==========================================
- Coverage   63.68%   63.31%   -0.38%
==========================================
  Files          28       28
  Lines        2360     2336      -24
==========================================
- Hits         1503     1479      -24
  Misses        709      709
  Partials      148      148

Files Changed
Coverage Δ

controllers/topolvm_controller.go
97.20% <100.00%> (-0.29%)
:arrow_down:

controllers/vgmanager_daemonset.go
100.00% <100.00%> (ø)

/lgtm


i18n: Translation of dynamic data?

Apologies if I just haven't read the docs in enough detail, but I did not see a clear answer for this: how is dynamic data that needs to be translated supported in the angular i18n proposal?
E.g. for a simple example, take:

{{ user.status }}

Where status can be selected from a set of known possibilities.
Another example, if I have a select over a set of attributes that are set dynamically:
<select ng-options="region.code as region.name for region in regions" ng-model="region">
   <option style="display:none" value="">select a region</option>
</select>

In something like angular-localization (similar to angular translate, dynamic i18n resolution system), these are handled as

{{ user.status | i18n }}

and
<select ng-options="region.code as (region.name | i18n) for region in regions" ng-model="region">
   <option style="display:none" value="">select a region</option>
</select>

This data cannot be known statically, at least not without significantly restructuring the surrounding code, but it's not clear to me that the current dynamic support proposal covers a situation like this, so I'm curious what the thinking is currently.
Thanks!

@brycecr could we please move this discussion to https://github.com/angular/i18n ? Thnx!


Setuptools does not copy package data

Hi,
I can't solve what seems to be a relatively simple problem, I have asked the question on StackOverflow but it didn't get interest/answers and I'm sure some of you have the answer.
My project has the following folder structure:
mypkg/
   -- setup.py
  |
   -- src/
       |
        -- mypkg/
             |
              -- actual source files (.py)
  |
   -- supporting_files/
  |
   -- ...

I want the supporting_files to be copied next to the package's source files after installation.
In other words, I want to end up with the following folder structure:
lib/
  |
   -- site-packages/
     |
      -- mypkg/
           |
            -- actual source files (.py)
           |
            -- supporting_files/

I have tried two things:

package_data = {'': ['supporting_files']}
data_files = [('supporting_files', [os.path.join("supporting_files", f) for f in os.listdir("supporting_files")])]

but none of these did the trick.
To be clear, I have no issue getting the supporting_files copied in the egg or the wheel, just copying it at install time doesn't work.
What am I missing here? 😃

I'm running Python 2, if that helps

I have a similar issue. Either using MANIFEST.in or package_data lists the files under SOURCES.txt but no files are actually copied. Only when data_files are explicitly used.

Hi @nonatomiclabs and @cvanelteren, package_data will only work for files that are placed inside the package directory (currently there is no support for files outside of it)
So in your example, the best would be moving supporting_files inside src/mypkg.
Please also notice that data_files is no longer recommended and will not work on wheels.


question on freezing target nework

Hi @yenchenlin1994 , love your implementation!
I went through your code and I can't seem to find where you've frozen the target network?
Unless Im missing something in my excess-caffeine induced brain fade,you continue to update the target every batch?
Wouldn't that hurt your convergence rate badly?

Hello,
Yeah you are right.
Actually I got a reimplemented version.
Will submit soon!
On Wed, Apr 20, 2016 at 17:46 Code-Deep-Blue<EMAIL_ADDRESS>wrote:

Hi @yenchenlin1994 https://github.com/yenchenlin1994 , love your
implementation!
I went through your code and I can't seem to find where you've frozen the
target network?
Unless Im missing something in my excess-caffeine induced brain fade,you
continue to update the target every batch?
Wouldn't that hurt your convergence rate badly?
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/yenchenlin1994/DeepLearningFlappyBird/issues/15

Hi again,
i'm trying to reproduce the results on keras and have trained for ~400,000 steps and the bird is unable to cross the first pipe consistently. My loss is low though (~ 0.2) and Q-values are in the range of [0,8]. How long did it take for you before it actually started working i.e. cross the first pipe consistently?

I can't remember the exactly number of iterations, but it's no more than ~1000,000 steps

Still cannot find freezing target network in current version's code. It's really no effect?

@hashbangCoder
I meet the same question that the silly bird keeps top of the screen.....Did you fix it?

I also couldn't find freezing target network code. But thanks for your code. It's helpful for me.

I write a version base on this repo with freezing target network.FlappyBird_DQN_with_target_network


Images stuck in not_analyzed while analyzer workers not busy

Is this a request for help?:
yes
Is this a BUG REPORT or a FEATURE REQUEST? (choose one):
Bug report
Version of Anchore Engine and Anchore CLI if applicable:
➜ anchore-cli --version
anchore-cli, version 0.8.0

What happened:
Some images got stuck with status not_analyzed although analyzer workers aren't busy and there are no images in analyzing state. Even if I re-add them using the anchore-cli and the --force option, the images aren't sent to the workers.
➜ anchore-cli image list | grep not_analyzed | wc -l
      11

➜ anchore-cli image list | grep analyzing | wc -l
       0

Catalog service is full of logs like this:
[service:catalog] 2020-08-17 09:00:08+0000 [-] [Thread-384501] [anchore_engine.clients.services.internal/dispatch()] [DEBUG] Dispatching: url=http://anchore-anchore-engine-simplequeue:8083/v1/queues/images_to_analyze/is_inqueue, headers={'Content-Type': 'application/json'}, body={"userId": "admin", "imageDigest": "sha256:b1e093df83ea8076d419c74b122ee6c5141a3468f263ef1eee085beeea6b8b17", "manifest": "{\"schemaVersion\": 2, \"mediaType\": \"application/vnd.docker.distribution.manifest.v2+json\", \"config\": {\"mediaType\": \"application/vnd.docker.container.image.v1+json\", \"size\": 7927, \"digest\": \"sha256:49a9fcc799ce0f2129652d4103cb0afe5cec23fda63a9a02d771f5c4679bcd01\"}, \"layers\": [{\"mediaType\": \"application/vnd.docker.image.rootfs.diff.tar.gzip\", \"size\": 22522276, \"d..., params=None, timeout=(None, None), files=None
[service:catalog] 2020-08-17 09:00:08+0000 [-] [Thread-384501] [anchore_engine.subsys.identities/_get_system_user_credentials()] [DEBUG] Loading system user creds
[service:catalog] 2020-08-17 09:00:08+0000 [-] [Thread-384501] [anchore_engine.subsys.identities/_get_system_user_credentials()] [DEBUG] Cached system credential still ok
[service:catalog] 2020-08-17 09:00:08+0000 [-] [Thread-384501] [anchore_engine.clients.services.http/anchy_post()] [DEBUG] POST url=http://anchore-anchore-engine-simplequeue:8083/v1/queues/images_to_analyze/is_inqueue httpcode=200
[service:catalog] 2020-08-17 09:00:08+0000 [-] [Thread-384501] [anchore_engine.services.catalog/handle_analyzer_queue()] [DEBUG] image already queued
[service:catalog] 2020-08-17 09:00:08+0000 [-] [Thread-384501] [anchore_engine.services.catalog/handle_analyzer_queue()] [DEBUG] image check
[service:catalog] 2020-08-17 09:00:08+0000 [-] [Thread-384501] [anchore_engine.services.catalog/handle_analyzer_queue()] [DEBUG] image in base state - sha256:0e2e9cb87f954517c5ac3bd543903943e0d956867663e478d0acf7f6185b6b57
[service:catalog] 2020-08-17 09:00:08+0000 [-] [Thread-384501] [anchore_engine.clients.services.common/get_service_endpoints()] [DEBUG] got endpoints (simplequeue): [['http://anchore-anchore-engine-simplequeue:8083/v1']]
[service:catalog] 2020-08-17 09:00:08+0000 [-] [Thread-384501] [anchore_engine.clients.services.internal/dispatch()] [DEBUG] Dispatching: url=http://anchore-anchore-engine-simplequeue:8083/v1/queues/images_to_analyze/is_inqueue, headers={'Content-Type': 'application/json'}, body={"userId": "admin", "imageDigest": "sha256:0e2e9cb87f954517c5ac3bd543903943e0d956867663e478d0acf7f6185b6b57", "manifest": "{\"schemaVersion\": 2, \"mediaType\": \"application/vnd.docker.distribution.manifest.v2+json\", \"config\": {\"mediaType\": \"application/vnd.docker.container.image.v1+json\", \"size\": 7925, \"digest\": \"sha256:c5c7e957e2229521b389075bce0f938531c91fd4edd2f34256e99e6c1ee5d329\"}, \"layers\": [{\"mediaType\": \"application/vnd.docker.image.rootfs.diff.tar.gzip\", \"size\": 22522276, \"d..., params=None, timeout=(None, None), files=None

The queue service is running and has no error logs, only:
[service:simplequeue] 2020-08-17 09:37:05+0000 [-] "<IP_ADDRESS>" - - [17/Aug/2020:09:37:04 +0000] "POST /v1/queues/images_to_analyze/is_inqueue HTTP/1.1" 200 3642 "-" "python-requests/2.23.0"
[service:simplequeue] 2020-08-17 09:37:05+0000 [_GenericHTTPChannelProtocol,1980365,<IP_ADDRESS>] [PoolThread-twisted.internet.reactor-19] [anchore_engine.subsys.auth.realms/get_authentication_info()] [DEBUG] Attempting to get cached credentials for [anchore-system]
[service:simplequeue] 2020-08-17 09:37:05+0000 [_GenericHTTPChannelProtocol,1980365,<IP_ADDRESS>] [PoolThread-twisted.internet.reactor-19] [anchore_engine.apis.authorization/authenticate_token()] [DEBUG] Login complete for user: IdentityContext(username='anchore-system', user_account='anchore-system', user_account_type=<AccountTypes.service: 'service'>, user_account_state=<AccountStates.enabled: 'enabled'>, user_type=<UserTypes.native: 'native'>, user_uuid='f375c216d0ec42468337188babd31d56')
[service:simplequeue] 2020-08-17 09:37:05+0000 [-] "<IP_ADDRESS>" - - [17/Aug/2020:09:37:04 +0000] "POST /v1/queues/images_to_analyze/is_inqueue HTTP/1.1" 200 3640 "-" "python-requests/2.23.0"
[service:simplequeue] 2020-08-17 09:37:05+0000 [-] "<IP_ADDRESS>" - - [17/Aug/2020:09:37:04 +0000] "GET /v1/queues/watcher_tasks?wait_max_seconds=30&visibility_timeout=0 HTTP/1.1" 200 455 "-" "python-requests/2.23.0"
[service:simplequeue] 2020-08-17 09:37:05+0000 [-] "<IP_ADDRESS>" - - [17/Aug/2020:09:37:04 +0000] "GET /health HTTP/1.1" 200 5 "-" "kube-probe/1.16"
[service:simplequeue] 2020-08-17 09:37:05+0000 [_GenericHTTPChannelProtocol,1980367,<IP_ADDRESS>] [PoolThread-twisted.internet.reactor-10] [anchore_engine.subsys.auth.realms/get_authentication_info()] [DEBUG] Attempting to get cached credentials for [anchore-system]
[service:simplequeue] 2020-08-17 09:37:05+0000 [_GenericHTTPChannelProtocol,1980368,<IP_ADDRESS>] [PoolThread-twisted.internet.reactor-7] [anchore_engine.subsys.auth.realms/get_authentication_info()] [DEBUG] Attempting to get cached credentials for [anchore-system]
[service:simplequeue] 2020-08-17 09:37:05+0000 [_GenericHTTPChannelProtocol,1980368,<IP_ADDRESS>] [PoolThread-twisted.internet.reactor-7] [anchore_engine.apis.authorization/authenticate_token()] [DEBUG] Login complete for user: IdentityContext(username='anchore-system', user_account='anchore-system', user_account_type=<AccountTypes.service: 'service'>, user_account_state=<AccountStates.enabled: 'enabled'>, user_type=<UserTypes.native: 'native'>, user_uuid='f375c216d0ec42468337188babd31d56')
[service:simplequeue] 2020-08-17 09:37:05+0000 [-] "<IP_ADDRESS>" - - [17/Aug/2020:09:37:04 +0000] "GET /v1/leases/analyzer_queue HTTP/1.1" 200 180 "-" "python-requests/2.23.0"
[service:simplequeue] 2020-08-17 09:37:05+0000 [_GenericHTTPChannelProtocol,1980367,<IP_ADDRESS>] [PoolThread-twisted.internet.reactor-10] [anchore_engine.apis.authorization/authenticate_token()] [DEBUG] Login complete for user: IdentityContext(username='anchore-system', user_account='anchore-system', user_account_type=<AccountTypes.service: 'service'>, user_account_state=<AccountStates.enabled: 'enabled'>, user_type=<UserTypes.native: 'native'>, user_uuid='f375c216d0ec42468337188babd31d56')

Analyzers only have healthcheck logs from the kubernetes probe.
[service:worker] 2020-08-14 09:53:52+0000 [-] [MainThread] [anchore_engine.twisted/makeService()] [INFO] Initializing configuration
[service:worker] 2020-08-14 09:53:52+0000 [-] [MainThread] [anchore_engine.twisted/makeService()] [INFO] Initializing logging
[service:worker] 2020-08-14 09:53:52+0000 [-] [MainThread] [anchore_engine.service/_register_instance_handlers()] [INFO] Registering api handlers
[service:worker] 2020-08-14 09:53:52+0000 [-] [MainThread] [anchore_engine.service/_process_stage_handlers()] [INFO] Processing init handlers for bootsrap stage: pre_config
[service:worker] 2020-08-14 09:53:52+0000 [-] [MainThread] [anchore_engine.service/_configure()] [INFO] Loading and initializing global configuration
[service:worker] 2020-08-14 09:53:52+0000 [-] [MainThread] [anchore_engine.service/_configure()] [INFO] Configuration complete
[service:worker] 2020-08-14 09:53:52+0000 [-] [MainThread] [anchore_engine.service/_process_stage_handlers()] [INFO] Processing init handlers for bootsrap stage: post_config
[service:worker] 2020-08-14 09:53:52+0000 [-] [MainThread] [anchore_engine.service/_process_stage_handlers()] [INFO] Processing init handlers for bootsrap stage: pre_db
[service:worker] 2020-08-14 09:53:52+0000 [-] [MainThread] [anchore_engine.service/_db_connect()] [INFO] Configuring db connection
[service:worker] 2020-08-14 09:53:52+0000 [-] [MainThread] [anchore_engine.service/_db_connect()] [INFO] Initializing database
[service:worker] 2020-08-14 09:53:52+0000 [-] [MainThread] [anchore_engine.service/_db_connect()] [INFO] DB connection initialization complete
[service:worker] 2020-08-14 09:53:52+0000 [-] [MainThread] [anchore_engine.service/_process_stage_handlers()] [INFO] Processing init handlers for bootsrap stage: post_db
[service:worker] 2020-08-14 09:53:52+0000 [-] [MainThread] [anchore_engine.service/_process_stage_handlers()] [INFO] Processing init handlers for bootsrap stage: pre_credentials
[service:worker] 2020-08-14 09:53:52+0000 [-] [MainThread] [anchore_engine.service/_credential_init()] [INFO] Bootstrapping credentials
[service:worker] 2020-08-14 09:53:52+0000 [-] [MainThread] [anchore_engine.service/_credential_init()] [INFO] Checking system creds
[service:worker] 2020-08-14 09:53:53+0000 [-] [MainThread] [anchore_engine.service/_credential_init()] [INFO] Found valid system creds
[service:worker] 2020-08-14 09:53:53+0000 [-] [MainThread] [anchore_engine.service/_credential_init()] [INFO] Credential initialization complete
[service:worker] 2020-08-14 09:53:53+0000 [-] [MainThread] [anchore_engine.service/_process_stage_handlers()] [INFO] Processing init handlers for bootsrap stage: post_credentials
[service:worker] 2020-08-14 09:53:53+0000 [-] [MainThread] [anchore_engine.service/_process_stage_handlers()] [INFO] Processing init handlers for bootsrap stage: pre_bootstrap
[service:worker] 2020-08-14 09:53:53+0000 [-] [MainThread] [anchore_engine.service/initialize_api()] [INFO] Initializing API from: /usr/local/lib/python3.6/site-packages/anchore_engine/services/analyzer/swagger/swagger.yaml
[service:worker] 2020-08-14 09:53:53+0000 [-] [MainThread] [anchore_engine.service/_bootstrap()] [INFO] Bootstrapping service
[service:worker] 2020-08-14 09:53:53+0000 [-] [MainThread] [anchore_engine.service/_bootstrap()] [INFO] Service bootstrap complete
[service:worker] 2020-08-14 09:53:53+0000 [-] [MainThread] [anchore_engine.service/_process_stage_handlers()] [INFO] Processing init handlers for bootsrap stage: post_bootstrap
[service:worker] 2020-08-14 09:53:53+0000 [-] [MainThread] [anchore_engine.service/_process_stage_handlers()] [INFO] Processing init handlers for bootsrap stage: pre_register
[service:worker] 2020-08-14 09:53:53+0000 [-] [MainThread] [anchore_engine.service/_register()] [INFO] Registering service: analyzer
[service:worker] 2020-08-14 09:53:53+0000 [-] [MainThread] [anchore_engine.service/_register()] [INFO] Service registration complete
[service:worker] 2020-08-14 09:53:53+0000 [-] [MainThread] [anchore_engine.service/_process_stage_handlers()] [INFO] Processing init handlers for bootsrap stage: post_register
[service:worker] 2020-08-14 09:53:53+0000 [-] [MainThread] [anchore_engine.twisted/makeService()] [INFO] Starting monitor thread
[service:worker] 2020-08-14 09:53:53+0000 [-] [MainThread] [anchore_engine.twisted/makeService()] [INFO] Building api handlers
[service:worker] 2020-08-14 09:53:53+0000 [-] twistd 20.3.0 (/usr/bin/python3.6 3.6.8) starting up.
[service:worker] 2020-08-14 09:53:53+0000 [-] reactor class: twisted.internet.epollreactor.EPollReactor.
[service:worker] 2020-08-14 09:53:53+0000 [-] Site starting on 8084
[service:worker] 2020-08-14 09:53:53+0000 [-] Starting factory <twisted.web.server.Site object at 0x7fde17e6afd0>
[service:worker] 2020-08-14 09:53:54+0000 [-] "<IP_ADDRESS>" - - [14/Aug/2020:09:53:54 +0000] "GET /health HTTP/1.1" 200 5 "-" "kube-probe/1.16"
[service:worker] 2020-08-14 09:54:04+0000 [-] "<IP_ADDRESS>" - - [14/Aug/2020:09:54:04 +0000] "GET /health HTTP/1.1" 200 5 "-" "kube-probe/1.16"
[service:worker] 2020-08-14 09:54:14+0000 [-] "<IP_ADDRESS>" - - [14/Aug/2020:09:54:14 +0000] "GET /health HTTP/1.1" 200 5 "-" "kube-probe/1.16"

The overall status of anchore engine in kubernetes:
NAME                                                 READY   STATUS      RESTARTS   AGE
anchore-anchore-engine-analyzer-7b6cb6d58-92m7r      1/1     Running     0          2d23h
anchore-anchore-engine-analyzer-7b6cb6d58-k4rsh      1/1     Running     0          2d23h
anchore-anchore-engine-analyzer-7b6cb6d58-qg4ps      1/1     Running     0          2d23h
anchore-anchore-engine-analyzer-7b6cb6d58-vxgj8      1/1     Running     0          2d23h
anchore-anchore-engine-api-6f7597f96-2lgcv           1/1     Running     0          38h
anchore-anchore-engine-api-6f7597f96-bsbn2           1/1     Running     0          4d21h
anchore-anchore-engine-catalog-5848c95fd4-4btvl      1/1     Running     0          2d22h
anchore-anchore-engine-policy-7b4d55cbf9-82m2v       1/1     Running     0          4d21h
anchore-anchore-engine-policy-7b4d55cbf9-td7l2       1/1     Running     0          4d21h
anchore-anchore-engine-simplequeue-8c944cd97-9vf8q   1/1     Running     0          2d22h
anchore-engine-upgrade-pd7xr                         0/1     Completed   0          2d22h
anchore-postgresql-0                                 1/1     Running     0          4d20h

Resource usage:
NAME                                                 CPU(cores)   MEMORY(bytes)
anchore-anchore-engine-analyzer-7b6cb6d58-92m7r      5m           305Mi
anchore-anchore-engine-analyzer-7b6cb6d58-k4rsh      4m           340Mi
anchore-anchore-engine-analyzer-7b6cb6d58-qg4ps      3m           386Mi
anchore-anchore-engine-analyzer-7b6cb6d58-vxgj8      5m           307Mi
anchore-anchore-engine-api-6f7597f96-2lgcv           3m           195Mi
anchore-anchore-engine-api-6f7597f96-bsbn2           5m           213Mi
anchore-anchore-engine-catalog-5848c95fd4-4btvl      372m         283Mi
anchore-anchore-engine-policy-7b4d55cbf9-82m2v       5m           392Mi
anchore-anchore-engine-policy-7b4d55cbf9-td7l2       9m           458Mi
anchore-anchore-engine-simplequeue-8c944cd97-9vf8q   716m         199Mi
anchore-postgresql-0                                 40m          977Mi

Reported by the anchore-cli:
➜ anchore-cli system status
Service apiext (anchore-anchore-engine-api-6f7597f96-2lgcv, http://anchore-anchore-engine-api:8228): up
Service analyzer (anchore-anchore-engine-analyzer-7b6cb6d58-92m7r, http://anchore-anchore-engine-analyzer:8084): up
Service analyzer (anchore-anchore-engine-analyzer-7b6cb6d58-vxgj8, http://anchore-anchore-engine-analyzer:8084): up
Service policy_engine (anchore-anchore-engine-policy-7b4d55cbf9-td7l2, http://anchore-anchore-engine-policy:8087): up
Service analyzer (anchore-anchore-engine-analyzer-7b6cb6d58-qg4ps, http://anchore-anchore-engine-analyzer:8084): up
Service analyzer (anchore-anchore-engine-analyzer-7b6cb6d58-k4rsh, http://anchore-anchore-engine-analyzer:8084): up
Service simplequeue (anchore-anchore-engine-simplequeue-8c944cd97-9vf8q, http://anchore-anchore-engine-simplequeue:8083): up
Service catalog (anchore-anchore-engine-catalog-5848c95fd4-4btvl, http://anchore-anchore-engine-catalog:8082): up
Service apiext (anchore-anchore-engine-api-6f7597f96-bsbn2, http://anchore-anchore-engine-api:8228): up
Service policy_engine (anchore-anchore-engine-policy-7b4d55cbf9-82m2v, http://anchore-anchore-engine-policy:8087): up

Engine DB Version: 0.0.13
Engine Code Version: 0.8.0

Anything else we need to know:
Once I delete an analyzer pod, images start to transition to analyzing state, being handled by all analyzer pods.
➜ anchore-cli image list | grep analyzing | wc -l
       1

Logs from the same analyzer worker (as the ones above, after recreating another analyzer pod):
[service:worker] 2020-08-17 09:45:04+0000 [-] [Thread-3987] [anchore_engine.services.analyzer/process_analyzer_job()] [INFO] analysis complete: admin : sha256:b1e093df83ea8076d419c74b122ee6c5141a3468f263ef1eee085beeea6b8b17
[service:worker] 2020-08-17 09:45:07+0000 [-] "<IP_ADDRESS>" - - [17/Aug/2020:09:45:07 +0000] "GET /health HTTP/1.1" 200 5 "-" "kube-probe/1.16"
[service:worker] 2020-08-17 09:45:09+0000 [-] [Thread-3994] [anchore_engine.services.analyzer/process_analyzer_job()] [INFO] analysis complete: admin : sha256:6c787a09153ceb66690a9fe95f3423b79f76f47e39312ba94bc6b7ce6b4cda78
[service:worker] 2020-08-17 09:45:13+0000 [-] [Thread-258689] [anchore_engine.services.analyzer/process_analyzer_job()] [INFO] image dequeued for analysis: admin : sha256:091a0a43018c29b3b968d9cdeb4b6f25c2104f9a710140a320b0fbbb105ee867
[service:worker] 2020-08-17 09:45:13+0000 [-] [Thread-258689] [anchore_engine.services.analyzer/perform_analyze_nodocker()] [INFO] performing analysis on image: ['admin', '<REDACTED>-6d3de176@sha256:091a0a43018c29b3b968d9cdeb4b6f25c2104f9a710140a320b0fbbb105ee867', '<REDACTED>-6d3de176:1.0.1']
[service:worker] 2020-08-17 09:45:13+0000 [-] [Thread-258689] [anchore_engine.clients.localanchore_standalone/pull_image()] [INFO] Downloading image <REDACTED>-6d3de176@sha256:091a0a43018c29b3b968d9cdeb4b6f25c2104f9a710140a320b0fbbb105ee867 for analysis to /analysis_scratch/bf1af7a1-4ac8-4109-ae49-519d3b27e252/raw
[service:worker] 2020-08-17 09:45:14+0000 [-] "<IP_ADDRESS>" - - [17/Aug/2020:09:45:14 +0000] "GET /health HTTP/1.1" 200 5 "-" "kube-probe/1.16"
[service:worker] 2020-08-17 09:45:17+0000 [-] "<IP_ADDRESS>" - - [17/Aug/2020:09:45:17 +0000] "GET /health HTTP/1.1" 200 5 "-" "kube-probe/1.16"
[service:worker] 2020-08-17 09:45:19+0000 [-] [Thread-258695] [anchore_engine.services.analyzer/process_analyzer_job()] [INFO] image dequeued for analysis: admin : sha256:77a121a724ae8331cdc9b8ac5d601b9f61509fe7229e8be24d1d633609e8fb73
[service:worker] 2020-08-17 09:45:19+0000 [-] [Thread-258695] [anchore_engine.services.analyzer/perform_analyze_nodocker()] [INFO] performing analysis on image: ['admin', '<REDACTED>-6d3de176@sha256:77a121a724ae8331cdc9b8ac5d601b9f61509fe7229e8be24d1d633609e8fb73', '<REDACTED>-6d3de176:0.12.6']
[service:worker] 2020-08-17 09:45:20+0000 [-] [Thread-258695] [anchore_engine.clients.localanchore_standalone/pull_image()] [INFO] Downloading image <REDACTED>-6d3de176@sha256:77a121a724ae8331cdc9b8ac5d601b9f61509fe7229e8be24d1d633609e8fb73 for analysis to /analysis_scratch/5922b0a8-7a5a-4a76-b653-30a10efb4453/raw

Notice the newly created pod:
NAME                                                 READY   STATUS      RESTARTS   AGE
anchore-anchore-engine-analyzer-7b6cb6d58-k4rsh      1/1     Running     0          2d23h
anchore-anchore-engine-analyzer-7b6cb6d58-pqkgf      1/1     Running     0          110s
anchore-anchore-engine-analyzer-7b6cb6d58-qg4ps      1/1     Running     0          2d23h
anchore-anchore-engine-analyzer-7b6cb6d58-vxgj8      1/1     Running     0          2d23h

and the new system status:
➜ anchore-cli system status
Service apiext (anchore-anchore-engine-api-6f7597f96-2lgcv, http://anchore-anchore-engine-api:8228): up
Service analyzer (anchore-anchore-engine-analyzer-7b6cb6d58-pqkgf, http://anchore-anchore-engine-analyzer:8084): up
Service analyzer (anchore-anchore-engine-analyzer-7b6cb6d58-92m7r, http://anchore-anchore-engine-analyzer:8084): down (unavailable)
Service analyzer (anchore-anchore-engine-analyzer-7b6cb6d58-vxgj8, http://anchore-anchore-engine-analyzer:8084): up
Service policy_engine (anchore-anchore-engine-policy-7b4d55cbf9-td7l2, http://anchore-anchore-engine-policy:8087): up
Service analyzer (anchore-anchore-engine-analyzer-7b6cb6d58-qg4ps, http://anchore-anchore-engine-analyzer:8084): up
Service analyzer (anchore-anchore-engine-analyzer-7b6cb6d58-k4rsh, http://anchore-anchore-engine-analyzer:8084): up
Service simplequeue (anchore-anchore-engine-simplequeue-8c944cd97-9vf8q, http://anchore-anchore-engine-simplequeue:8083): up
Service catalog (anchore-anchore-engine-catalog-5848c95fd4-4btvl, http://anchore-anchore-engine-catalog:8082): up
Service apiext (anchore-anchore-engine-api-6f7597f96-bsbn2, http://anchore-anchore-engine-api:8228): up
Service policy_engine (anchore-anchore-engine-policy-7b4d55cbf9-82m2v, http://anchore-anchore-engine-policy:8087): up

Engine DB Version: 0.0.13
Engine Code Version: 0.8.0

@StefanCenusa thanks for filing the issue. We'll take a look to investigate further. I'd like to focus on the analyzer and the queue services. Are those the only logs you see from the queue service? I should be showing GET /v1/queues/images_to_analyze every 5-10 seconds from the analyzer hosts.
So, I suspect maybe a network or routing issue from the analyzers to the queue service since they use the k8s service endpoint. It looks like the catalog is able to reach the queue based on the queue log you provided but I see no requests from any of the analyzers. Can you confirm that until you remove the analyzer pod there are no requests to GET /v1/queues/images_to_analyze ?

@zhill Thanks for the answer. I've managed to fix the issue by not using a shared volume between analyzers for the analysis-scratch volume. Previously I was using a cephfs volume mounted as RWX(ReadWriteMany) in kubernetes - because I thought it would be helpful for all the workers to share the image cache if possible. Since this change, I haven't encountered any more problems. Maybe this is a thing to look for, when analyzers are using the same volume for caching/downloading images.
Unfortunately I don't have the same setup anymore in order to provide you with more logs. From my perspective this issue can be closed.
I could run the same setup to test and share the full logs with you, but maybe in a few weeks.


Add delay when reconnecting to schema notifications

What this PR does:
Add an exponential backoff if the schema cache loses connectivity with the bridge.
Which issue(s) this PR fixes:
N/A
Checklist

[x] Changes manually tested
[ ] Automated Tests added/updated
[ ] Documentation added/updated
[x] CLA Signed: DataStax CLA

Should the state of the schema cache connection (or any connection) to the bridge affect how the service reports readiness?

That does seem like a good idea. This relates, I think, to our framework upgrade work.
Although TBH, HealthResource we have for new REST is not using OSGi and could be wired relative easily using simple @Inject etc I think.
So maybe worth filing a separate issue?

@olim7t do you think this can now be merged, to verify Haxx/perf tests passing?


build and test

Hi @jiangok-open, not sure this is useful for the general population, so I'm going to close it.


[ZEPPELIN-5442] Allow users to set a separate namespace for the interpreter

What is this PR for?
Currently, if zeppelin is running under k8s mode and the zeppelin server is running inside the k8s cluster, it will start the interpreter pod under the namespace specified in /var/run/secrets/kubernetes.io/serviceaccount/namespace. But if zeppelin-server is running outside of kubernetes cluster, it will use the config zeppelin.k8s.namespace to start the interpreter pod under the given namespace. However, this approach will cause users to only set a global namespace.
Hence, this PR allows user to set a set separate namespace for the interpreter in the interpreter settings:
Example usage:
%spark.conf
zeppelin.k8s.interpreter.namespace spark

The spark interpreter pod will be created under the namespace: spark.
%python.conf
zeppelin.k8s.interpreter.namespace python

The python interpreter pod will be created under the namespace: python.
What type of PR is it?
[Improvement]
Todos

[ ] - Task

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-5442

How should this be tested?

CI pass and manually tested

Screenshots (if appropriate)
Questions:

Does the licenses files need update? No
Is there breaking changes for older versions? No
Does this needs documentation? No

Feature should also be available for Zeppelin server running in K8s + CI failures.

Hi, @Reamer
Thanks for your comment. I have fixed the CI failure.
After my test, when zeppelin server is also running inside k8s, there'll be a few more problems:

the zeppelin server needs to obtain the cluster role instead of role to create pods and services in other namespaces.
There're 2 namespaces: the namespace of zeppelin server and the namespace of zeppelin interpreter. So the namespace return by method getNamespace may be wrong if we return the zeppelin.k8s.interpreter.namespace while zeppelin server is also running in k8s.
When zeppelin server pod and interpreter pod are in different namespaces, the ownerReference can not be set. Since Cross-namespace owner references are disallowed by design https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/.

But if zeppelin server is running outside k8s, it can correctly create the interpreter pod in different namespaces. There won't be a problem of cross-namespace communication between zeppelin server and interpreter.
So how about support the feature of separate namespace while zeppelin server is running in k8s in another PR.

Thank you very much for your analysis. I understand the problems when Zeppelin server is running in K8s and they are not easy to solve.
Please add the documentation which explain the flag. Please make it clear that this feature only works when Zeppelin is running outside of K8s.

Hi, @Reamer
Thanks for the comment. Due to potential code conflicts, I think it's better to achieve the feature of setting the separate namespace for the interpereter when Zeppelin server is running inside k8s.
So I added several commits to address the above three problems. Specifically,

Change role and rolebinding to clusterrole and clusterrolebinding
Update the conditions for setting ownerReference
Return the correct namespace of Zeppelin Server

Hence, it can set the separate namespace for each interpreter no matter Zeppelin server is running inside k8s or not.
For example, user can start a spark interpreter in the namespace of spark:

Meanwhile, user can start a shell interpreter in the namespace of sh:

Hi, @Reamer
Thanks for the comments. I have moved the common functions (isRunningOnKubernetes(), readFile(), getInterpreterNamespace() and getCurrentK8sNamespace()) into org.apache.zeppelin.interpreter.launcher.K8sUtils to avoid duplicate code.
And I also make it more clear to distinguish the namespace of server and interpreter.

Instead of creating a ClusterRole and a ClusterRoleBinding, I would prefer an additional blueprint with a Role and a RoleBinding. Inside the RoleBinding the user has to replace the Namespace.

But if zeppelin server needs to create interpreter pod in another namespace, its service account needs to be granted a clusterrolebinding because a rolebinding only grants permissions within a specific namespace. What do you think?

In my opinion an additional step should be needed to give the Zeppelin server more rights on a namespace via a RoleBinding. Creating a ClusterRoleBinding gives too many rights to the zeppelin-server. With a ClusterRoleBinding the Zeppelin server has rights in namespaces that are not used by the Zeppelin server at all.

In my opinion an additional step should be needed to give the Zeppelin server more rights on a namespace via a RoleBinding. Creating a ClusterRoleBinding gives too many rights to the zeppelin-server. With a ClusterRoleBinding the Zeppelin server has rights in namespaces that are not used by the Zeppelin server at all.

Sorry for the late response. I agree that clusterrole may gives too many rights to the zeppelin server. But the zeppelin server  pod uses the service account created in https://github.com/apache/zeppelin/blob/master/k8s/zeppelin-server.yaml#L199, which is a namespaced resource.
Assuming that the zeppelin server and its service account are created in default namespace. And if the zeppelin-server.yaml uses the rolebinding to obtain the permissions in another namespace (such as spark), it cannot find the created service account in default namespace.
Or is there other ways to set the separate namespace with role and rolebinding?

To avoid creating the same Role in every namespace, a ClusterRole would be good.
You can then create a RoleBinding in each namespace, which has the Zeppelin server ServiceAccount as subject.
Example RoleBinding in spark namespace.
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: zeppelin-server-role-binding
  namespace: spark
subjects:
  - kind: ServiceAccount
    name: zeppelin-server
    namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: zeppelin-server-role

I think that should be sufficient. This RoleBinding should be created manuell by the user for each namespace, which should be used be zepperlin.

To avoid creating the same Role in every namespace, a ClusterRole would be good.
You can then create a RoleBinding in each namespace, which has the Zeppelin server ServiceAccount as subject.
Example RoleBinding in spark namespace.
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: zeppelin-server-role-binding
  namespace: spark
subjects:
  - kind: ServiceAccount
    name: zeppelin-server
    namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: zeppelin-server-role

I think that should be sufficient. This RoleBinding should be created manuell by the user for each namespace, which should be used be zepperlin.

Thanks for the comments. I have changed the clusterrolebinding to rolebinding and updated the docs. Now by default, user can create the interpreter pod in the default namespace. If the user wants to create and run the interpreter in another namespace, he also needs to add a rolebinding in the zeppelin-server.yaml

Now by default, user can create the interpreter pod in the default namespace.

Is it necessary to add a namespace?

Please remove the namespace parameter. Afterwards, the RoleBinding is created in the namespace in which the other resources are also created.

Sorry for the late response since I am busy with school stuff.
I have removed the namespace parameter and rebased this PR to avoid conflicts.


Added all contributors to readme

Closes: #14
please add level3, gssoc and gssoc-ext labels

@adarsh-singh01 please look into this pr

@adarsh-singh01 please merge the pr as only a day remains in the GSSOC...
At this its a request please dont let my work go waste 🙏


Fix RawStringDistinctExecutor integer overflow

select DISTINCT(n_comment) from nation limit 10

on tpc-h dataset fails on v2 engine due to integer overflow
[
  {
    "errorCode": 200,
    "message": "QueryExecutionError:\njava.lang.RuntimeException: Error executing query: [0]@<IP_ADDRESS>:49954 MAIL_RECEIVE(RANDOM_DISTRIBUTED)\n└── [1]@localhost:49966 MAIL_SEND(RANDOM_DISTRIBUTED)->{[0]@<IP_ADDRESS>@{49954,49954}|[0]}\n   └── [1]@localhost:49966 AGGREGATE_FINAL\n      └── [1]@localhost:49966 MAIL_RECEIVE(HASH_DISTRIBUTED)\n         └── [2]@localhost:49966 MAIL_SEND(HASH_DISTRIBUTED)->{[1]@localhost@{49966,49967}|[0]}\n...\nCaused by: java.lang.RuntimeException: Received error query execution result block: {200=QueryExecutionError:\njava.lang.AssertionError\n\tat org.apache.pinot.core.query.distinct.raw.BaseRawStringSingleColumnDistinctExecutor.getResult(BaseRawStringSingleColumnDistinctExecutor.java:70)\n\tat org.apache.pinot.core.query.distinct.raw.RawStringSingleColumnDistinctOnlyExecutor.getResult(RawStringSingleColumnDistinctOnlyExecutor.java:29)\n\tat org.apache.pinot.core.operator.query.DistinctOperator.getNextBlock(DistinctOperator.java:64)"
  }
]

@shenyu0127 This was supposed to be fixed within #10940, but introduced back in #10914. Let's be more careful next time :)

@Jackie-Jiang Well noted!


Hosting static assets

Note in Host and deploy pertaining to local hosting of static assets:

Blazor sets up its own static file middleware to handle static file requests to content from the Blazor client app, not from the server project.

Note first tho that a CDN should be used for static assets and mention+link Azure Storage/CDN for this purpose.
Note that static assets for the client-side app can't be hosted in the server-side app using Static File Middleware at this time.

Part of this issue (server-side can't host the static assets) was fixed in a release after this issue was opened.
Part of this issue (recommending a CDN) is something that won't be recommended explicitly.
Closing as a 'wont-fix'.


Unable to compile

My OS is Ubuntu 14.04.2 LTS and my machine is Acer Travelmate 240/250.
I'm unable to compile libvdpau-va-gl.
adam@acer:~/Other2/libvdpau-va-gl/libvdpau-va-gl-master/build$ cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr ..
-- The C compiler identification is GNU 4.8.4
-- The CXX compiler identification is GNU 4.8.4
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.26")
-- Looking for XOpenDisplay in /usr/lib/i386-linux-gnu/libX11.so;/usr/lib/i386-linux-gnu/libXext.so
-- Looking for XOpenDisplay in /usr/lib/i386-linux-gnu/libX11.so;/usr/lib/i386-linux-gnu/libXext.so - found
-- Looking for gethostbyname
-- Looking for gethostbyname - found
-- Looking for connect
-- Looking for connect - found
-- Looking for remove
-- Looking for remove - found
-- Looking for shmat
-- Looking for shmat - found
-- Looking for IceConnectionNumber in ICE
-- Looking for IceConnectionNumber in ICE - found
-- Found X11: /usr/lib/i386-linux-gnu/libX11.so
-- checking for module 'vdpau'
--   found vdpau, version 0.7
-- checking for module 'glib-2.0'
--   found glib-2.0, version 2.40.2
-- checking for module 'libva-x11'
--   found libva-x11, version 0.35.0
-- checking for module 'libswscale'
--   found libswscale, version 3.1.101
-- checking for modules 'gl;glu'
--   found gl, version 11.0.4
--   found glu, version 9.0.0
-- Configuring done
-- Generating done
-- Build files have been written to: /home/adam/Other2/libvdpau-va-gl/libvdpau-va-gl-master/build
adam@acer:~/Other2/libvdpau-va-gl/libvdpau-va-gl-master/build$ sudo make installScanning dependencies of target shader-bundle-tool
[  4%] Building C object glsl/CMakeFiles/shader-bundle-tool.dir/shader-bundle-tool.c.o
Linking C executable shader-bundle-tool
[  4%] Built target shader-bundle-tool
Scanning dependencies of target shader-bundle-src
[  9%] Generating shaders.c, shaders.h
[  9%] Built target shader-bundle-src
Scanning dependencies of target shader-bundle
[ 13%] Building C object glsl/CMakeFiles/shader-bundle.dir/shaders.c.o
Linking C static library libshader-bundle.a
[ 18%] Built target shader-bundle
Scanning dependencies of target vdpau_va_gl_obj
[ 22%] Building C object src/CMakeFiles/vdpau_va_gl_obj.dir/api-bitmap-surface.c.o
[ 27%] Building C object src/CMakeFiles/vdpau_va_gl_obj.dir/api-csc-matrix.c.o
[ 31%] Building C object src/CMakeFiles/vdpau_va_gl_obj.dir/api-device.c.o
[ 36%] Building C object src/CMakeFiles/vdpau_va_gl_obj.dir/api-output-surface.c.o
[ 40%] Building C object src/CMakeFiles/vdpau_va_gl_obj.dir/api-presentation-queue.c.o
[ 45%] Building C object src/CMakeFiles/vdpau_va_gl_obj.dir/api-video-decoder.c.o
/home/adam/Other2/libvdpau-va-gl/libvdpau-va-gl-master/src/api-video-decoder.c: In function ‘vdpDecoderCreate’:
/home/adam/Other2/libvdpau-va-gl/libvdpau-va-gl-master/src/api-video-decoder.c:78:14: error: ‘VDP_DECODER_PROFILE_H264_CONSTRAINED_BASELINE’ undeclared (first use in this function)
case VDP_DECODER_PROFILE_H264_CONSTRAINED_BASELINE:
^
/home/adam/Other2/libvdpau-va-gl/libvdpau-va-gl-master/src/api-video-decoder.c:78:14: note: each undeclared identifier is reported only once for each function it appears in
/home/adam/Other2/libvdpau-va-gl/libvdpau-va-gl-master/src/api-video-decoder.c: In function ‘vdpDecoderQueryCapabilities’:
/home/adam/Other2/libvdpau-va-gl/libvdpau-va-gl-master/src/api-video-decoder.c:383:10: error: ‘VDP_DECODER_PROFILE_H264_CONSTRAINED_BASELINE’ undeclared (first use in this function)
case VDP_DECODER_PROFILE_H264_CONSTRAINED_BASELINE:
^
/home/adam/Other2/libvdpau-va-gl/libvdpau-va-gl-master/src/api-video-decoder.c: In function ‘vdpDecoderRender’:
/home/adam/Other2/libvdpau-va-gl/libvdpau-va-gl-master/src/api-video-decoder.c:711:33: error: ‘VDP_DECODER_PROFILE_H264_CONSTRAINED_BASELINE’ undeclared (first use in this function)
if (decoderData->profile == VDP_DECODER_PROFILE_H264_CONSTRAINED_BASELINE ||
^
make[2]: *** [src/CMakeFiles/vdpau_va_gl_obj.dir/api-video-decoder.c.o] Error 1
make[1]: *** [src/CMakeFiles/vdpau_va_gl_obj.dir/all] Error 2
make: *** [all] Error 2

VDP_DECODER_PROFILE_H264_CONSTRAINED_BASELINE was added only in the recent version of libvdpau, that's why it's absent on 0.7. I've added compatibility macros' in 3fc29a50ad134551a8cf5424f93ce123d71cbb74. Could you check whenever source from master branch compiles now?
There were also other changes in code, so it would be nice if you check whenever it also works, not only compiles.

adam@acer ~/O/l/build> cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr ..
-- The C compiler identification is GNU 4.8.4
-- The CXX compiler identification is GNU 4.8.4
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.26")
-- Looking for XOpenDisplay in /usr/lib/i386-linux-gnu/libX11.so;/usr/lib/i386-linux-gnu/libXext.so
-- Looking for XOpenDisplay in /usr/lib/i386-linux-gnu/libX11.so;/usr/lib/i386-linux-gnu/libXext.so - found
-- Looking for gethostbyname
-- Looking for gethostbyname - found
-- Looking for connect
-- Looking for connect - found
-- Looking for remove
-- Looking for remove - found
-- Looking for shmat
-- Looking for shmat - found
-- Looking for IceConnectionNumber in ICE
-- Looking for IceConnectionNumber in ICE - found
-- Found X11: /usr/lib/i386-linux-gnu/libX11.so
-- checking for module 'vdpau'
--   found vdpau, version 0.7
-- checking for module 'glib-2.0'
--   found glib-2.0, version 2.40.2
-- checking for module 'libva-x11'
--   found libva-x11, version 0.35.0
-- checking for modules 'gl;glu'
--   found gl, version 11.0.4
--   found glu, version 9.0.0
-- Configuring done
-- Generating done
-- Build files have been written to: /home/adam/Other2/libvdpau-va-gl/build
adam@acer ~/O/l/build> sudo make install
Scanning dependencies of target shader-bundle-tool
[  4%] Building C object glsl/CMakeFiles/shader-bundle-tool.dir/shader-bundle-tool.c.o
Linking C executable shader-bundle-tool
[  4%] Built target shader-bundle-tool
Scanning dependencies of target shader-bundle-src
[  9%] Generating shaders.c, shaders.h
[  9%] Built target shader-bundle-src
Scanning dependencies of target shader-bundle
[ 13%] Building C object glsl/CMakeFiles/shader-bundle.dir/shaders.c.o
Linking C static library libshader-bundle.a
[ 18%] Built target shader-bundle
Scanning dependencies of target vdpau_va_gl
[ 22%] Building CXX object src/CMakeFiles/vdpau_va_gl.dir/api-bitmap-surface.cc.o
[ 27%] Building CXX object src/CMakeFiles/vdpau_va_gl.dir/api-csc-matrix.cc.o
[ 31%] Building CXX object src/CMakeFiles/vdpau_va_gl.dir/api-decoder.cc.o
/home/adam/Other2/libvdpau-va-gl/src/api-decoder.cc: In function ‘VdpStatus vdp::Decoder::Render_h264(std::shared_ptrvdp::Decoder::Resource, std::shared_ptrvdp::VideoSurface::Resource, void* const*, uint32_t, const VdpBitstreamBuffer*)’:
/home/adam/Other2/libvdpau-va-gl/src/api-decoder.cc:531:77: error: invalid static_cast from type ‘void* const*’ to type ‘const VdpPictureInfoH264*’
const auto *vdppi = static_cast<VdpPictureInfoH264 const *>(picture_info);
^
make[2]: *** [src/CMakeFiles/vdpau_va_gl.dir/api-decoder.cc.o] Error 1
make[1]: *** [src/CMakeFiles/vdpau_va_gl.dir/all] Error 2
make: *** [all] Error 2
It fails again.

There were another change in libvdpau: VdpPictureInfo type was changed from void * to void, which made C++ compiler unhappy.
I see no sane way to make compatibility wrappers for that change, so I've changed code to use bundled VDPAU headers instead (4f53876cce65e3fc79312f13c697f83bef226480 and a8cd82fe4dd86fde30f362febfe298d1639b79c3). Should be better now.

Now it compiles with any problem, but I don't think it works:
adam@acer:/media/adam-theta/Video/Youtube$ VDPAU_DRIVER=va_gl
adam@acer:/media/adam-theta/Video/Youtube$ mpv -vo=vdpau How_To_Stop_Procrastinating-Joe_Jo.mp4
Playing: How_To_Stop_Procrastinating-Joe_Jo.mp4
(+) Video --vid=1 () (h264)
(+) Audio --aid=1 --alang=eng () (aac)
[VS] Software VDPAU backend library initialized
libva info: VA-API version 0.35.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/i386-linux-gnu/dri/i915_drv_video.so
libva info: va_openDriver() returns -1
[VS] error (vdpDeviceCreateX11): gl error 1280
[vo/vdpau] Error when calling vdp_device_create_x11: 25
Error opening/initializing the selected video_out (-vo) device.
Video: no video
Exiting... (Errors when loading file)

Are you sure you are using recent version? I think I removed all occurrences of string "vdpDeviceCreateX11" from the code. There is no even a function named that anymore.
It could be there are another version of libvdpau-va-gl built, and libvdpau uses it. There is a list of paths that are checked, and first found library is loaded.
Try to set environment variable VDPAU_DRIVER_PATH to the path where newly built libvdpau_va_gl.so.1 resides.

I've just tried to test on Ubuntu 12.04, and found that libvdpau it has, doesn't know about VDPAU_DRIVER_PATH variable yet. You need either to place libvdpau_va_gl.so to some library path, or add its directory to the library search path. You could do the latter by export LD_LIBRARY_PATH=$PWD. That will add current directory.

adam@acer:~/Other2/libvdpau-va-gl/build$ VDPAU_DRIVER_PATH=/home/adam/Other2/libvdpau-va-gl/build/
adam@acer:~/Other2/libvdpau-va-gl/build$ LD_LIBRARY_PATH=$PWD
adam@acer:~/Other2/libvdpau-va-gl/build$ VDPAU_DRIVER=va_gl
adam@acer:~/Other2/libvdpau-va-gl/build$ mpv -vo=vdpau /media/adam-theta/Video/Youtube/How_To_Stop_Procrastinating-Joe_Jo.mp4
Playing: /media/adam-theta/Video/Youtube/How_To_Stop_Procrastinating-Joe_Jo.mp4
(+) Video --vid=1 () (h264)
(+) Audio --aid=1 --alang=eng () (aac)
[VS] Software VDPAU backend library initialized
libva info: VA-API version 0.35.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/i386-linux-gnu/dri/i915_drv_video.so
libva info: va_openDriver() returns -1
[VS] error (vdpDeviceCreateX11): gl error 1280
[vo/vdpau] Error when calling vdp_device_create_x11: 25
Error opening/initializing the selected video_out (-vo) device.
Video: no video
Exiting... (Errors when loading file)
Still no avail :/

$ LD_LIBRARY_PATH=$PWD

That only sets variable for current shell session, but not for child processes. Try to use
$ export LD_LIBRARY_PATH=$PWD

Nope, still the same.
I think it might be some sort of a problem with i915 driver though.

Do you have another compiled libvdpau_va_gl.so anywhere in the filesystem?

I think it might be some sort of a problem with i915 driver though.

It could be. Function vdpDeviceCreateX11() tries to compile couple of GPU shaders that used for colorspace conversion. As far as I know, Mesa falls back to software emulation of shaders if hardware have no appropriate support. But it could be that particular version used is returning error instead.

As the initially reported bug was solved, I'm going to close this issue.
As for "Acer Travelmate 240/250" machine, I did a quick search and found that's it has a Pentium 4 processor. It doesn't have a hardware video decoder, which was only introduced in Core family since about 2010.
Without VA-API support, libvdpau-va-gl will be slower and more CPU time consuming than using Xv output in video players like MPlayer or mpv, since it does data format conversion in a non-optimal way.


[TEIIDTOOLS-842]  Updating classes based on the change on using syndesis parent pom

Working thru the changes due to changing to use the syndesis as the parent pom.   This is not the last of the changes, only the next set that's been resolved.

Van, I don't see your pom change in any of the prs.  Are you simply removing the checkstyle and some of the basepom property overrides when switching to their parent pom?

There's a combination of changes, as I move thru the next issue that pop's up.   The dependencies changes,  I think were fixed in the initial PR.  The subsequent changes appear to be class changes based on the checkstyle issue.   But until I get to the end, I'm not sure what issue I still may hit.


Thank you for raising an issue 💗
You can self assign the issue by commenting /assign in comment 😀
Make sure you follow CODE OF CONDUCT and CONTRIBUTING GUIDELINES 🚀
Don’t Forget to ⭐ our GameZone🎮
Make sure you join our Discord🕹

Hey @shraddha761!
We are already having a similar game request in #1935 👀
Make sure you come up with a cool unique idea 😀
Waiting for your new game idea 💗.

Hey @shraddha761 ! Thank you so much for your raising the issue💗
It’s all yours, you can come anytime again and make some contributions! 🚀
Alone, we can do little, but together we can do so much! 😇


HDR Vivid descriptor and AVS2 audio descriptor are only in MPEG context (no DVB PDS)

Also, build registration ids from 'readable' values.

Good idea this MakeREGID.


Principle analysis

hello! I have recently been researching the principles behind TrollRestore. I find this method of installing TrollStore to be quite surprising and impressive. I am wondering if iOS performs a backup operation on the /var/backup directory after a restart. I have searched for some information but haven't found any relevant introductions.
Please provide me with some guidance on this matter. I would be very grateful for your help.
Have a good day!

Yes, the FinishRestoreFromBackup binary is executed early on by launchd, its job is to move files from /var/backup to their respective paths.

Hi! Thank you for the great work. I'm also looking into how the restore exploit works. I understand that the exploit first writes the target file to a temp location, creates a hard link in the system directory linking to the target file, and then breaks the hard link to the original file so that the system file is replaced with our custom file. I wonder if this understanding is correct? I'm also confused by the inode and owner parameters when creating the backup files to restore, which is set to 0 and 33 (or 501) respectively. What's special about these values?
Thanks in advance.

Yes, the FinishRestoreFromBackup binary is executed early on by launchd, its job is to move files from /var/backup to their respective paths.

Thank you for your explanation!
I tried to write content of PersistenceHelper_Embedded directly to the target app for installation. But this method failed. The target app crashed when it was opened.
I want to know the reason. The code is as follows.
back = backup.Backup(
        files=[
            backup.ConcreteFile(
                "",
                f"SysContainerDomain-../../../../../../../../var/backup/var/containers/Bundle/Application/{app_uuid}/{app}/{app.split('.')[0]}",
                owner=33,
                group=33,
                contents=helper_contents,

            )
            backup.ConcreteFile("", "SysContainerDomain-../../../../../../../.." + "/crash_on_purpose", contents=b""),
        ]
    )

Thank you in advance for your reply!

Yes, the FinishRestoreFromBackup binary is executed early on by launchd, its job is to move files from /var/backup to their respective paths.

Thank you for your explanation!
I tried to write content of PersistenceHelper_Embedded directly to the target app for installation. But this method failed. The target app crashed when it was opened. I confirmed that the app has been replaced.
I want to know the reason. The code is as follows.
back = backup.Backup(
        files=[
            backup.ConcreteFile(
                "",
                f"SysContainerDomain-../../../../../../../../var/backup/var/containers/Bundle/Application/{app_uuid}/{app}/{app.split('.')[0]}",
                owner=33,
                group=33,
                contents=helper_contents,

            )
            backup.ConcreteFile("", "SysContainerDomain-../../../../../../../.." + "/crash_on_purpose", contents=b""),
        ]
    )

Thank you in advance for your reply!

Hi! Thank you for the great work. I'm also looking into how the restore exploit works. I understand that the exploit first writes the target file to a temp location, creates a hard link in the system directory linking to the target file, and then breaks the hard link to the original file so that the system file is replaced with our custom file. I wonder if this understanding is correct? I'm also confused by the inode and owner parameters when creating the backup files to restore, which is set to 0 and 33 (or 501) respectively. What's special about these values?
Thanks in advance.

I think setting inode to 0 just makes the hard links of the target app and PersistenceHelper_Embedded point to the same file. I have tried changing the inode to other values to achieve binary replacement.
Regarding the onwer, I guess it is related to the apple backup file Manifest.mbdb`, the structure of mbdb, and involves the file permissions.

Hi! Thank you for the great work. I'm also looking into how the restore exploit works. I understand that the exploit first writes the target file to a temp location, creates a hard link in the system directory linking to the target file, and then breaks the hard link to the original file so that the system file is replaced with our custom file. I wonder if this understanding is correct? I'm also confused by the inode and owner parameters when creating the backup files to restore, which is set to 0 and 33 (or 501) respectively. What's special about these values?
Thanks in advance.

I think setting inode to 0 just makes the hard links of the target app and PersistenceHelper_Embedded point to the same file. I have tried changing the inode to other values to achieve binary replacement.
Regarding the onwer, I guess it is related to the apple backup file Manifest.mbdb, the structure of mbdb, and involves the file permissions.

Hi! Thank you for the great work. I'm also looking into how the restore exploit works. I understand that the exploit first writes the target file to a temp location, creates a hard link in the system directory linking to the target file, and then breaks the hard link to the original file so that the system file is replaced with our custom file. I wonder if this understanding is correct? I'm also confused by the inode and owner parameters when creating the backup files to restore, which is set to 0 and 33 (or 501) respectively. What's special about these values?
Thanks in advance.

Regarding the owners of the files, 33 is _installd (which is who the all binary should be owned by) and 501 is mobile. As for the actual process of restoring the binary, I think @JJTech0130 and @verygenericname would be better suited to explain.

Thanks for your explanation!

The reason we hard link to install TrollStore is because if you were to try to set it to executable in the exploitable backup domain, it would just ignore the executable bit. However, if you hard link it to a domain that allows the executable bit, the bit also gets applied to the one in the exploitable domain. The reason why we don't write the binary straight to the apps bundle is because the sandbox doesn't allow for it as it's a hard link. This is the reason why we break the hard link, too, because if we didn't FinishRestoreFromBackup would fail because of the sandbox, and cause a bootloop as a result because apple just made it panic the device when it fails.


show balances on the send page

Really annoying to not know what your balance is when you are trying to send

Yeah important!

Hey @irisli I see that there is a contribution label here and I was wondering if this is still an issue? I agree that moving asset balance to step 2 would be very helpful so that you can choose asset based on balance -- maybe even show minimum balance here as well?

Yes, that would also be useful. You can contribu

@irisli shall we close this? Looks like it exists in Step 3:

Yeah, lets close, then do a overhaul of send


Accelerate CVE DB update: parallelize fetching last modification timestamps

When checking the need for updating the CVE DB the last modification timestamps of the yearly CVE update files are being fetched from a remote host. Currently (year 2017) this sums up to 17 requests. As these requests are being executed sequentially this can make up a couple of seconds (depeding on the internet connection). This could easily be parallelized.

Thanks for the PR!


TextField's text is reset after turning off display or navigating away from the page

Steps to Reproduce

Create a new project
Include only TextField widget
Run the project
Type some text, then navigate away from the page by going to the device home screen or turning off display.
Go back to the appliation.

Expected results: text is still in the TextField
Actual results: data is lost

Logs
flutter run --verbose:
[  +20 ms] executing: [B:\src\flutter\] git -c log.showSignature=false log -n 1 --pretty=format:%H
[  +84 ms] Exit code 0 from: git -c log.showSignature=false log -n 1 --pretty=format:%H
[        ] f139b11009aeb8ed2a3a3aa8b0066e482709dde3
[        ] executing: [B:\src\flutter\] git describe --match v*.*.* --first-parent --long --tags
[  +94 ms] Exit code 0 from: git describe --match v*.*.* --first-parent --long --tags
[        ] v1.12.13+hotfix.9-0-gf139b1100
[   +7 ms] executing: [B:\src\flutter\] git rev-parse --abbrev-ref --symbolic @{u}
[  +89 ms] Exit code 0 from: git rev-parse --abbrev-ref --symbolic @{u}
[        ] origin/stable
[        ] executing: [B:\src\flutter\] git ls-remote --get-url origin
[  +84 ms] Exit code 0 from: git ls-remote --get-url origin
[        ] https://github.com/flutter/flutter.git
[ +124 ms] executing: [B:\src\flutter\] git rev-parse --abbrev-ref HEAD
[  +93 ms] Exit code 0 from: git rev-parse --abbrev-ref HEAD
[        ] stable
[ +129 ms] executing: B:\Android\android-sdk\platform-tools\adb.exe devices -l
[ +651 ms] Exit code 0 from: B:\Android\android-sdk\platform-tools\adb.exe devices -l
[   +1 ms] List of devices attached
           270cecbf10017ece       device product:crownltexx model:SM_N960F device:crownlte transport_id:3
           emulator-5554          device product:sdk_gphone_x86_arm model:AOSP_on_IA_Emulator device:generic_x86_arm transport_id:1
[  +31 ms] More than one device connected; please specify a device with the '-d <deviceId>' flag, or use '-d all' to act on all devices.
[   +5 ms] B:\Android\android-sdk\platform-tools\adb.exe -s 270cecbf10017ece shell getprop
[ +721 ms] ro.hardware = samsungexynos9810
[   +3 ms] B:\Android\android-sdk\platform-tools\adb.exe -s emulator-5554 shell getprop
[ +690 ms] ro.hardware = ranchu
[   +4 ms] SM N960F            • 270cecbf10017ece • android-arm64 • Android 10 (API 29)
[  +26 ms] AOSP on IA Emulator • emulator-5554    • android-x86   • Android 9 (API 28) (emulator)
[  +19 ms] "flutter run" took 2,372ms.

#0      throwToolExit (package:flutter_tools/src/base/common.dart:28:3)
#1      RunCommand.validateCommand (package:flutter_tools/src/commands/run.dart:289:7)
<asynchronous suspension>
#2      FlutterCommand.verifyThenRunCommand (package:flutter_tools/src/runner/flutter_command.dart:590:11)
#3      FlutterCommand.run.<anonymous closure> (package:flutter_tools/src/runner/flutter_command.dart:515:33)
<asynchronous suspension>
#4      AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:146:29)
#5      _rootRun (dart:async/zone.dart:1126:13)
#6      _CustomZone.run (dart:async/zone.dart:1023:19)
#7      _runZoned (dart:async/zone.dart:1518:10)
#8      runZoned (dart:async/zone.dart:1465:12)
#9      AppContext.run (package:flutter_tools/src/base/context.dart:145:18)
#10     FlutterCommand.run (package:flutter_tools/src/runner/flutter_command.dart:505:20)
#11     CommandRunner.runCommand (package:args/command_runner.dart:197:27)
#12     FlutterCommandRunner.runCommand.<anonymous closure> (package:flutter_tools/src/runner/flutter_command_runner.dart:413:21)
<asynchronous suspension>
#13     AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:146:29)
#14     _rootRun (dart:async/zone.dart:1126:13)
#15     _CustomZone.run (dart:async/zone.dart:1023:19)
#16     _runZoned (dart:async/zone.dart:1518:10)
#17     runZoned (dart:async/zone.dart:1465:12)
#18     AppContext.run (package:flutter_tools/src/base/context.dart:145:18)
#19     FlutterCommandRunner.runCommand (package:flutter_tools/src/runner/flutter_command_runner.dart:364:19)
#20     CommandRunner.run.<anonymous closure> (package:args/command_runner.dart:112:25)
#21     new Future.sync (dart:async/future.dart:224:31)
#22     CommandRunner.run (package:args/command_runner.dart:112:14)
#23     FlutterCommandRunner.run (package:flutter_tools/src/runner/flutter_command_runner.dart:250:18)
#24     run.<anonymous closure>.<anonymous closure> (package:flutter_tools/runner.dart:63:22)
#25     _rootRun (dart:async/zone.dart:1126:13)
#26     _CustomZone.run (dart:async/zone.dart:1023:19)
#27     _runZoned (dart:async/zone.dart:1518:10)
#28     runZoned (dart:async/zone.dart:1502:12)
#29     run.<anonymous closure> (package:flutter_tools/runner.dart:61:18)
<asynchronous suspension>
#30     AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:146:29)
#31     _rootRun (dart:async/zone.dart:1126:13)
#32     _CustomZone.run (dart:async/zone.dart:1023:19)
#33     _runZoned (dart:async/zone.dart:1518:10)
#34     runZoned (dart:async/zone.dart:1465:12)
#35     AppContext.run (package:flutter_tools/src/base/context.dart:145:18)
#36     runInContext (package:flutter_tools/src/context_runner.dart:64:24)
#37     run (package:flutter_tools/runner.dart:50:10)
#38     main (package:flutter_tools/executable.dart:65:9)
#39     main (file:///B:/src/flutter/packages/flutter_tools/bin/flutter_tools.dart:8:3)
#40     _startIsolate.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:305:32)
#41     _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:174:12)

flutter analyze:
No issues found! (ran in 3.3s)

flutter doctor -v :
[√] Flutter (Channel stable, v1.12.13+hotfix.9, on Microsoft Windows [Version 10.0.18363.778], locale en-US)
    • Flutter version 1.12.13+hotfix.9 at B:\src\flutter
    • Framework revision f139b11009 (4 weeks ago), 2020-03-30 13:57:30 -0700
    • Engine revision af51afceb8
    • Dart version 2.7.2

[√] Android toolchain - develop for Android devices (Android SDK version 29.0.2)
    • Android SDK at B:\Android\android-sdk
    • Android NDK location not configured (optional; useful for native profiling support)
    • Platform android-29, build-tools 29.0.2
    • ANDROID_HOME = B:\Android\android-sdk
    • Java binary at: B:\Program Files\Android\Android Studio\jre\bin\java
    • Java version OpenJDK Runtime Environment (build 1.8.0_212-release-1586-b04)
    • All Android licenses accepted.

[√] Android Studio (version 3.6)
    • Android Studio at B:\Program Files\Android\Android Studio
    • Flutter plugin version 45.1.1
    • Dart plugin version 192.7761
    • Java version OpenJDK Runtime Environment (build 1.8.0_212-release-1586-b04)

[√] VS Code (version 1.44.2)
    • VS Code at C:\Users\m_sha\AppData\Local\Programs\Microsoft VS Code
    • Flutter extension version 3.9.1

[√] Connected device (2 available)
    • SM N960F            • 270cecbf10017ece • android-arm64 • Android 10 (API 29)
    • AOSP on IA Emulator • emulator-5554    • android-x86   • Android 9 (API 28) (emulator)

• No issues found!

I tested this on an old project but it doesn't have this issue, then I copied the android folder of this old project to the project with the issue , and that fixed it!
I think it's something related to the AndroidX upgrade.
I have uploded the repo to repoduce this issue, it has two branches, master (with the issue) and fixed (no issue):
https://github.com/mshwf/flutter_TextField_issue

Hi @mshwf
I checked and I have this result https://github.com/flutter/flutter/issues/51947#issuecomment-595108332 with stable v1.12.13+hotfix.9.
Are you experiencing this with current beta channel? https://github.com/flutter/flutter/issues/47628#issuecomment-618460501
https://github.com/flutter/flutter/issues/51947#issuecomment-604007683
Please follow up on https://github.com/flutter/flutter/issues/51947 and https://github.com/flutter/flutter/issues/47628 and comment there,
I'm closing the current one as duplicate.
If you disagree please write in the comments and I will reopen it.
Thank you


How to erase contents in store() after each test?

Hi guys,
I am having an issue emptying out the store across my tests. This causes errors when I define models with the same id across multiple tests. Here is a really simply JSbin to illustrate - http://jsbin.com/qiwev/4/edit
  /*=================== APP ====================*/
Ember.MODEL_FACTORY_INJECTIONS = false;
App = Ember.Application.create();

App.Foo = DS.Model.extend({
});

/*=================== TESTS ====================*/

emq.globalize();
App.setupForTesting();
App.injectTestHelpers();
setResolver(Ember.DefaultResolver.create({ namespace: App }));
App.rootElement = '#ember-testing';

moduleForModel('foo', 'Foo model', {
  teardown: function() {
    // this.store().unloadAll('foo');
  }
});

test('Test 1', function() {
  var store = this.store();

  var model = this.subject({
    id: 1
  });

  ok(true);

});

test('Test 2', function() {
  var store = this.store();

  var model = this.subject({
    id: 1
  });

  ok(true);
});

test('Test 3', function() {
  var store = this.store();

  var model = this.subject({
    id: 1
  });

  ok(true);
});

The error I get is:
Error: Assertion Failed: The id 1 has already been used with another record of type App.Foo.

The commented out this.store().unloadAll('foo'); on teardown doesn't help either. So any suggestions on how I can reset the store across each test? Should that possibly be the default case as well to keep all tests independent? Thanks

I made a helper function:
// tests/helpers/store.js

import Ember from 'ember';

const {
  run
} = Ember;

function waitForStore(assert, timeout=50) {
  const done = assert.async();

  run.later(this, function() {
    done();
  }, timeout);
}

export {
  waitForStore
};

my tests look like that:
import Ember from 'ember';
import { waitForStore } from '../../helpers/store';

const {
  run
} = Ember;

moduleForModel('project', 'Project', {
  needs: ['model:task']
});

test('it creates a simple project', function(assert) {
  expect(1);

  const store = this.store();

  run(this, function() {
    let project = store.createRecord('project', {
      name: 'Simple project'
    });

    assert.equal(project.get('name'), 'Simple project');
    assert.equal(project.get('tasks.length'), 0);
  });

  // extend the timeout waitForStore(assert, 100);
  waitForStore(assert);
});

@fsmanuel Could you expand a little on what waitForStore actually does? It appears as if you're using waitForStore() to call the async callback after a specified period. And this should give anything in teardown enough time to run? I'm still having trouble getting properties of my object to properly clear out after running Ember.run App, App.destroy

this seems no longer relevant, closing due to age


False positive DOC503 in 0.5.12

False positive DOC503 error in version 0.5.12?
------------------------Bug report---------------------
[Example]:
def check_nonnegative_bug(number: float) -> None:
    """Check if the provided number is non-negative.

    Args:
        number(float): The number to check.

    Returns:
        None

    Raises:
        ValueError: when the number is less than zero.
        ValueError: when the number is zero.
    """
    if number < 0.0:
        raise ValueError("The number must be positive.")
    if number == 0.0:
        raise ValueError("The number must be positive.")

[To reproduce]
Running pydoclint --style=google --check-return-types=False .  on terminal in a conda env (Env details see below) gives the pydoclint error message below.
[pydoclint error message]
DOC503: Function check_positive exceptions in the "Raises" section in the docstring do not match those in the function body Raises values in the docstring: ['ValueError', 'ValueError']. Raised exceptions in the body: ['ValueError'].
[Expected output]

no violations

[Environment]

pydoclint                 0.5.12                   pypi_0    pypi
python                    3.10.16         h870587a_1_cpython    conda-forge

------------------------Good example---------------------
My guess: It seems that the false alarm on DOC503 occurs only when there are multiple errors of the same type. For errors of different types (see the example below), no pydoclint complaints.
[Example]:
def check_nonnegative_ok(number: float) -> None:
    """Check if the provided number is non-negative.

    Args:
        number(float): The number to check.

    Returns:
        None

    Raises:
        TypeError: when the type of number is not float.
        ValueError: when the number is less than zero.
    """
    if not isinstance(number, float):
        raise TypeError("The type of number must be float.")
    if number <= 0.0:
        raise ValueError("The number must be non-negative.")

[Output]:

no violations

[Other details]:
All other details are the same as in the bug-example.

Hi @Waschenbacher , this is not a bug, but a feature.
Please see the code here: https://github.com/jsh9/pydoclint/blob/dfd9494b25cede6b46bdfa13f2124f54377e0bdb/pydoclint/utils/return_yield_raise.py#L98-L100
We deduplicated and sorted the exceptions within a function.  The reason is that very often we have functions with many exceptions (e.g., 5 TypeErrors, 3 ValueErrors, 2 IOErrors, ...), so it's not very practical to list every raised exception in the correct order in the docstring.  (Note: if you don't want deduplication, the only sensible for a linter is to enforce the order.)
It's not difficult to add another config option to suppress deduplication, but the key question is whether we should add such an option.  My current position is no, unless there are very compelling reasons.


test issue 8

Hi, I have some suggestions to help solving this issue!
Here is a list of similar issues:

test issue 123

test issue 7

test issue 5

test issue 8

test issue 7

Hope this is useful!
Cheers,
Cinnabot.


[Cookbook] Basic deployment to firebase hosting

I'd like to contribute a quick recipe to describe the simplest way to deploy vue-cli to firebase hosting.
This is introductory level, the purpose of it is to help new users quickly deploy. It is not meant to be a best practice in industry standard production deployments.

I think this is a great idea for a recipe, but the recipe linked doesn't follow cookbook format. Looking forward to your revision!

Sorry for the delay. Instead of writing out a recipe, I added a deployment section to the vue-cli docs/README.md . I felt that was more appropriate. Thanks !


./musicsync --spotify-apicode produces no output

Good evening,
thanks a lot for this great project!
I managed to compile musicsync in my WSL Ubuntu 18.04 environment without any error.
However, when it comes to connecting it to Spotify there seems to be an issue.
That is, calling `./musicsync --spotify-apicode´  just produces the standard output:
`Usage: spotify-toniebox-sync.jar [--daemon | --input INPUT --output OUTPUT]
--daemon
Run in daemon mode to sync periodically all lists in the properties file
--input INPUT
Define an input source for a one-time run
--output OUTPUT
Define an output destination for a one-time run
Input Plugins:
--spotify-apicode
Generate an API Code URL to get attach Application to Spotify Account
--spotify-code CODE
Generate a Refresh token out of the Spotify Code
Output Plugins:
--toniebox-username USERNAME
Set the Toniebox username to login to Toniebox
--toniebox-password PASSWORD
Set the password for the Toniebox user for login
--filesystem-directory DIRECTORY
Set the output path for the files`
No error and the logfile is empty as well.
I followed the instructions carefully and checked it several times...
Do you know about this issue?
Kind regards
Florian

I think this is the same error that I'm also getting #22

@cg-laser Thanks for your immediate response!
Yes, sounds pretty similar.
My Java version is identical with yours:
openjdk version "1.8.0_275"
OpenJDK Runtime Environment (build 1.8.0_275-8u275-b01-0ubuntu1~18.04-b01)
OpenJDK 64-Bit Server VM (build 25.275-b01, mixed mode)
Maybe its interesting to you that my musicsync script already was executable...

I tracked back the issue and found the root cause in the parseArguments method in the Application.java
https://github.com/maximilianvoss/spotify-toniebox-sync/blob/da17628f99ce711fa8b66c992029168086b346ec/application/src/main/java/rocks/voss/musicsync/Application.java#L121
That is, if no --input --output is specified, and the daemon is turned off as well, the parser complains and will not call the parser methods of the plugins.
Inserting another "else if" statement before the else in line 121 which returns true does the trick!
else if (!isDaemon && StringUtils.isEmpty(argInputUri) && StringUtils.isEmpty(argOutputUri)) {
            log.debug(("No input and output specified. Parameters will be passed to plugin parsers.");
            return true;
        }

@maximilianvoss  maybe you can check,  whether this has any undesired side-effects and then fix it...

Thanks @floflunse to mention this fix. I found out one thing that might be good to add to the documentation: The spotify redirect URI also needs to be the same as specified on the "spotify app" that I creased on developers.spotify.com


the private keyword is misplaced in emitted ocaml code

The following reason code
type privateVariant = private
  | BigSize of int
  | SmallSize of int;

type doubleEqualsDoublePrivateVariant =
  privateVariant =
  private
    | BigSize of int
    | SmallSize of int;

emits the following ill-formed ocaml code:
type privateVariant = private
  | BigSize of int
  | SmallSize of int
type doubleEqualsDoublePrivateVariant = private privateVariant =
  | BigSize of int
  | SmallSize of int

The expected well-formed ocaml code is:
type privateVariant = private
  | BigSize of int
  | SmallSize of int
type doubleEqualsDoublePrivateVariant = privateVariant = private
  | BigSize of int
  | SmallSize of int

Note the movement of the private keyword to the right-hand side of the final equals sign. This test case appears in formatTest/wrappingTest.re.

closed as it's not an 'us' issue