Could not get class storage for eloquent

Describe the bug
Running Psalm with --debug-by-line on a Laravel 8 project I get:
 /var/www/html/app/Models/User.php:1

   Exception

  Could not get class storage for eloquent
Stack trace in the forked worker:
#0 /var/www/html/vendor/vimeo/psalm/src/Psalm/Internal/Analyzer/ClassAnalyzer.php(750): Psalm\Internal\Provider\ClassLikeStorageProvider->get('Eloquent')
#1 /var/www/html/vendor/vimeo/psalm/src/Psalm/Internal/Analyzer/ClassAnalyzer.php(394): Psalm\Internal\Analyzer\ClassAnalyzer::addContextProperties(Object(Psalm\Internal\Analyzer\ClassAnalyzer), Object(Psalm\Storage\ClassLikeStorage), Object(Psalm\Context), 'App\\Models\\Mapp...', 'Illuminate\\Data...', Array)
#2 /var/www/html/vendor/vimeo/psalm/src/Psalm/Internal/Analyzer/FileAnalyzer.php(213): Psalm\Internal\Analyzer\ClassAnalyzer->analyze(Object(Psalm\Context), Object(Psalm\Context))
#3 /var/www/html/vendor/vimeo/psalm/src/Psalm/Internal/Codebase/Analyzer.php(343): Psalm\Internal\Analyzer\FileAnalyzer->analyze(NULL)
#4 /var/www/html/vendor/vimeo/psalm/src/Psalm/Internal/Fork/Pool.php(193): Psalm\Internal\Codebase\Analyzer->Psalm\Internal\Codebase\{closure}(5, '/var/www/html/a...')
#5 /var/www/html/vendor/vimeo/psalm/src/Psalm/Internal/Codebase/Analyzer.php(409): Psalm\Internal\Fork\Pool->__construct(Array, Object(Closure), Object(Closure), Object(Closure), Object(Closure))
#6 /var/www/html/vendor/vimeo/psalm/src/Psalm/Internal/Codebase/Analyzer.php(272): Psalm\Internal\Codebase\Analyzer->doAnalysis(Object(Psalm\Internal\Analyzer\ProjectAnalyzer), 5)
#7 /var/www/html/vendor/vimeo/psalm/src/Psalm/Internal/Analyzer/ProjectAnalyzer.php(636): Psalm\Internal\Codebase\Analyzer->analyzeFiles(Object(Psalm\Internal\Analyzer\ProjectAnalyzer), 5, false, true)
#8 /var/www/html/vendor/vimeo/psalm/src/psalm.php(700): Psalm\Internal\Analyzer\ProjectAnalyzer->check('/var/www/html/', true)
#9 /var/www/html/vendor/vimeo/psalm/src/psalm.php(884): Psalm\{closure}(Array)
#10 /var/www/html/vendor/vimeo/psalm/psalm(2): require_once('/var/www/html/v...')
#11 {main}

Ofc I have the issue also without --debug-by-line.
Impacted Versions
barryvdh/laravel-debugbar                      v3.5.7
barryvdh/laravel-ide-helper                    v2.9.1
fruitcake/laravel-cors                         v2.0.4
laravel/framework                              v8.44.0
laravel/sail                                   v1.7.0
laravel/tinker                                 v2.6.1
psalm/plugin-laravel                           v1.4.5
spatie/laravel-ray                             1.18.0
vimeo/psalm                                    4.7.3

Additional context
User.php is the standard user class that comes with every fresh new project.
I tried to remove psalm and psalm/plugin-laravel, clear the cache, and I get the same Could not get class storage for eloquent in other parts of the codebase, for example
/var/www/html/tests/Feature/Models/Mapper/v1/TestCase.php:1

I'm stuck with this.
Edit: I forgot to mention that I'm on php 8.0.3

I get a similar error except for illuminate\foundation\application. It might be related so I decided to add it here instead of creating a new issue (I hope that is okey)
./vendor/bin/psalm --debug-by-line

...<large part left out>

Have populated Signet\DataContainer\DataContainerException
Have populated Symfony\Component\Console\Exception\InvalidOptionException
Have populated Symfony\Component\Console\Helper\TableStyle
Have populated Symfony\Component\Console\Helper\TableRows
Have populated Symfony\Component\Console\Helper\TableCell
Have populated Symfony\Component\Console\Helper\TableSeparator
Have populated Symfony\Component\Console\Helper\ProgressBar
Have populated Signet\DataContainer\Util
Have populated Symfony\Component\Console\Helper\TableCellStyle
ClassLikeStorage is populated
FileStorage is populating
FileStorage is populated
Registering stub files
Parsing /workspace/vendor/vimeo/psalm/stubs/CoreGenericFunctions.phpstub
Deep scanning /workspace/vendor/vimeo/psalm/stubs/CoreGenericFunctions.phpstub
Parsing /workspace/vendor/vimeo/psalm/stubs/CoreGenericClasses.phpstub
Deep scanning /workspace/vendor/vimeo/psalm/stubs/CoreGenericClasses.phpstub
Uncaught InvalidArgumentException: Could not get class storage for illuminate\foundation\application in /workspace/vendor/vimeo/psalm/src/Psalm/Internal/Provider/ClassLikeStorageProvider.php:42
Stack trace:
#0 /workspace/vendor/vimeo/psalm/src/Psalm/Internal/PhpVisitor/Reflector/ClassLikeNodeScanner.php(196): Psalm\Internal\Provider\ClassLikeStorageProvider->get('illuminate\\foun...')
#1 /workspace/vendor/vimeo/psalm/src/Psalm/Internal/PhpVisitor/ReflectorVisitor.php(182): Psalm\Internal\PhpVisitor\Reflector\ClassLikeNodeScanner->start(Object(PhpParser\Node\Stmt\Interface_))
#2 /workspace/vendor/nikic/php-parser/lib/PhpParser/NodeTraverser.php(200): Psalm\Internal\PhpVisitor\ReflectorVisitor->enterNode(Object(PhpParser\Node\Stmt\Interface_))
#3 /workspace/vendor/nikic/php-parser/lib/PhpParser/NodeTraverser.php(91): PhpParser\NodeTraverser->traverseArray(Array)
#4 /workspace/vendor/vimeo/psalm/src/Psalm/Internal/Scanner/FileScanner.php(86): PhpParser\NodeTraverser->traverse(Array)
#5 /workspace/vendor/vimeo/psalm/src/Psalm/Internal/Codebase/Scanner.php(577): Psalm\Internal\Scanner\FileScanner->scan(Object(Psalm\Codebase), Object(Psalm\Storage\FileStorage), true, Object(Psalm\Progress\DebugProgress))
#6 /workspace/vendor/vimeo/psalm/src/Psalm/Internal/Codebase/Scanner.php(328): Psalm\Internal\Codebase\Scanner->scanFile('/workspace/vend...', Array, true)
#7 /workspace/vendor/vimeo/psalm/src/Psalm/Internal/Codebase/Scanner.php(447): Psalm\Internal\Codebase\Scanner->Psalm\Internal\Codebase\{closure}(1, '/workspace/vend...')
#8 /workspace/vendor/vimeo/psalm/src/Psalm/Internal/Codebase/Scanner.php(292): Psalm\Internal\Codebase\Scanner->scanFilePaths(1)
#9 /workspace/vendor/vimeo/psalm/src/Psalm/Codebase.php(471): Psalm\Internal\Codebase\Scanner->scanFiles(Object(Psalm\Internal\Codebase\ClassLikes), 1)
#10 /workspace/vendor/vimeo/psalm/src/Psalm/Config.php(1834): Psalm\Codebase->scanFiles()
#11 /workspace/vendor/vimeo/psalm/src/Psalm/Internal/Analyzer/ProjectAnalyzer.php(623): Psalm\Config->visitStubFiles(Object(Psalm\Codebase), Object(Psalm\Progress\DebugProgress))
#12 /workspace/vendor/vimeo/psalm/src/psalm.php(700): Psalm\Internal\Analyzer\ProjectAnalyzer->check('/workspace/', true)
#13 /workspace/vendor/vimeo/psalm/src/psalm.php(884): Psalm\{closure}(Array)
#14 /workspace/vendor/vimeo/psalm/psalm(2): require_once('/workspace/vend...')
#15 {main}
(Psalm 4.7.3@38c452ae584467e939d55377aaf83b5a26f19dd1 crashed due to an uncaught Throwable)

Using php version 8.0.7:
barryvdh/laravel-ide-helper                    v2.9.1    Laravel IDE Helper, generates correct PHPDocs for all Fac...
laravel/framework                              v8.46.0   The Laravel Framework.
laravel/tinker                                 v2.6.1    Powerful REPL for the Laravel framework.
php-standard-library/psalm-plugin              1.1.1     Psalm plugin for the PHP Standard Library
psalm/plugin-laravel                           v1.4.7    A Laravel plugin for Psalm
psalm/plugin-mockery                           0.7.0     Psalm plugin for Mockery
psalm/plugin-phpunit                           0.16.0    Psalm plugin for PHPUnit
vimeo/psalm                                    4.7.3     A static analysis tool for finding errors in PHP applicat...

I am using this in a Laravel library using orchestral/testbench.

@Ilyes512 I may have just resolved your issue with https://github.com/psalm/psalm-plugin-laravel/pull/152 -- but if not, can you please open a new ticket? It's a bit different than this issue

@giraz82 could you please share some code that is triggering the issue? For example, what is referencing Eloquent ?

I'm experincieng similar issues recently, and it just popped up again:
Scanning files...

   InvalidArgumentException

  Could not get class storage for illuminate\support\facades\ratelimiter

  at vendor/vimeo/psalm/src/Psalm/Internal/Provider/ClassLikeStorageProvider.php:42
     38▕     public function get(string $fq_classlike_name): ClassLikeStorage
     39▕     {
     40▕         $fq_classlike_name_lc = strtolower($fq_classlike_name);
     41▕         if (!isset(self::$storage[$fq_classlike_name_lc])) {
  ➜  42▕             throw new \InvalidArgumentException('Could not get class storage for ' . $fq_classlike_name_lc);
     43▕         }
     44▕
     45▕         return self::$storage[$fq_classlike_name_lc];
     46▕     }

Clearing the cache with psalm --clear-cache solves the issue.

could you please share some code to reproduce @caugner

@giraz82 could you please share some code that is triggering the issue? For example, what is referencing Eloquent ?

@mr-feek in my case, it's just the standard User class that comes with Laravel:
<?php declare(strict_types=1);

namespace App\Models;

use Illuminate\Database\Eloquent\Factories\HasFactory;
use Illuminate\Foundation\Auth\User as Authenticatable;
use Illuminate\Notifications\Notifiable;

/**
 * Class User.
 */
final class User extends Authenticatable
{
    use HasFactory;
    use Notifiable;

    /**
     * The attributes that are mass assignable.
     *
     * @var array
     */
    protected $fillable = [
        'name',
        'email',
        'password',
    ];

    /**
     * The attributes that should be hidden for arrays.
     *
     * @var array
     */
    protected $hidden = [
        'password',
        'remember_token',
    ];

    /**
     * The attributes that should be cast to native types.
     *
     * @var array
     */
    protected $casts = [
        'email_verified_at' => 'datetime',
    ];
}

Nothing special as you can see.
But I think that the behaviour is quite random and not related to a specific class nor block of code.
I launched again psalm in this project (I needed to comment out it to proceed in my project) and now I'm getting the same error but on a different place:
...
Forking analysis
Getting /var/www/html/app/Providers/EventServiceProvider.php
Analyzing /var/www/html/app/Providers/EventServiceProvider.php
/var/www/html/app/Providers/EventServiceProvider.php:1
/var/www/html/app/Http/Middleware/Authenticate.php:22

   Exception

  Could not get class storage for eloquent
Stack trace in the forked worker:
#0 /var/www/html/vendor/vimeo/psalm/src/Psalm/Internal/Analyzer/ClassAnalyzer.php(750): Psalm\Internal\Provider\ClassLikeStorageProvider->get('Eloquent')
#1 /var/www/html/vendor/vimeo/psalm/src/Psalm/Internal/Analyzer/ClassAnalyzer.php(394): Psalm\Internal\Analyzer\ClassAnalyzer::addContextProperties(Object(Psalm\Internal\Analyzer\ClassAnalyzer), Object(Psalm\Storage\ClassLikeStorage), Object(Psalm\Context), 'App\\Models\\Soap...', 'Illuminate\\Data...', Array)
#2 /var/www/html/vendor/vimeo/psalm/src/Psalm/Internal/Analyzer/FileAnalyzer.php(213): Psalm\Internal\Analyzer\ClassAnalyzer->analyze(Object(Psalm\Context), Object(Psalm\Context))
#3 /var/www/html/vendor/vimeo/psalm/src/Psalm/Internal/Codebase/Analyzer.php(343): Psalm\Internal\Analyzer\FileAnalyzer->analyze(NULL)
#4 /var/www/html/vendor/vimeo/psalm/src/Psalm/Internal/Fork/Pool.php(193): Psalm\Internal\Codebase\Analyzer->Psalm\Internal\Codebase\{closure}(0, '/var/www/html/a...')
...

Also here Authenitcate middleware is the Laravel standard one:
<?php declare(strict_types=1);

namespace App\Http\Middleware;

use Illuminate\Auth\Middleware\Authenticate as Middleware;
use Illuminate\Http\Request;

/**
 * Class Authenticate.
 */
final class Authenticate extends Middleware
{
    /**
     * Get the path the user should be redirected to when they are not authenticated.
     *
     * @param Request $request
     *
     * @return string|null
     */
    protected function redirectTo($request): ?string
    {
        if (!$request->expectsJson()) {
            return route('login');
        }

        return null;
    }
}

Let me know if you need further info.

Here's another one:
Parsing /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Relations/Concerns/SupportsDefaultModels.php
Deep scanning /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Relations/Concerns/SupportsDefaultModels.php
Parsing /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Relations/Concerns/InteractsWithPivotTable.php
Deep scanning /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Relations/Concerns/InteractsWithPivotTable.php
Parsing /var/www/html/vendor/laravel/framework/src/Illuminate/Support/Facades/Facade.php
Deep scanning /var/www/html/vendor/laravel/framework/src/Illuminate/Support/Facades/Facade.php

   InvalidArgumentException

  Could not get class storage for illuminate\support\facades\paralleltesting

  at vendor/vimeo/psalm/src/Psalm/Internal/Provider/ClassLikeStorageProvider.php:42
     38▕     public function get(string $fq_classlike_name): ClassLikeStorage
     39▕     {
     40▕         $fq_classlike_name_lc = strtolower($fq_classlike_name);
     41▕         if (!isset(self::$storage[$fq_classlike_name_lc])) {
  ➜  42▕             throw new \InvalidArgumentException('Could not get class storage for ' . $fq_classlike_name_lc);
     43▕         }
     44▕
     45▕         return self::$storage[$fq_classlike_name_lc];
     46▕     }

Running it again yielded the same error.

Opened https://github.com/vimeo/psalm/issues/5971 upstream

I think https://github.com/psalm/psalm-plugin-laravel/commit/b287a0ad17f5086eb40fbc1c481768fe24588950 should help resolve these issues.

I'm going to close this -- please open a new ticket with code to reproduce if you run into any more issues

@mr-feek Will you release a new minor version soon? I will test and report back then, because our Psalm runs fail consistently.

@caugner in the mean time try using dev-master! Or else invoke CI with --no-cache. I'm hoping to have a new minor version out in the next week or so.

@mr-feek I'm still experiencing the same issue with da4e6bc:
$ composer show | grep -E 'psalm|laravel'
andrey-helldar/laravel-lang-publisher v10.0.4            Publisher lang files for the Laravel Framework, Jetstream, Fortify, Cashier, Spark and Nova from Laravel-Lang/lang
andrey-helldar/laravel-support        v2.0.1             Various helper files for the Laravel and Lumen frameworks
barryvdh/laravel-ide-helper           v2.9.1             Laravel IDE Helper, generates correct PHPDocs for all Facade classes, to improve auto-completion.
beyondcode/laravel-dump-server        1.7.0              Symfony Var-Dump Server for Laravel
fruitcake/laravel-cors                v2.0.4             Adds CORS (Cross-Origin Resource Sharing) headers support in your Laravel application
laravel-lang/lang                     10.0.2             Languages for Laravel
laravel/framework                     v8.49.0            The Laravel Framework.
laravel/sail                          v1.8.3             Docker files for running a basic Laravel application.
laravel/sanctum                       v2.11.2            Laravel Sanctum provides a featherweight authentication system for SPAs and simple APIs.
laravel/tinker                        v2.6.1             Powerful REPL for the Laravel framework.
laravel/ui                            v3.3.0             Laravel UI utilities and presets.
psalm/plugin-laravel                  dev-master da4e6bc A Laravel plugin for Psalm
sentry/sentry-laravel                 2.7.0              Laravel SDK for Sentry (https://sentry.io)
spatie/laravel-ray                    1.17.4             Easily debug Laravel apps
vimeo/psalm                           4.8.1              A static analysis tool for finding errors in PHP applications
$ vendor/bin/psalm
Scanning files...

   InvalidArgumentException

  Could not get class storage for illuminate\support\facades\ratelimiter

  at vendor/vimeo/psalm/src/Psalm/Internal/Provider/ClassLikeStorageProvider.php:43
     39▕     public function get(string $fq_classlike_name): ClassLikeStorage
     40▕     {
     41▕         $fq_classlike_name_lc = strtolower($fq_classlike_name);
     42▕         if (!isset(self::$storage[$fq_classlike_name_lc])) {
  ➜  43▕             throw new \InvalidArgumentException('Could not get class storage for ' . $fq_classlike_name_lc);
     44▕         }
     45▕
     46▕         return self::$storage[$fq_classlike_name_lc];
     47▕     }

I'm afraid I can't really help without a reproducer -- I haven't run into this specific issue at all

Alright, I will run Psalm directly via PhpStorm from now on, with Xdebug enabled and a breakpoint set in that line 43. Hopefully I can reproduce the issue with that setup, so that I can dig a bit in the debug state to see what's going on and why the class is missing in the storage.

Not sure that running it over PhpStorm will help. I'm seeing Could not get class storage for illuminate\support\facades\paralleltesting as well on a pretty much fresh setup

I still have the error appearing randomly in a new Laravel project that was working fine until today:
Could not get class storage for eloquent

  at vendor/vimeo/psalm/src/Psalm/Internal/Provider/ClassLikeStorageProvider.php:43
     39▕     public function get(string $fq_classlike_name): ClassLikeStorage
     40▕     {
     41▕         $fq_classlike_name_lc = strtolower($fq_classlike_name);
     42▕         if (!isset(self::$storage[$fq_classlike_name_lc])) {
  ➜  43▕             throw new \InvalidArgumentException('Could not get class storage for ' . $fq_classlike_name_lc);
     44▕         }
     45▕
     46▕         return self::$storage[$fq_classlike_name_lc];
     47▕     }

I'm using
"psalm/plugin-laravel": "1.5.1",
"vimeo/psalm": "4.13.1"

But also updating psalm to 4.14.0 makes any difference.
Any idea?


missing checksum.exs file

With the latest release, I get a step further (thank you!). But I'm now seeing a missing checksum file when I deploy. Any idea how to fix this?
I added this to my mix.exs:
{:hnswlib, github: "elixir-nx/hnswlib", tag: "v0.1.2"}
Then on compilation:
#0 44.63 Error happened while installing hnswlib from precompiled binary: "missing checksum.exs file".
Full error:
#0 44.28 Downloading precompiled NIF to /root/.cache/hnswlib-nif-2.17-x86_64-linux-gnu-0.1.2.tar.gz
#0 44.63 Error happened while installing hnswlib from precompiled binary: "missing checksum.exs file".
#0 44.63
#0 44.63 Attempting to compile hnswlib from source...
#0 44.64 /bin/sh: 4: cmake: not found
#0 44.64 incomplete build of hnswlib found in '/app/_build/prod/lib/hnswlib/cmake_hnswlib', please delete that directory and retry
#0 44.64 make: *** [Makefile:37: /app/_build/prod/lib/hnswlib/priv/hnswlib_nif.so] Error 1
#0 44.64 could not compile dependency :hnswlib, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile hnswlib --force", update it with "mix deps.update hnswlib" or clean it with "mix deps.clean hnswlib"

Originally posted by @cbh123 in https://github.com/elixir-nx/hnswlib/issues/6#issuecomment-1719683148

@cbh123 please use the released package, as that includes the checksum.exs file. otherwise you need to have cmake installed so it compiles it locally. :)

Ah, whoops! Yeah, updated dep to use {:hnswlib, "~> 0.1.2"} and works perfectly. Thanks!!


Make "macros" sub project cross build.

This PR contains two parts to enable cross build:

Side by side with the Scala 2 macro "withStrRep" we now have the
Scala 3 macro version of it.
The "q" annotation was a Scala 2 annotation and now got changed
to a Java runtime annotation. That way we can easily support
both Scala versions without the need for two different code paths.
This comes at the price of not being binary compatible with old versions
of QueryDB, but we are source compatible. As discussed on Slack we deem
this OK.

I incorporated your suggestions and also removed the Scala 2 reflection workaround code which should not be necessary anymore since we stop using Scala 2 reflection.


Guidance needed

Hi,
Thank you for writing this plugin, it looks very promising, but I'm not sure how to make it work.
I have tried to use a Graphite data source with a serie like:
alias(cumulative(xxx.stats_counts.processing.success), 'processingCount')
The output in a SingleStat panel works, I get a number.
Now, how am I supposed to say in the Flowcharting plugin that I want to have this value somewhere ?
Thank you
Alex

Hi Guidance,
Thanks for using Flowcharting, i'll try it with graphite and i'll make an animated gif for example.
Version 3 coming soon next week i think,
French ?

Thank you ! Yes Arnaud, I am French indeed :)

Hi, any news on this please ? thank you

Salut bidule
0.3 not stable, 0.2.5 today or tomorrow.
Je m'active

0.2.5 pushed, in progress
you can found zip at
https://github.com/algenty/grafana-flowcharting/files/3036986/agenty-flowcharting-panel.zip

dont forget to name your metrics with alias

Thanks a lot !

pas de quoi


viewer.html not working?! hyperlapse.tllabs.io outage?

Hello,
I found this project and I tried to make a video using the files here on github but the file viewer.html seems to have a serious issue... It keeps displaying the same message: "ERROR: Could not retrieve panorama for the following reason: ZERO_RESULTS" an then a black screen appears...
Elsewhere, I can't access the hyperlapse.tllabs.io website. Is the project over?
Thank you for your time and support,
Julien
Btw I'm a complete newbie at programming, excuse me!

im in the same boat please help

+1

Yep, Teehan+lax closes their doors last year. I got it working a few months ago but it seems dead now :(

Just for the reference:
https://medium.com/the-prototype/the-best-300-000-teehan-lax-ever-spent-4f5b059b37e6#.cmcymve7k


OMNIDB - INVALID APP TOKEN

Hello Good night!
Please, someone can help me, I am taking INVALID APP TOKEN error to access the OMNIDB app.
I already changed the port to 8001 and apparently it is running, it just doesn't open the app screen. My OS is Windows.
Thank you in advance.
I am also having multiple user error.

For this solution, this morning, I moved and changed port 8000 to 8001 and everything worked.
I don't know if there would be another solution, but the one I found was this one.
Thank you very much.

Hello Fernandomori2019
I have exactly this problem with installing OmniDB.
Could you please help me you solve this problem and change the port?
Thanks,
Siena


Update Calico-Docker link

Please disregard Travis, should be a temporary issue
Ok to merge after review :smile:

:tada:


chore: Make tests more resilient when run alone.

When running tests locally, somehow they fail due probably on my global pnpm setup.
I made minor modifications to make them more resilient.

@ivan-tymoshenko are you ok with this? can it land?

@ivan-tymoshenko are you ok with this? can it land?

yes, lgtm


Make it possible to add space between abbreviation and postfix currency #577

Thanks!!!
Just a little fix: could you please rearrange your commits as follow please?

one commit with all the code + changelog
one commit with AUTHORS (so that potentially reverting your code commit won't remove you from the contributors list :smile:)

Thanks in advance

@all-contributors please add @marvec  for bug, code

@BenjaminVanRyseghem Thank you for your quick reaction to this PR. I hope I managed to put it in the right shape. Do you have any estimates on when this could make its way to a release?

Hello @BenjaminVanRyseghem, I understand that you probably have other ton of work and I appreciate your time invested in this awesome project. However, we are sort of dependent on this fix. I would like to figure out whether it is worth waiting for a next numbro release, or we should invest the time and fork it temporarily... Many thanks!

@marvec I understand, and I am sorry I can't be faster. I'll try to release somehow tonight after I put the kids to bed.
In the meantime, you can always have npm point to a github sha and avoid a fork

Thank you very much! We really appreciate your help. I did not know about the ability to point to a github sha. Thank you for extending my knowledge!

Just letting you know - sha does not work - there are missing files that get created with the build (language.min.js for example). So no luck :-(

@marvec I'm sorry 😞 I'll try to release it asap, and I will try so setup a built-branch so that someone can pick a sha


Workaround various GitHub Actions bugs

This change is

+@SeanCurtis-TRI for review.

+@SeanCurtis-TRI for review.

We are always single reviewer here. CI pretty tells you everything you need to know in this case. Broken -> Workaround -> Not broken.

We are always single reviewer here. CI pretty tells you everything you need to know in this case. Broken -> Workaround -> Not broken.


Rewire deliverable APIs to the backend

Use latest backend types
Update these APIs to use the backend instead of mock data:

List deliverables
Get deliverable
Update deliverable

TODO:
[x] Rename 'reason' to 'feedback' in a few other places
[x] Hook up the update call
[x] Figure out where the templateUrl is supposed to come from
[x] Search and sort on deliverables on the frontend (in #2242)

Current dependencies on/for this PR:

#2235  👈
main

This stack of pull requests is managed by Graphite.

Merge activity

Feb 29, 6:45 PM EST: @nickgraz started a stack merge that includes this pull request via Graphite.


Add aerosol support

It should be pretty simple to add support for varying aerosols, using the same design as the cloud file.

For anyone who has arrived here thinking 'where is the aerosol support?' this is the place for you!
I do not currently use aerosols with libradtran, so I am not clear what aerosol support should look like. So this is where I need your help. What should the aerosol support look like? How would you interact with it? What features are necessary?
I can't promise I will add them all immediately, but if you ave any suggestions, please let me know.


✨ migrations: seeding is part of init db task

refs #7489
When merging this PR we have achieved:

running all init tasks as separate process

So the init task contains three steps:

create tables
add fixtures
add default settings

All these steps must happen before Ghost can start.
important to know

we keep all fixture code for now (we just copied the heart of it out of the current location to schema folder) - fixture utils and fixtures.json
bootstrapping code only contains one single line after merging this PR

return sephiroth.utils.isDatabaseOK()

Sephiroth checks the health of the database by ensuring that all init tasks were fully executed.
So migrations table looks like this when Ghost is allowed to start:
"1-create-tables.js"	"init"
"2-create-fixtures.js"	"init"
"3-insert-settings.js"	"init"

next steps

backup and reset
add migrate task to sephiroth
delete all old code for migrations
use Ghost-CLI

TODO

[x] test env

ready for review 🇹🇭


[feat] OpenWrt packaging

Is your feature request related to a problem? Please describe.
We package Domoticz for OpenWrt, and so far we have built it to use OpenZWave.
Newer versions of Domoticz have deprecated the OpenZWave support, so I'm looking at how to package zwave-js-ui for OpenWrt so that it can be part of the embedded router firmware.
I would very much appreciate some help working out how to do clean packaging from known source releases.
There are some OpenWrt node packages in https://github.com/partylover/openwrt-node-packages maintained by @partylover and I've experimented a little bit but I haven't made much progress. Packaging zwave-js itself seems like it might be a little simpler since it's in npm.

Hi David! Unfortunally I have zero experience with OpenWrt, couldn't you use a packaged version of zwave-js-ui instead?
https://github.com/zwave-js/zwave-js-ui/releases/download/v8.4.1/zwave-js-ui-v8.4.1-linux.zip

If I could work out how to build the packaged version from a known set of sources

My idea is not you to build it but to directly use the already builded version.

OpenWrt is an embedded distribution using musl instead of glibc; the one I run domoticz on is a MIPS CPU. There aren't prebuilt versions for those platforms — we need to work out how to build it.

@dwmw2
zwave-js is now supported. Please try it out.
https://github.com/nxhack/openwrt-node-packages/tree/master/node-zwave-js
22.03, 21.02 will be merged after build test is completed.

Thanks! I'll see if I can test it. Not quite sure how to test zwave-js on its own... the next request is to package zwave-js-ui :)

I installed it and tried putting some of the example at https://zwave-js.github.io/node-zwave-js/#/getting-started/quickstart into a foo.js file and running it (with a package.json with type=module. I get:
root@U7623:~# node foo.js
node:internal/process/esm_loader:97
    internalBinding('errors').triggerUncaughtException(
                              ^

Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'zwave-js' imported from /root/foo.js
Did you mean to import zwave-js/build/index.js?

Sorry to be clueless... what should I do here to test it?
FWIW if I do npm i zwave-js then it does get a little bit further... it at least opens the serial port, and then segfaults.

Thanks for the help with this, @nxhack. It would be great to be able to upgrade the OpenWrt package of Domoticz to the latest version and switch to using zwave-js-ui.
If it's more convenient, I'm happy to let you have remote access to a test OpenWrt system with a Z-Wave dongle attached.


CLOUD-48457 make the mybatis log nicer

@akanto please take a look!

Jenkins build finished, all tests passed.
Refer to this link for build results: http://ci.sequenceiq.com/job/periscope-build-pull-request/36/


Adding new directories

Adding directories:
app/modules        - it is better to store modules in this directory instead of lib - lib should be left for monkey patching or extending functionality of ruby core (String.to_bool etc.)
app/workers         - put all workers (sidekiq/resque)
app/validations     - put all custom validation classes there

@ondrejbartas @XeeD do we still want this here? Let's fix/merge or close it pls.

@ognevsky I think this is good idea. But please ask @ondrejbartas in the office what he thinks about it.


feature suggestion:  configurable extra option for docker run command

current latest minifabric can set additional environment variable to docker containers via settings section in spec.yaml
but it doesn't have a way to set additional option to docker run command.
then, I cannot specify '--restart" nor "--log-opt" policies  even I want to use minifabric  in long run use cases.
first I think adding  '--restart=always'  by hard coding style into playbook, but logging policy may be different in each case.
then, I think it is good to have configurable extra docker option in spec.yaml as described bellow snippet.
it may be single(choice 1, same option to all containers), or multiple (choice 2, different option for each type).
I'm not sure which choice is preferable for minifabric maintainers. any suggestion?
# ansible playbooks
- name: Start xxx
  command: >-
    docker run -d ... --name {{item.fullname}} ... {{ extra_docker_option }} ... # append extra docker option.

# spec.yaml
# choice 1: single;  add the same option to all containers(ca,peer,orderer)
fabric:
  ca:
  peers:
  orderers:
  settings:
  extra_docker_option: " --restart=always  --log-opt max_size=Xmb --log-opt max_file=N " # new key

# choice 2: different option in each type, same style as 'settings( for environment variable)'.
fabric:
  :
  settings:
  extra_docker_option:
     ca:       " --restart=always  --log-opt max_size=3mb  --log-opt max_file=3"
     peers:    " --restart=always  --log-opt max_size=10mb --log-opt max_file=10"
     orderers: " --restart=always  --log-opt max_size=5mb  --log-opt max_file=5"

Either is fine, prefer choice1. Thanks.

Not sure we need to call it extra docker option, can we call it container options?

@litong01 ok, I take choice 1, and change name to container_options.


Some blocks don't have integration with Presence Footsteps

In order of appearance:

Vent Wall
White Latex Block
Infuser
Purifier
Dark Latex Block
Box Pile
Vent Hatch
Retinal Scanner
Laser Emitter
Lab Table
Cardboard Box
Iron Crate
Lab Door
Blue Lab Door
Computer
IV Rack
Speaker
Vent Fan
Wolf Gas Canister
Canned Peaches
Latex Puddles (if you can walk on it long enough to hear it...)

Relevant Video:
https://user-images.githubusercontent.com/100542617/230703145-3d44ac4f-fed4-41c7-9da2-0bc9d0c72944.mp4

After looking into this issue, it appears to be from Presence Footsteps being an over-engineered mod, and requiring mod developers to create their own block -> sound map, instead of determining sounds by block type. However, I will look into creating a block -> sound map for the Changed mod.

Alright cool, no more (mostly) silent footsteps. Thanks! :D

Actually, something just came to my mind @LtxProgrammer
Presence Footsteps has a option to change the "stance" from 2 legs to 4 legs. Making the footsteps sound as if you had 4 legs, obviously.
Unsure how easy or if possible, but this option could be perhaps changed by the mod whenever you become the fused knight transfer to 4 legs (because... you have 4 legs.)

I'll look into that.

Done. Need to possibly implement custom processor, for slithering.


feat(provisioning): Keycloak provisiong from custom config and policy data bootstrapping from config

This is definitely not the best implementation of custom provisioning, just something i made to get it working.
Known issues:

there is some code duplication with the current keycloak provisioing, they should eventually be combined into one file with the option of providing a config yaml
the data provisioning structs are kind of just recreating the existing proto scructs, should be rewritten to utilize those instead
there is some functionality missing from the data provisioing like resource_mappings and kas_registry

@elizabethhealy this is awesome work! Would this deprecate the other key cloak provisioner? If so, could we validate that it can do the same thing and remove the defunct code with this PR?

@elizabethhealy this is awesome work! Would this deprecate the other key cloak provisioner? If so, could we validate that it can do the same thing and remove the defunct code with this PR?

@jrschumacher not fully, this doesn't support the token exchange setup yet, i can work on adding that

@elizabethhealy Can you sync with @mkleene. I think he has some keycloak command changes in pr #546 as well.

@jrschumacher i cleaned this up a little to remove some of the duplication with the other keycloak provisioning, the other kc provision command still exists but they now use the same functions


Get analysis result out of tool.check() or tool.visualize()

This way the main Clinic cli can print recommendations to the command line output, primarily for use in CI.

Some feedback if you don't mind.

"Doctor check" is cute, but I think you should choose a name that makes sense for the entire clinic suite. Maybe just analyze.

Include the raw data, for future projects we may wish to utilize that.

Consider excluding results.recommendation for now, it looks quite ad-hoc and I'm not sure it is required.


CppWinRTOptimized and .g.cpp

I have two cppwinrt projects. One of generates .g.cpp files and other does'nt.  Could'nt figure what build flag triggers this. Finally, may have traced it to 'CppWinRTOptimized'. Is this correct ?
If so, the behaviour is odd - because it breaks all compilation - rather randomly. Should the user component file to include ".g.cpp" ? is this by design ? If so, we can make it consistent irrespective of 'CppWinRTOptimized'

.g.cpp files are typically associated with types created to implement XAML applications and controls (the .g files are the portions of the C++ code generated by the XAML compiler). Is one app a XAML app, and the other not?
Ben

Both are winrt component DLLs.


Gmail search syntax for Remove-GSGmailMessage

Hi Nate,
I'm looking for some additional functionality that is closely related to the Remove-GSGmailMessage cmdlet. Specifically, feature parity with the Gmail search syntax functionality that a similar command in GAM currently has. I'd like to request one (or both) of the following features:

The ability to use Remove-GSGmailMessage with the Gmail search syntax. For example, if I wanted to delete all messages from a particular sender (from:example@domain.com).

A new cmdlet with the ability to modify labels on a particular email (also using the Gmail search syntax). For example, if I wanted to mark all messages from a particular Gmail search syntax query as spam or trash.

You can read the documentation for the GAM commands here:
https://github.com/jay0lee/GAM/wiki/ExamplesEmailSettings#modifying-user-emails
The reason I'm looking for this functionality is that I often use these commands to combat spam or phishing. Attackers will often send several emails with different message IDs. The search syntax would allow far greater flexibility in targeting these emails. I still find myself using GAM for removing emails specifically for the ability to use Gmail search syntax.
Thoughts?
Thanks,
Josh

Hey Josh - Definitely good ideas! I use PSGSuite myself for almost the same purpose (among many other), so I completely get the goal here!
For item 1 - I'll check out expanding the functionality so it's wrapped into a single command, but you can currently use the pipeline though to do it in a short one-liner (see below). Would it be worth adding in an additional MaxThreshold parameter in case you're not doing something as precise as searching by MessageId and want to ensure you don't accidentally delete more than expected?
# Trash everything not sent directly to me
Get-GSGmailMessageList -Filter "to:-me" | Remove-GSGmailMessage -Confirm:$false

For item 2 - Definitely need to expand on label management! I'll work on this as I can, pull requests also very much accepted! 😄

Alright, everything all fixed up and about to deploy in v2.15.2!

Added in Update-GSGmailMessageLabel with help from @dwrusse and included -Filter parameter
Updated Remove-GSGmailMessage with the same -Filter parameter
Fixed the pipeline support so that -User isn't needed to pass through the pipeline so the pipeline works exactly as expected and pulls the User from the objects passed into it

Demo with WhatIf + Verbose showing new Remove-GSGmailMessage functionality:

v2.15.2 has been deployed! Update and check it out, cheers!


Node selector - fix for snapshot import on v10

Combination of 2 small things:

I need the ability to choose which node of my cluster my tezos node is running on. This is for an external project leveraging tezos-k8s - this is pretty standard thing for a helm chart to support
snapshot import is broken on v10. While it's being fixed, I am adding a workaround. issue

@harryttd I changed the node selector setting to go at nodes level instead of globally. It works for me. Please re-review.

I changed the pr description a little. I hope that's ok


Fix links to static methods in marklogic namespace

Fixes (part of) https://github.com/marklogic/node-client-api/issues/231


Include and exclude by tag

Hi there,
is it possible to include and exclude objects by tag? I just want to separate some elements of a context diagram into grouped items and create additional diagrams. Documentation says we can include/exclude only by name and it looks a bit tedious if we have 2-3 or more groups.
Is there a better way?
views {
systemContext System "SystemContext" "System Context diagram." {
include *
        # exclude Grouped systems
        exclude System1 System2 System3

        # exclude Grouped relationships
        exclude rel_System1 rel_System2 rel_System3

        autoLayout
    }

    systemContext Grouped "Group1" "System Context diagram." {

        # include Grouped systems
        include System1 System2 System3

        # exclude relationship to Group
        exclude rel_to_Grouped
        autoLayout
    }

If you're using the Structurizr web renderer, you could try using the filtered view functionality, which allows you to exclude elements/relationships via tags -> https://structurizr.com/help/filtered-views

Thank you for prompt answer, I will try :)

It would be great to have a better way of filtering - tags, regex, etc


Micro-XRCE-DDS-Gen : How to handle different include directories ?

Hello everyone, I was currently using the develop branch of the xrce-gen tool and was working through some of the examples listed and went through the include.idl (available here) . As per the idl structure it seems that we try to include basic.idl into the former file and I tried to do a similar thing but this time keeping the files in different folders and I wasn't able to generate the necessary files..on the contrary the tool gives the error that the include files are not defined.I tried passing the include folder path as a potential value to the ppPath flag..unfortunately it did not fair any better.It would be great if I could get some help in this domain.
Environment : Using Ubuntu 22.04 and using the develop branch of this repo
Expected Behaviour : DDS gen successfully generates the necessary files for a given IDL(like the one here) that "includes" another IDL but the latter is in a different directory
Current Behaviour : DDS gen not able to generate the said files and runs into error :

error: 'your/diff/path/to/basic.idl' was not defined previously  when u try to straight up run the gen tool on the said idl
ERROR<callPreprocessor>: Cannot execute the preprocessor. Reason: Cannot run program "/your/diff/path/to/basic.idl": error=13, Permission denied when u try to add the necessary paths as values to ppPath tags

Could you check if you have the correct permission for every file? Check error=13, Permission denied

I can reproduce this. I have verified the file have user read/write permissions. Here is the output.
ryan@ryan-B650:~/Development/ardu_ws/src/ardupilot/libraries/AP_XRCE_Client/Idl$ microxrceddsgen -ppPath sensor_msgs/msg/ sensor_msgs/msg/JoyFeedbackArray.idl
openjdk version "11.0.18" 2023-01-17
OpenJDK Runtime Environment (build 11.0.18+10-post-Ubuntu-0ubuntu122.04)
OpenJDK 64-Bit Server VM (build 11.0.18+10-post-Ubuntu-0ubuntu122.04, mixed mode, sharing)
Loading templates...
Processing the file sensor_msgs/msg/JoyFeedbackArray.idl...
ERROR<callPreprocessor>: Cannot execute the preprocessor.  For now, we will put all the messages in the same folder and manually modify the files to not have include statements with paths.

@Ryanf55 The feature is ready, check it out: https://github.com/eProsima/Micro-XRCE-DDS-Gen/pull/60

Closing for now as #60 is merged.
Feel free to reopen if further modifications are needed.


File document/a1851451.txt contains UPPER_CASE_WORD insertions

Hi there!
Thank you for all your work on this project! It has been so helpful.
While observing the file, I think I have found low severity UPPERCASES issues on lines 5. The commit that introduced the issues was https://github.com/davmlaw/2024_assignment_5_adelaide_uni_bioinfo/commit/623532a735043b2d4e3c0acc87fe4bdd846ee1af, file named "extend extensible paradigms," made on the 23rd of September 2024.
The lines expected to read as;
Line 5 - "charge environmental car leader executive great gas. research loss lead since table court."
However, it actual lines read as;
Line 5 - "charge ENVIRONMENTAL XXXXXXX CAR leader executive great gas. RESEARCH loss lead since"
I plan on submitting a pull request once the issue is remediated by reverting the uppercase letters to lower case from the file.
Thank you again!

looks good, lost 1 point as commit should have been 5581dc9d6156206fc47385c3480a0927d21a4d95 (comment is right) that was for different file a1851176.txt


PushPlugin is undefined only in phonegap build

Expected Behaviour
After device is ready, window.PushNotification is defined
Actual Behaviour
After device is ready, window.PushNotification is undefined
Reproduce Scenario (including but not limited to)
When built using phonegap build, after device is initialized, wait a few seconds and then alert(window.PushNotification). See undefined
When building using Phonegap developer, it works.
Steps to Reproduce

Add to project config.xml:

<plugin name="phonegap-plugin-push" spec="~1.6.3">
<variable name="SENDER_ID" value="XXXXXXX" />
</plugin>

Build project on phonegap build for iOS only. Verify that, in plugins tab on build.cordova.com, the plugin is being recognized and built (1.6.3)
Open app. After device is ready and app is loaded, alert(window.PushNotification). See undefined.

Platform and Version (eg. Android 5.0 or iOS 9.2.1)
iOS
(Android) What device vendor (e.g. Samsung, HTC, Sony...)
Cordova CLI version and cordova platform version
cli-5.2.0 (3.9.1 / 4.1.1 / 3.8.1)
(this is the version on Build, which doesn't work)
6.2.1 for developer (which works)
Plugin version
1.6.3
Sample Push Data Payload
Sample Code that illustrates the problem
after device ready, alert(window.PushNotification)
Logs taken while reproducing problem

Oh, after 4+ hours of fighting with it, I just fixed it by downgrading to 1.4.5. Is it expected that the newest version doesn't work?

I'm having likely the same problem - IOS has undefined for PushNotification where Android is fine. I was also able to downgrade to v1.4.5 and have it work. Odd thing is that I believe it was working recently. The things I can remember changing recently are adding a couple plugins (com.telerik.plugins.wkwebview) comes to mind as a significant one and also changing to ios v3.9.2 for the cordova version so I could  build said plugin. Here is a list of my current plugins that are running on cordova<EMAIL_ADDRESS>if I upgrade to 1.6.3 the PushNotifications object is undefined:
com.telerik.plugins.nativepagetransitions 0.6.3 "Native Page Transitions"
com.telerik.plugins.wkwebview 0.6.9 "WKWebView Polyfill"
com.verso.cordova.clipboard 0.1.0 "Clipboard"
cordova-plugin-app-version 0.1.8 "AppVersion"
cordova-plugin-camera 2.2.0 "Camera"
cordova-plugin-compat 1.0.0 "Compat"
cordova-plugin-crosswalk-webview 1.6.1 "Crosswalk WebView Engine"
cordova-plugin-customurlscheme 4.1.5 "Custom URL scheme"
cordova-plugin-device 1.1.2 "Device"
cordova-plugin-file 4.2.0 "File"
cordova-plugin-file-transfer 1.5.1 "File Transfer"
cordova-plugin-google-analytics 0.8.1 "Google Universal Analytics Plugin"
cordova-plugin-inappbrowser 1.2.1 "InAppBrowser"
cordova-plugin-splashscreen 3.2.2 "Splashscreen"
cordova-plugin-statusbar 2.1.3 "StatusBar"
cordova-plugin-webserver 1.0.3 "CordovaWebServer"
cordova-plugin-whitelist 1.2.2 "Whitelist"
cordova-plugin-x-socialsharing 5.0.12 "SocialSharing"
ionic-plugin-keyboard 2.0.1 "Keyboard"
phonegap-plugin-barcodescanner 4.1.0 "BarcodeScanner"
phonegap-plugin-push 1.4.5 "PushPlugin"

Seems to work onwards to version 1.5.3. From 1.6.0 and higher it stops working for me on PGB.

Thanks. I have the exact same issue. IOS phonegap build doesn't work at all with the latest version 1.6.3. After changing the plugin version to 1.4.5 and 1.5.3, it works fine on ios now. Something must be wrong with the latest version.

Duplicate #779


Add author labels

Add a little menu in vscode to add icons/author details for your application. This would help with making applications and adding icons to them.
Thank you :)

@Colack can you please clarify what menu you would like and where? I do not fully understand what you would like here.
Thanks

This is a fair feature request, however we do not have it on our immediate plan. Thus assigning to backlog for now to gather more feedback. Thanks

Thank you :)

We closed this issue because we don't plan to address it in the foreseeable future. If you disagree and feel that this issue is crucial: we are happy to listen and to reconsider.
If you wonder what we are up to, please see our roadmap and issue reporting guidelines.
Thanks for your understanding, and happy coding!

Thank you


Feature: Switching input method while connecting to a remote host.

People who uses non-English input methods (e.g. Chinese, Japanese) have to switch input methods when returning to normal mode. This can be very annoying.
This problem can be solved in native Neovim by installing vim-im-select. But when connecting to a remote Linux host, im-select will not work, because Neovim is actually running on the remote host while I want to switch the local input methods.
Vscode-neovim is an excellent extension, and I hope this feature will be supported to make it even better. VSCodeVim supports a similar feature: https://github.com/VSCodeVim/Vim#input-method. It is also realised with im-select.

I accidentally found out that this feature can actually be realised. After installing vim-im-select and im-select on my Windows PC, the input method switches automatically when I return to normal mode with vscode-nvim, unlike in an SSH terminal.


Document FullCalendar add-on

Description
See https://github.com/jmix-framework/jmix/issues/2216

@Flaurite - call the add-on Calendar, rename module to calendar.
Dependencies will stay the same (io.jmix.fullcalendar:jmix-fullcalendar-flowui-starter).


⚠ Bofrost.at has degraded performance

In dbb1dd8, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 958 ms

Resolved: Bofrost.at performance has improved in 76d23fc after 13 minutes.


Convert expanded-state.js to vanilla JS

There's no actual need for jQuery in there, might as well remove that dependency.

This was fixed in https://github.com/cfpb/cfgov-refresh/commit/c5cd6be5fb28309e317beb9174070070373eceaa


Fix tf-addons for upcoming keras 3 default.

Description
Keras 3.0 will become default in TF 2.16 (and is currently default in tf-nightly).  This breaks this tf-addons package.  Here we make minimal changes to return functionality in a backward-compatible way.
Fixes # (issue)
None known reported.
Type of change

[X] Bug fix

Checklist:

[X] I've properly formatted my code according to the guidelines

[] By running Black + Flake8
[X] By running pre-commit hooks

[ ] This PR addresses an already submitted issue for TensorFlow Addons
[ ] I have made corresponding changes to the documentation
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] This PR contains modifications to C++ custom-ops

How Has This Been Tested?
Imported with tf-nightly, keras-nightly, tf_keras-nightly, setting both TF_USE_LEGACY_KERAS=0 and 1.

@howl-anderson @pedrolarben @ssaishruthi @hyang0129
You are owners of some files modified in this pull request.
Would you kindly review the changes whenever you have the time to?
Thank you very much.


Allow disabling automatic fallback to <IP_ADDRESS> and <IP_ADDRESS>

https://github.com/lima-vm/lima/blob/b1212298e59e4799ae16b0e4c36561c5bd294d83/pkg/hostagent/dns/dns.go#L24
We should probably have an env var like LIMA_DISABLE_FALLBACK_DNS=1.
I also wonder if there is some "standard" env var to disable these well-known fallback DNS.
Similar to:

https://github.com/moby/moby/issues/47541

would moving to Quad9 be sufficient?
Also,  is Clouldflare's <IP_ADDRESS> compliant?


[C-Api] fix typo

Fix typo in sample code for custom easy filter.
Signed-off-by: Jaeyun<EMAIL_ADDRESS>

:memo: TAOS-CI Version: 1.5.20200925. Thank you for submitting PR #238. Please a submit 1commit/1PR (one commit per one PR) policy to get comments quickly from reviewers. Your PR must pass all verificiation processes of cibot before starting a review process from reviewers. If you are new member to join this project, please read manuals in documentation folder and wiki page. In order to monitor a progress status of your PR in more detail, visit http://ci.nnstreamer.ai/.


fix cn for vars cn versions

construct cn unit by using dict, not class instance

tested on old version~ pass


Move ResourceID to fabric8 kubernetes client

The io.javaoperatorsdk.operator.processing.event.ResourceID class seems to be useful also outside the operator, it would be nice to move it to the fabric8 kubernetes client

I think we had discussed this at some point but I guess that didn't go further :)
It would maybe make more sense to open that issue in the fabric8 client project, though?

I will tomorrow

https://github.com/fabric8io/kubernetes-client/issues/3991

To mention also here, this is called ObjectKey, in controller runtime.
https://github.com/kubernetes-sigs/controller-runtime/blob/c46b410039fdc9d620a4217bc7c63c5a7fffae46/pkg/client/interfaces.go#L31-L31
Would be a little better to have consistent naming, but it's not that important.

Maybe worth renaming it before we release 3.0?

Actually looking at the code it's actually named NamespacedName and it's coming from apimachinery… I'd prefer using NamespacedName over ObjectKey actually :)

created this: https://github.com/java-operator-sdk/java-operator-sdk/issues/1073
I like ObjectKey more, there is a reason why it's probably also aliased in controller runtime. It describes better to identify an Object.

To be consistent actually fabri8 client should have a class with NamespacedName and we would just to extend it.

created this: #1073
I like ObjectKey more, there is a reason why it's probably also aliased in controller runtime. It describes better that is used to identify an Object IMHO.

I don't agree. It's aliased in controller runtime because it's used in a cache-like interface. That's not strictly what we're using it for and NamespacedName corresponds much better to the intent as opposed to something that's both too generic (what is the Object in ObjectKey referring to?) and too restrictive (the Key part implies it's used only in map-like structures)… I'd rather we keep ResourceID than use ObjectKey if we don't want to use NamespacedName.

I don't agree. It's aliased in controller runtime because it's used in a cache-like interface.

It's used to identify an object/resource specify the object of a given type.
(The client here does not necessary use cache.) This is really just the information needed to identify the object in case the other  required information is present.  Mainly because of this I don't like the ID in the name. This is not really an id, it does not identify the resource without additional information (api group, kind, version). So that's why I think Key is better.
Object and Resource are basically synonyms, so if we change it, would be either ResourceKey or ObjectKey. I'm fine with either. Since this is how we use it.
See the more correct definition between object and resource in kubernetes: https://stackoverflow.com/questions/52309496/difference-between-kubernetes-objects-and-resources   - the Object seems to be the correct term here to me.
Also this is quite obvious it's a Kubernetes object from the context, not a Java Object. We already use ObjectMeta, ObjectList and other names like this. Actually it's better in this sense, that in fabric8 client this is used.
So that is why I think that using Object and Key is the good combination here. But if you think ResourceKey is better, I'm fine with that too.
If there is no consensus, we can just keep it as it is.

The problem is that Resource has a specific meaning for us which doesn't really map to the Kubernetes concept of resource. So any attempt to use something from controller-runtime will just be confusing in my opinion. For example, ResourceID is used in JOSDK as key for external objects as well. Trying to use the name from controller-runtime when it's actually used for something different would be a mistake, and actually, this might also mean that porting the concept to the fabric8 client might not be such a good idea, unless we clarify the semantics (and in this case, the concept of NamespacedName is much more accurate than ObjectKey for how we use it, imo).

For example, ResourceID is used in JOSDK as key for external objects as well

Where exactly? I don't think so. It always targets a Kubernetes Object.

Trying to use the name from controller-runtime when it's actually used for something different would be a mistake

It's not really something different to my understanding, they are also just addressing an object.
But that is why I tried to describe the rational behind the name above. (So why Key instead of ID, and why Object instead of Resource)
Why I don't like NamespacedName is because it's a very generic concept. It really not intention revealing, we want to identify an object by the ResourceID/ObjectKey. (It' not fundamntelly wrong by no means, it just does not feel very cohesive, or on the same level of abstraction as the actual context)

this might also mean that porting the concept to the fabric8 client might not be such a good idea, unless we clarify the semantics

I don't have strong opinion on this. Maybe we should involve @manusa and @shawkins for that.

For example, ResourceID is used in JOSDK as key for external objects as well

Where exactly? I don't think so. It always targets a Kubernetes Object.

CachingEventSource for example. The target type is not necessarily a Kubernetes object.

Trying to use the name from controller-runtime when it's actually used for something different would be a mistake

It's not really something different to my understanding, they are also just addressing an object. But that is why I tried to describe the rational behind the name above. (So why Key instead of ID, and why Object instead of Resource)

Well, for JOSDK, a Resource is anything that needs to be managed, it's not restricted to Kubernetes objects for sure. So in JOSDK a Resource might be a Kubernetes Object but it might be a MySQL database or an external API.

Why I don't like NamespacedName is because it's a very generic concept. It really not intention revealing, we want to identify an object by the ResourceID/ObjectKey. (It' not fundamntelly wrong by no means, it just does not feel very cohesive, or on the same level of abstraction as the actual context)

I agree that NamespacedName is very generic but so is ObjectKey. ObjectKey is worse for us because we have nothing that we call Object in our code, so it doesn't refer to any concept we already use. Probably NamespacedName is not good either, so let's keep ResourceID :)

Ok, I can live with that :)


Added extra size parameter to scatter plot example

In response to this issue: https://github.com/holoviz/hvplot/issues/1117

Ready for review.

Thank you for the PR :medal_sports:


search: Only select search bar when opening or using hotkey.

We need to select the input when initiating search with the hotkey, to move keyboard focus there. But doing it all the time results in an inability to select some of the search input, reported here:
https://chat.zulip.org/#narrow/stream/9-issues/topic/search.20bar.3A.20selection.20jumping.20to.20whole.20query/near/1652562

Wait, but why is initiate_search being called when selecting text in the search input? I think the actual bug is a rogue click handler. I suspect:
    // We don't want to make this a focus handler because selecting the
    // typehead seems to trigger this (and we don't want to open search
    // when an option is selected and we're closing search).
    // Instead we explicitly initiate search on click and on specific keyboard
    // shortcuts.
    $search_query_box.on("click", initiate_search);

Is incorrectly applying even when it's already expanded, and that's the bug.

Thanks! I agree with you, fixed it by adding that check.

Merged, thanks @evykassirer!


Always getting 200 with MISSING_PARAMETER when trying to invalidate OTP

I'm using the Yubikey invalidation script for slack and my colleague pointed out that even though the API server returns 200, the key is not invalidated. It is still able to be used to authenticate into our VPN server.
I've verified for myself that this is not an issue with the script as the documentation shows the same URL and request params.
Steps to repro:

Copy the output of:

$ node
> crypto.pseudoRandomBytes(16).toString('hex')

Copy an OTP from my yubikey in a text editor
Construct a curl request using the docs above:

curl -vvv "https://api.yubico.com/wsapi/2.0/verify?id=$URL_ENCODED_API_KEY&otp=$OTP&nonce=$NONCE"

The response gives a 200 OK with the text value:
h=[HMAC]
t=2019-08-19T16:17:49Z0114
status=MISSING_PARAMETER

I can submit that request multiple times with different nonces and get the same response. Does the /verify API not actually invalidate the OTP?

Realized I input the client secret as the API ID. This API server should really have better response codes though. Issuing a 200 OK when an OTP is still live is a very dangerous practice.


Exception: Authentication key file not supported

Bug report
Your Environment

Netflix add-on version: <1.9.1>, tested also with <1.10.0> and <1.10.1>
Operating system version/name: <LibreElec , Kodi 18.8 Leia>
Device model:  <Raspberry Pi 4 model B>

Used Operating system:

[ ] Android
[ ] iOS
[ ] Linux
[ ] OSX
[ X] Raspberry-Pi
[ ] Windows

Describe the bug

Hi, I followed every step in the ReadMe to generate an Authentication Key for Netflix add-on and I get the error : Exception: Authentication key file not supported.
I generated the key using MacOS and last Python file available (1.0.3) that worked like a charm.
Thanks in advance!
Expected behavior

Actual behavior

Steps to reproduce the behavior

Possible fix

Debug log

The debug log can be found from this link:
https://we.tl/t-mxGySLEgUo
Additional context or screenshots (if appropriate)
Installation

[ ] I'm using other Netflix Repo
[ ] I'm using a different source

Other information

Screenshots

Hey,
I think it's only supported only in version 1.10.2 (- Add support to NFAuthentication for MacOS).
On my side, as a temporary workaround, I edited the source file directly on LibreElec to add "MacOS" case.
https://github.com/CastagnaIT/plugin.video.netflix/blob/master/resources/lib/common/credentials.py#L216
Regards.

Thanks @simon-vasseur , after further investigations it is indeed simply not working on MacOS...
I think @CastagnaIT last commit will fix it in a later version.
Meanwhile the easiest workaround I found (the direct file editing was also a cool option) was to install Raspberry OS on a second SD Card and run the script from there 👍 !
Thanks a lot!

Maybe it should be mentioned in the wiki, cause there is the title For Linux and MacOS.


Increment package version after release of azure-core-http

Increment package version after release of azure-core-http

We would still like to have a new entry added into CHANGELOG though.


Update backup-workload-cluster.md

Based on customer feedback, for deploying velero on AKS enabled by azure arc, fixing formatting errors

@pichuang It seems that you've pulled in some commits from another branch, so we cannot merge this. Can you re-do this PR with a clean working branch?

#sign-off


Update hy version to 0.26

Hello,
First of all thanks for making this project, it's very useful.
Recently hy v0.26 was released. I looked at hyccup's code and I think it doesn't break anything (although I'm pretty new to Hy and Lisp in general). Is it possible to bump hy to 0.26?
Thanks,
Franklin Liu

Hello, a new version of Hyccup was published with Hy updated to v0.26 :)

Thank you!


Create Snake Pattern

Create the following n*m snake pattern.

@AdarshAddee can I take this up?

Sure, you can


GitHub repository in Trusted Publisher should (probably) be case insensitive

From https://github.com/pypa/gh-action-pypi-publish/issues/217: some users configure their trusted publisher with a different casing than their actual GitHub repository (e.g. the repo might be Bar and they configure bar).
GitHub appears to treat these names as equivalent (repo names appear to be case insensitive generally), but PyPI performs a case sensitive comparison. As a result, some trusted publishing activities are erroneously rejected, as in the linked issue.
I think the fix here is just to switch to case insensitive comparisons on the repository claim, and (maybe?) on the repository_owner as well.

GitHub appears to treat these names as equivalent

Is this common across all of our publishers, or unique to GitHub?

For GitHub, we already do case-insensitive comparisons for the sub claim, which includes the repo name:
https://github.com/pypi/warehouse/blob/01be76a12ed14d0919216896505fd7912d26b3b7/warehouse/oidc/models/github.py#L99.
Extending that to the repo can be done by modifying https://github.com/pypi/warehouse/blob/01be76a12ed14d0919216896505fd7912d26b3b7/warehouse/oidc/models/github.py#L113-L119
so that repository is checked by a new _check_repository() function that behaves similarly to _check_job_workflow_ref()

and (maybe?) on the repository_owner as well.

@woodruffw When creating a GitHub trusted publisher, PyPI queries GitHub to see if the owner exists, and stores the name returned by GitHub as the "ground truth":
https://github.com/pypi/warehouse/blob/e50010ac7c54479ce504bb69cb08b04c9ebcda29/warehouse/manage/views/__init__.py#L1273
So for the repository_owner claim the normal (case-sensitive) string comparison should be fine, since the ground truth is whatever normalized name GitHub returned during that first check.

#15501 should fix this.


[util] Removing mypy tool

Currently, a few python scripts are being checked by mypy in CI.
Unless mypy becomes mandatory type checker in OpenTitan project, this confuses contributors. For instance, importing 3rd party library may or may not cause errors depending on which python file the contributor revises.

mypy is utilised extensively within the OTBN python tooling where the type checking it provides is valuable. I would strongly prefer we continue to run those checks in CI.

CC @rswarbrick
Are we ok closing this issue for now?
It seems like we'd want to keep the mypy checks in place.

Yes, let's close this.


Big Sur 11.5.2 issues

Using Big Sur 11.5.2 on my T530 (i7 3720QM, 1920x1080 screen)
When adding in the SSDT-PM generated with the script linked in readme (also tried -target 2 to generate for MacBookPro11,1 SMBIOS), the system gets stuck here booting:

When booting without SSDT-PM, the system will boot, but there is no graphics acceleration for the Intel HD 4000, and the TouchPad & Trackpoint don't work (keyboard works though).

Since Big Sur requires MacBookPro11,x to boot, ssdtPRGen fails to generate SSDT-PM, because it relies on Board-IDs containing data for Plugin-Type 0. As a workaround, you can either:

use SSDTTime to generate a SSDT-PLUG.aml or
use MacBookPro10,1 but add -no_compat_check to boot-args.

Advantages of using MacBookPro10,1 with -no_compat_check are:

You can boot Big Sur and use ssdtPRGen.
The CPU runs at lower clock speeds in idle since this SMBIOS was written for Ivy Bridge, while 11,x was written for Haswell CPUs. Therefore the CPU produces less heat and the machine runs quieter.
Another benefit of using MacBookPro10,1 is that you get the correct P-States and C-States for your CPU from ssdtPRGen.

Disadvantages of using MacBookPro10,1:

You won't be able to install System Updates because you won't be notified about them. But there's a simple workaround:

Change SystemProductName back to MacBookPro11,1
Set csr-active-config to 67080000 (for Big Sur)
Disable -no_compat_check boot-arg (add a '#' in front of it)
Reboot
Reset NVRAM
Boot macOS
Check for and install Updates
After the Updates are installed, revert to SMBIOS MacBookPro10,1
re-enable  -no_compat_check boot-arg
Reboot

I've generated the SSDT-PM for you: https://github.com/5T33Z0/Lenovo-T530-Hackinosh-OpenCore/tree/main/SSDT-PM
SSDT-PM_i73720QM_900 is for a low frequency of 900 mHz (try this first)
SSDT-PM_i73720QM_1200 is for low frequency of 1200 mHz (use this if system crashes during boot)


[Feature][core] Flink sql task support customized test case input

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description
Mock sink has been implemented in dinky-1.2.0. This issue is for the discussion of mock the source part of flink sql. Currently, dinky only support get table inputs from the real production environment, but most kind of test cases, especially edge condition cases cannot be caught in a short time. As a result, it will be more helpful for users to submit customized test cases and test their tasks as comprehensive as possible.
Use case
Provide a form for users to submit customized test cases. Flink sql tasks can read these test cases as input when some mock-source options is set.
Based on MockStatmentExplainer.class, the connector of source tables can also be changed to the customized one, which help users can submit customized test cases. Here lists my scheme:

An open-api is designed for getting the test cases submitted by users. Then all test cases will be collected to next operators
I have submitted a Draft PR, which contains most of the implement code for this scheme.
If this scheme is acceptable, I will complete the development these days and ready for review. If there are better schemes for this feature, I'm also willing to implement them and contribute!
Looking forward to your discussion and reply.
Related issues
https://github.com/DataLinkDC/dinky/issues/3893
Are you willing to submit a PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Your design is great! I agree with your idea because Alibaba Cloud Flink also has the function of uploading datasets for debugging.
This feature can effectively assist users in conducting edge condition testing.
In addition, the data for mock source and sink testing should have clear table structures.
In order to better simulate and manage this data in memory and facilitate further expansion of functionality, my idea is to build a temporary in memory database in dinky1.3.0 to manage mock data, similar to the ValuesDatabase in FlinkCDC. Users can decide whether their data should be persisted to the file system or MySQL data tables through configuration options. This ValuesDatabase can effectively restore SchangeChangeEvents and perform DataUpsert.
I believe that building mock raw data through Dinky's interface has more advantages than manually assembling CSV data.
What do you think?

Your design is great! I agree with your idea because Alibaba Cloud Flink also has the function of uploading datasets for debugging. This feature can effectively assist users in conducting edge condition testing. In addition, the data for mock source and sink testing should have clear table structures. In order to better simulate and manage this data in memory and facilitate further expansion of functionality, my idea is to build a temporary in memory database in dinky1.3.0 to manage mock data, similar to the ValuesDatabase in FlinkCDC. Users can decide whether their data should be persisted to the file system or MySQL data tables through configuration options. This ValuesDatabase can effectively restore SchangeChangeEvents and perform DataUpsert. I believe that building mock raw data through Dinky's interface has more advantages than manually assembling CSV data. What do you think?

@aiwenmo Thank for your reply!
I agree with your idea that it's more convenience for users to build mock data by interfaces while not CSV data. As a result, in current implement I submitted in Draft PR, a editable table modal of front part is generated by the table structures gotten from backend, users can submit their data set by this editable table, which makes dataset edit more clear.
What I ignored is that during FlinkCDC task debug, change events is important. And that is hard to simulate in current scheme I designed, I think your idea about build a ValuesDataBase is excellent, a temporary database can not only simulate data change event for cdc task but also generate data for batch or stream flink sql task.
As all discussion mentioned above, I will redesign the scheme details based on ValuesDataBase and implement in the draft PR and ready for review when all development and tests ready!

@MactavishCui Thank you very much for your contribution!
In order to facilitate contributions to the Apache incubator, Dinky's project positioning will be corrected to a debugging and optimization framework for stream computing, exploring the key components currently missing in the stream computing ecosystem. This positioning is more in line with Dinky's core capabilities that have been completed and planned for the future, and will give Dinky greater core advantages.
Unfortunately, currently most users' understanding of Dinky is only limited to the Flink platform, which is not in line with the current situation and goals of the project. Dinky laid the foundation for exploring innovation in the field of stream computing in its initial design, with the original intention of better integrating into enterprise big data architecture as a middleware to supplement the capabilities of stream computing. Now we have an increasingly active community and a growing number of users, and we believe that in the future, it can become a core component and essential textbook content in big data architecture.
Finally, your contribution to Mock testing has greatly advanced the debugging direction of stream computing. In the future, my personal contribution focus will also be on this direction, such as implementing mock testing of application patterns through websock and other methods, capturing and storing Schema Change Events in FlinkCDC for auditing and tracing, supporting Mock testing of FlinkCDCPipeline, and tracing the performance of stream data changes through row level blood relations. Welcome to continue communication~

@aiwenmo Hi~ I have studied the code of ValuesDatabase from flink-cdc and made some redesigns for this feature. After several attempts, here are still some points puzzling me.
The first problem I want to figure out is that what is the more suitable position for our MockDatabase? A util that can apply change events? Or more like a cache database, here lists my attempt schemes:

As is shown above, change events is transformed between front and backend part, and flink tasks can generate results or simulate change events by these information.
The disadvantage of this scheme is that all info should be persisted and mockdatabase object should be created for every requests.

The second scheme is based on a singleton database instance exists in memory, the database has a map whose key is task id and value is tables and its records. Tasks can get there own mock-database by MockDataBase.getInstance(taskId).  Will it  be okay to manage entries like LRU and the capacity of mock database can be set by configuration center?
Because I'm not much familiar with cdc tasks and dinky-cdc model now(Surely I'll learn them these days, lol),  the second point I'm worried about is that will these schemes good fo rfurther expansion like the source mock for cdc tasks?
Which scheme do you think is better? Or is there some better schemes and suggestions?
Looking forward to your reply.

Due to attending FFA 2024 in the past two days, there is no equipment available for design and response. I apologize for this.
In order to better design this content, I suggest starting with a macro level design. What we expect is that Dinky can provide a complete Mock Test, including Source and Sink, Data and Event. At this point, I call it MockDataBase, which simulates all behaviors in stream processing. MockDataBase includes two write behaviors: Append (Changelog) and Upsert. MockDataBase is an abstract design that cannot be instantiated. It is instantiated by defining persistent implementation classes, such as memory schema or MySQL schema.

This may not be easy to understand, but I can give an example to express this idea very well. We can deploy a Kafka cluster specifically for mock sources and sinks, but we need to modify the connector configuration to match that of the Kafka cluster. So at this point, Dinky can proxy similar to its implementation of the RS file system protocol, with only one MockDataBase, and its specific implementation is configured through the configuration center.
This is a preliminary idea, I will continue to update the details. What is your opinion?
Look forward to your reply.

Due to attending FFA 2024 in the past two days, there is no equipment available for design and response. I apologize for this.
Your design ideas are great. Here are some new ideas I have based on your design.
In order to better design this content, I suggest starting with a macro level design. What we expect is that Dinky can provide a complete Mock Test, including Source and Sink, Data and Event. At this point, I call it MockDataBase, which simulates all behaviors in stream processing. MockDataBase includes two write behaviors: Append (Changelog) and Upsert. MockDataBase is an abstract design that cannot be instantiated. It is instantiated by defining persistent implementation classes, such as memory schema or MySQL schema.

This may not be easy to understand, but I can give an example to express this idea very well. We can deploy a Kafka cluster specifically for mock sources and sinks, but we need to modify the connector configuration to match that of the Kafka cluster. So at this point, Dinky can proxy similar to its implementation of the RS file system protocol, with only one MockDataBase, and its specific implementation is configured through the configuration center.
This is a preliminary idea, I will continue to update the details. What is your opinion?
Look forward to your reply.

@aiwenmo Thanks very much for your reply! Your response really helps me out of my confusions! Now I can understand that a MockDataBase design means dinky provides a 2x2 abilities (schemes & data x query & write, like what ValuesDataBase of flink-cdc does) by interfaces. And the specific implements are based on user configurations. As a result of this scheme, dinky can build a IO control bridge between users and tasks. From the perspective of users, task input can be customized and output can be previewed, while from the perspective of tasks, test cases can be read and execute result can be insert. I'm very glad to update my codes based this new scheme!
Based on this design, I think not only for this issue, but there are many optimizations can be made, like refactoring mock data sink, making mock database implement extension, supporting mock source for cdcsource tasks and etc.
Following the principle of KISS (Keep it simple and stupid), may I submit necessary codes just related to this issue in linked PR and make these optimizations by other issues and PRs?
Looking forward to your reply!

Of course, Looking forward to your contribution.
We expect to complete this core feature in Dinky-1.3.0. In order to facilitate code review, this feature needs to be broken down into multiple small pull requests.
For example:

Design the interface for MockDataBase and implement memory mode (instead of ResultPool).
Modify MockSink and Select to be implemented based on MockDataBase.
Add MockSource based on MockDataBase implementation.
MockSink and Select implement socket mode to support application mode.
Expand the implementation of MockDataBase, such as Kafka or Fluss.

This does not represent the final result, we can continue to update the content of MockDataBase in this issue. Next, I think we can discuss the design of MockDataBase.
Looking forward to your reply.

@MactavishCui Your design is excellent!
Today I have a bold idea: It would be more appropriate to define MockDataBase as Sandbox.
Sandbox is a hybrid data storage system for simulating data.
It has the following capabilities:

Support and simulate data generation. Data formats include snapshot data, ChangeLog, stream data, vector data, etc.
Support and simulate data writing. Writing methods include append, update, zipper, logical deletion, etc.
Support unified metadata management.
Support high-performance comparison of two data sets.
Support real-time capture of end-to-end data latency.
Support the mutual conversion of data in different database formats and forms.
Support data rollback at a specified point in time.
Support OLAP.

We can implement an MVP version (Minimum Viable Product version) first, including the following capabilities:

Support metadata management.
Support high-performance writing of simulated data (separation of structure and data), and the writing methods include append, update, replacing ResultPool.
Support single-table query.
Support simulation data generation, including snapshot data and ChangeLog.
Support writing in socket mode.

In addition, for the questions you raised, I have the following responses:

The method parameters of Sandbox should be decoupled from the Task of Dinky, and replace taskId with the string boxName (even though they are the same value).
Map<String, String> data should include the original data types, not just strings.
Do not reference the classes of FlinkCDC, but can be implemented by imitation.

This innovative new feature is really attractive. Details will be continuously updated.
Looking forward to your reply.

@MactavishCui Your design is excellent!
Today I have a bold idea: It would be more appropriate to define MockDataBase as Sandbox.
Sandbox is a hybrid data storage system for simulating data.
It has the following capabilities:

Support and simulate data generation. Data formats include snapshot data, ChangeLog, stream data, vector data, etc.
Support and simulate data writing. Writing methods include append, update, zipper, logical deletion, etc.
Support unified metadata management.
Support high-performance comparison of two data sets.
Support real-time capture of end-to-end data latency.
Support the mutual conversion of data in different database formats and forms.
Support data rollback at a specified point in time.
Support OLAP.

We can implement an MVP version (Minimum Viable Product version) first, including the following capabilities:

Support metadata management.
Support high-performance writing of simulated data (separation of structure and data), and the writing methods include append, update, replacing ResultPool.
Support single-table query.
Support simulation data generation, including snapshot data and ChangeLog.
Support writing in socket mode.

In addition, for the questions you raised, I have the following responses:

The method parameters of Sandbox should be decoupled from the Task of Dinky, and replace taskId with the string boxName (even though they are the same value).
Map<String, String> data should include the original data types, not just strings.
Do not reference the classes of FlinkCDC, but can be implemented by imitation.

This innovative new feature is really attractive. Details will be continuously updated. Looking forward to your reply.

@aiwenmo  Thank you very much for your suggestions.
I agree with your idea about sandbox! Because different type data have different features, like the event or handle time and the message order of stream data, different features of different data will cause different task execute results, the simulation of these feature can be greatly simulated by a sandbox design! With the help of OLAP functions, we can build 2 tasks in different versions and compare the result differ of them, which really help users to analyze their tasks better!
About our MVP sandbox abilities, I made following detail designs and here are something I want to discuss with you:

Support metadata management. A abstract method getMetadataManager need to be implemented by inheritor classes, based on this method a instance implement interface MetadataManager can be got for metadata management. The abilities of MetadataManager including the management of sandbox metadata and table metadata. Meta data of sandbox including sandBoxName, data type(A enum including stream data for flink sql task in stream mode, snapshot data for flink sql task in batch mode, and cdc change log data for cdc task, different types of data also have different strategy of primary key generations, for example stream data can use message order as primary key and cdc change log data can generate keys by key columns) and supported data change events. Meta data of tables including table info like ids, namespaces, schemes and whether it is for mock-sink or mock-source.
Support high-performance writing of simulated data (separation of structure and data), and the writing methods include append, update, replacing ResultPool. Following existed design, a default method applyChangeEvents and several protected abstract change methods.
Support single-table query. Get results by tableId. (Maybe we could design a DSL(Domain Specific Languages) or use other methods to support complex query further to support OLAP).
Support simulation data generation, including snapshot data and ChangeLog.  Generate data by snapshot ability can be offered by a default method that generate several data insert events by snapshot data.
Support writing in socket mode. This is my confusion point, in my understanding mockDataSandBox is running in dinky-admin application. Mock data management abilities is provided by web and open-apis, mock insert of application jobs can insert results by these apis, a web socket can build between dinky-admin application and flink jobs, insert operation can be forwarded by application, so will it be necessary to build web socket between sand box and Flink job directly? That is your opinion.

All mentioned above is my design and confusion, looking forward to your reply!

Thank you for your reply! The following is my response:

Dinky Entity Relations: SQLTask -1:n-> History -1:1-> JobInstance

The definition of the sandbox is closer to a collection of catalogs and does not include data types. The sandbox metadata includes the sandbox name and table metadata, and the unique table is determined by boxName.catalogName.databaseName.tableName. Data types (stream data, snapshot data, Event, Zipper data) should be bound to the table object, which provides the basis for future hybrid analysis. (For single job debugging, the sandbox is like a message queue (there is no relationship between two topics). By default, a dinky task will create a new exclusive sandbox instance through the jobInstanceId, and the sandbox instances are completely isolated; for the debugging of multiple jobs in the data warehouse, the sandbox is more like a database, and more attention is paid to the result after the influence of multiple jobs, that is, the dinky task may write data to the existing shared sandbox instance. With the evolution of functions, in the future, Sandbox may manage all users' message queues and data lake warehouses in an abstract way and provide read and write capabilities in a proxy way, and even implement hybird reading and hybird writing. At this time, business users only know the existence of one Sandbox, that is, HBox (Hybird Box).)
The high-performance structure of data is mainly the separation of schema and data. The data type of the data part can be compressed into binary. This design inspiration comes from the high-performance data structure of the FlinkCDC YAML job. So the length of N data list captured by mock data can be optimized to one schema and N-1 data. For data tables of the Event type (including SchemaChangeEvent and DataChangeEvent), its structure can refer to the Event in FlinkCDC.
Single-table query is related to the implementation method of storage. Query in memory, hdfs, kafka, etc. can be achieved through Apache Calcite, and others can be converted into corresponding query statements for querying.
Data generation is an auxiliary function. When used as a data source, its data sources may be script generation, csv import, page editing, and current storage tables. In addition, regarding the table in the first point of metadata, I think it should not be distinguished whether it is a simulation source or a target, because a table can be used as both a source and a target, such as building a minute-level hierarchical data warehouse in Apache Paimon. It is more appropriate to change the description here to support being a data source for stream computing.
When the job runs in yarn/k8s applictaion mode, mock sink supports writing the captured data to the sandbox in real time through the web socket method.

Looking forward to your reply.

@MactavishCui Thank you for your reply! The following is my response:

Dinky Entity Relations: SQLTask -1:n-> History -1:1-> JobInstance

The definition of the sandbox is closer to a collection of catalogs and does not include data types. The sandbox metadata includes the sandbox name and table metadata, and the unique table is determined by boxName.catalogName.databaseName.tableName. Data types (stream data, snapshot data, Event, Zipper data) should be bound to the table object, which provides the basis for future hybrid analysis. (For single job debugging, the sandbox is like a message queue (there is no relationship between two topics). By default, a dinky task will create a new exclusive sandbox instance through the jobInstanceId, and the sandbox instances are completely isolated; for the debugging of multiple jobs in the data warehouse, the sandbox is more like a database, and more attention is paid to the result after the influence of multiple jobs, that is, the dinky task may write data to the existing shared sandbox instance. With the evolution of functions, in the future, Sandbox may manage all users' message queues and data lake warehouses in an abstract way and provide read and write capabilities in a proxy way, and even implement hybird reading and hybird writing. At this time, business users only know the existence of one Sandbox, that is, HBox (Hybird Box).)
The high-performance structure of data is mainly the separation of schema and data. The data type of the data part can be compressed into binary. This design inspiration comes from the high-performance data structure of the FlinkCDC YAML job. So the length of N data list captured by mock data can be optimized to one schema and N-1 data. For data tables of the Event type (including SchemaChangeEvent and DataChangeEvent), its structure can refer to the Event in FlinkCDC.
Single-table query is related to the implementation method of storage. Query in memory, hdfs, kafka, etc. can be achieved through Apache Calcite, and others can be converted into corresponding query statements for querying.
Data generation is an auxiliary function. When used as a data source, its data sources may be script generation, csv import, page editing, and current storage tables. In addition, regarding the table in the first point of metadata, I think it should not be distinguished whether it is a simulation source or a target, because a table can be used as both a source and a target, such as building a minute-level hierarchical data warehouse in Apache Paimon. It is more appropriate to change the description here to support being a data source for stream computing.
When the job runs in yarn/k8s applictaion mode, mock sink supports writing the captured data to the sandbox in real time through the web socket method.

Looking forward to your reply.

@aiwenmo
Thanks very much for your reply!
I'm so excited that I can understand the design of sand box and how it works in dinky application better! Previously I only focus on the debug process in only one Tasks, you provide a pretty much better perspective!  The optimization designs of it really attracts me too!
In these one or two days I'll organize our discussions conclusions and upload a new design scheme with more details in pic based on the latest conclusion. Please review at that time!
THANKS VERY MUCH again!

@MactavishCui Thank you for your participation! Regarding this, I am also very excited. This is the spark generated by the collision of two ideas, and both are indispensable.
Continuous communication provides more details for the concretization of the sandbox. The sandbox brings innovative and unique capabilities to Dinky, and I believe it will attract more people.
Currently, I am preparing for the official release of Dinky-1.2.0. After the release is completed, I will devote most of my energy to the construction of the sandbox.
I am looking forward to your new design scheme with more details, and I am willing to review it. thx

Hi! @aiwenmo I have made some attempts there days about sandbox and how to use it. According to our discussion last week, In the MVP version, sandbox has the abities including metadata, schema and records management. Based on this new design, l think there are following points that can be worked on:

Refactor mock sink. currently mock sink only supports Local, Standalone, Yarn ession, Kubernetes Session, but now with the help of sand box websocket sink, fink jobs in all modes can be supported.
Implement for the new feature of this issue: support user customized test case input.

And here lists my new designs of the 2nd point.
Firstly, a prototype design was carried out for the front-end page.

I designed a indepence mock configration page. lt has 3 options: mock sink option, customized mock source  option and sandbox selection.

mock sink option: Like what it does currently, mock sink operator will be used instead of the real operator when this set enabled.
customized source option: When this setting is enabled, jobs will get data from sandbox, while jobs will get data from real source data when it is disabled.
Sandbox selection: Users can select a existed sandbox by sandBoxName. input data used from this sand box can be previewed (Filter data that only used in current task and show them).

The left containers has a sand box management page, this page is farmiliar with the datasource management page, but has more functions. In this page, users can edit sanbox table schemas and records. Schemas can be initialized by users input, init from Flink Sql task statements or from datasource connection. Metadata records can be edited by web or uploaded by files.

About the backend implement
SandBoxController and SandBoxService is designed for sandbox management, whose MVP version has following interfaces:

Init schema by taskld and sandboxName
insert record into sandBoxTableld(sandboxname.namespace.schema.tableName)
update record from sandBoxTableld
delete record from sandBoxTableld
get all sand boxes
get tables by sandboxName
get schema and all records by sandboxName
An open api of ApiController get records by sandBoxTableld

Besides, task debuging process is designed as follows:

The connector of sourcetable will be convert to sanbox-mock-connector, the connector params including:

sandBoxTableld for this table.
data type: snapshot/change log
host and port information to build websoct connection and visit openapi
Some alternative options like the delay of stream data

All mentioned above is my new design of this feature. Is this scheme acceptable, and may l submit a PR for this feature based on sandbox? I find that you will work on the sandbox construction, so if the answer is yes, before the construction of sandbox finished, I'll mock the usage of sandboxes temporaly, focus on the front end codes and backend codes for sand box aplication of this feature and make sychronous changes when sandbox is ready. I also find that there are some code opimization after my last new feature PR, so l'll study these opimization codes and pay more attention to the scheme and code style.
If l can submit a PR for this new feature, l'll be really grateful if you give me a notification when sandbox is ready. Additionally, l'll also continue working on other issues.
Looking forward to your reply.

Thank you very much for your attempt and design. I agree with your scheme. Besides, there is a small issue below that needs correction:
host and port information to build websocket connection and visit openapi.
The connector of source table will be converted to sandbox-mock-connector, not sanbox-mock-connector.

Finally, I will give you a notice when the sandbox is ready. thx


AttributeError: module 'nano' has no attribute 'Exp'

I retrain yolo-nano with this script python3 tools/train.py -n yolox-n ano -d 1 -b 64 --fp16 -o. However it raised this error
AttributeError: module 'nano' has no attribute 'Exp' ImportError:exps/default/nano.py doesn't contains class named 'Exp'
Could you help me to fix?
Thanks very much!

hi, we test our code and it runs well. So I suggest you to pull the updates and reinstall yolox with:
pip install -e -v .

@ruinmessi , Thank for reply soon,
I found that I have a package name nano too so I modify the file name and it run well.


MomentsAggregator should have a Numeric version

MomentsAggregator requires a Double input. We could composePrepare to get a double from a Numeric[N] similar to what was done with AverageValue.

Actually this already exists: https://github.com/twitter/algebird/blob/0.9.0/algebird-core/src/main/scala/com/twitter/algebird/MomentsGroup.scala


Make Page Turn and Zoom Bottoms much more visible on smaller screens

Overall page needs a rework

Relocate annotation elements to not be so far down the page, should be front and centre for using. Possibly relocate the login and element load toolbox to smaller size at top or bottom of screen since that is not used as much during the annotations.
Make Page Turn and Zoom Bottoms much more visible on smaller screens. Makes sense if they are on massive monitor, elements get jumbled on smaller screens. Possible to add visual cue to show where these tools are on the page

We had a discussion about this in more detail. We think that the following steps would be useful:
Use bootstrap to create a top nav bar with the logo and login/logout button
When a user logs in, show only input boxes to enter a URL, choose an example, or search the CE
When the user selects an item, hide the input boxes. Include a [New] button in the menu bar that allows the user to start again. Use react router for this.
When a score is loaded, give the user 2 options: Annotate, or use F-tempo. When an option is selected show the tools only for the specific operation that the user selected.
Continue with the layout of the score on the right and tools on the left.
Configure the score to stay in place even if the tools scroll

Most of this workflow was done in<PHONE_NUMBER>c3f953e87d181ac2cb3317a3aebc38 and a9da832232cea31e05505a101279bdd46949b63f
Still remaining is to add the score and main annotation interface to a react grid in order to make it easier to deal scrolling, and the selection of annotate/use f-tempo