#51 - Error message improved for unwrapped Collections

Hi,
First time commit: resolving issue #51 by changing the displayed error message to "Field is not a wrapped collection type."
Thanks.

Hey @kwahsog thanks so much for taking the time to grab an issue.
I feel like this error message could be improved even more. I'm thinking along the lines of a message that's more specific. Meaning, a field must be wrapped and copied, so there are several combinations of errors that can be made (no copying or wrapping; no wrapping; no copying) and the message could indicate which ones are the problem.
If I write up an issue describing the behaviour I think would be useful, would you be interested in implementing it?

Hi,
Thanks for the merge. I am keen to contribute further, so it'd be great if you could write up the issue and I'll get working on it.
On a side note: I was trying to fix [https://github.com/MutabilityDetector/MutabilityDetector/issues/43](issue 43), but couldn't work out where the first message was being generated to edit it.


turbine 1.0.0 does not work with hystrix metrics

see https://github.com/Netflix/Hystrix/issues/703
using turbine 0.5 works perfectly

see also https://github.com/Netflix/Hystrix/issues/792


Fix components to respond to RTL

What

Fix our ACS components to respond to RTL
Components addressed:

FluentThemeProvider
MessageThread
ControlBar
TypingIndicator always ltr because strings in strings files should already be in the right direction

Added rtl toolbar button in storybook.
Video of rtl switching in storybook:
https://user-images.githubusercontent.com/79475487/124593650-bc4f7180-de13-11eb-926e-3cbd1be4c9bd.mp4
Will document how to set RTL as a separate ticket.
Why

https://skype.visualstudio.com/SPOOL/_workitems/edit/2506615
How Tested

Manual testing
Process & policy checklist

[ ] I have updated the project documentation to reflect my changes if necessary.
[ ] I have read the CONTRIBUTING documentation.

Is this a breaking change?

[ ] This change causes current functionality to break.

Note: found some small bugs (not ours or not blocking for this PR though):

send button icon should be mirrored:

Fluent is supposed to automatically flip this so it could be a bug in the react-icons package (https://developer.microsoft.com/en-us/fluentui#/styles/web/localization)
Video tile name when too long cuts off the start of the name (in LTR it cuts off the end of the name)

This should probably be fixed in a totally seperate item where we use ellipses or something for a long name
Should punctuation be moved to the start of the text in messages?

Update: looks to be a bug in the northstar package, they're very inconsistent on when it does and doesn't (https://fluentsite.z22.web.core.windows.net/0.56.0/components/chat/definition):


[Bug]: CLI Allows Invalid Theme App Extensions to be Deployed

Please confirm that you have:

[x] Searched existing issues to see if your issue is a duplicate. (If you’ve found a duplicate issue, feel free to add additional information in a comment on it.)
[x] Reproduced the issue in the latest CLI version.

In which of these areas are you experiencing a problem?
App
Expected behavior
The Shopify CLI should have exited with an error when the {% schema %} contained invalid JSON. Or, on the Shopify backend, extensions containing invalid schemas need to be rejected.
Otherwise, a simple mistake can have a catastrophic impact on merchants.
Actual behavior
The deploy completed successfully, despite syntax errors. This led to widespread breakages of several production stores, through no action of their own. In fact, the stores who reported this issue had not even enabled this app embed on their store, but their storefront still broke.
Verbose output
The log output was too long to fit in this field; I've included it in this Gist: https://gist.github.com/thosakwe/065da5d93261ca8c751c94a72c3b1350
Reproduction steps
I discovered a major issue in the theme app extension framework:
Just one trailing comma can bring down an entire site, both the Online Store and the Theme Editor. Here’s an example schema:
{% schema %}
{
  "name": "Add To Cart Embed",
  "target": "head",
  "javascript": "regios_add_to_cart_embed.js",
  "settings": [
    {
      "type": "paragraph",
      "content": "For information about how to use this app embed, read [this article](https://regiostech.crisp.help/en/article/auto-add-to-cartfree-gift-with-purchase-1yd9hs0/)."
    },
    {
      "type": "number",
      "default": 1000,
      "id": "interval",
      "label": "Interval (ms)",
      "info": "How long to wait before checking if the cart is eligible for an auto add to cart offer. Default is 1 second."
    },
    {
      "type": "number",
      "id": "max_retries",
      "label": "Max Retries",
      "info": "How many times to try to add a variant to the cart before giving up to prevent breaking your cart. Default is 3.",
    }
  ]
}
{% endschema %}

Notice the trailing comma after “max_retries” info. Although this syntax error was my mistake, the Shopify CLI should ideally block deploying extensions with such errors. In this case, the extension deployed successfully, and every site with my app installed experienced a 500 error on the storefront and Theme Editor.
Typically, if there is a semantic error, such as an unrecognized field name within a setting, the CLI will catch it, but it seems like invalid JSON slips under the radar.
The lack of error messaging or logging made this hard to troubleshoot, and we only caught it after a few hours, though several merchants uninstalled. :sweat_smile:
Steps to reproduce:

Create an app embed theme app extension.
Include invalid JSON between the {% schema %} tags (for example, include a trailing comma).
Run npm run deploy
The extension will deploy successfully.

Operating System
Mac OS Monterey 12.2.1
Shopify CLI version (check your project's package.json if you're not sure)
3.69.4
Shell
zsh
Node version (run node -v if you're not sure)
v22.6.0
What language and version are you using in your application?
TypeScript 5.5.4, Node 22.5.1, Ruby 3.3.4 (not used in the app, but that's my installed Ruby version)

+1

We accept JSONC now, so this is technically not a syntax error. Real syntax errors do raise problems.

Even if you support JSONC for theme app extensions, Theme Editor and Storefront don't support it.


[BUG]

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior
The example in apache-dubbo-test-api.http is error.
shengyu getway proxy batchSave
PUT http://localhost:9195/dubbo/batchSave
Accept: application/json
Content-Type: application/json
{
"sofaTestList": [
{
"id": "123",
"name": "test"
},
{
"id": "123",
"name": "test"
},
{
"id": "123",
"name": "test"
}
]
}
Error, NullPointException.
Expected Behavior
Success.
Steps To Reproduce
No response
Environment
ShenYu version(s): current version
Need to check v2.4.1 and v2.4.0
Need to check all dubbo examples project.

Debug logs
No response
Anything else?
No response

please assign this to me.


Update ReadMe with link to app and improved description

Improve the README.md if you have received feedback about that. Inlcude in yout submission the link to the feedback and the link to the commit/s with the changes.
Please modify your GitHub repo description in the top right corner where it says “About” to include
A short description of your app (might already be there)
The link to your final deployed dashboard (it is often useful to still keep this in the README as well)
A few keywords describing which plots, widgets, and interactions you have used in your dashboards, like it is done in this demo app.

On my Side:

Comment: “The male/female bar chart shows interesting insights. Maybe you can make it possible to select male or female so that if the user wants to know the value for female, they wouldn’t have to subtract the male count from the total.”

Added the Radio Button to select by Gender to address following comment:

Documentation Improvements and Adjustments (https://github.ubc.ca/fdandrea/532-peer-review/issues/19#issuecomment-23057):
* Comment: “The dashboard is no longer in planned stage, and it is already been implemented. Perhaps it would be a better idea to remove the dashboard prototype images from the README.md, or move to a separate file/bottom of the file.”
    * Created an archived folder to place all the documentation/images from the planning stage

* Comment: ”Perhaps we could consider reordering the headings, starting with the important bit for the user (e.g., "Usage and Deployment Link"), followed by "What can you do with suicide_indicator_r dashboard?", then "Description of the Dashboard" and "Motivation", etc.”
    * Re - organized the structure of the README file from most important to least important for the user
* Comment: “A short paragraph in each of tab (perhaps in the control panel) describing what the graphs are maybe useful.”
    * Added a short sentences with descriptions into each tab.


Stop SDK from saving Appwrapper yaml to local dir

Save the created appwrapper to
~/.codeflare/appwrapper/<APPWRAPPER.YAML>
instead of local dir

@dimakis what is the alternative here? saving them to a /clusters/ dir or something?

@MichaelClifford hiding the appwrapper by default, but allowing for it to be printed if requested.

As the generated .yaml is intended for immediate processing within the same notebook environment, saving directly to the remote file system may remain more efficient. This avoids the download-upload cycle and reduces the steps involved in the workflow.
After a discussion we have decided to still achieve an abstraction, without resorting to local storage, we could move the appwrappers to a .codeflare/appwrapper/<aw.yaml> directory within the current notebook. Removing the direct view to the appwrapper but keeping a similar structure.


[Bug Report]: The .hidden class does not cascade throughout all devices

Tailwindcss version: ^1.4.6
Description:
The .hidden class when added only works in small devices, in order to it work properly the .md:hiden must be added so the style cascade throughout the devices above.
Acording to the documentation in the responsive design section:

By default, Tailwind uses a mobile first breakpoint system, similar to what you might be used to in Bootstrap or Foundation.
What this means is that unprefixed utilities (like uppercase) take effect on all screen sizes, while prefixed utilities (like md:uppercase) only take effect at the specified breakpoint and above.

Steps to reproduce the bug:
Add the .hidden class in your HTML code and resize the window.
PS:. As i could not find an issue template, i did my best to describe the problem.

Can you share the exact snippet of HTML where you are seeing this problem? This is probably a misunderstanding.
As you can see here, it does hide things at all sizes:
https://tailwind.run/BcXdvF
It will only stop hiding the element if you've added another class that unhides it, like sm:block (both hidden and block target the CSS display property).
Tailwind's responsive classes are mobile-first, so hidden applies to screens 0px and up, sm:hidden applies to screens 640px and up, md:hidden applied to screens 768px and up, etc.

Here's the code:
https://tailwind.run/BcXdvF/1
I even posted it in stack overflow when i was thinking the the problem was with the browser. You can check it here (Browser not updating after remove or add a class in HTML element)[https://stackoverflow.com/questions/62334196/browser-not-updating-after-remove-or-add-a-class-in-html-element]
Probably i'm missing something...

Yep so you have md:grid there which is what is overriding hidden. grid and hidden both set the display property.
Your code says "hide this element until the md breakpoint, then display it as a grid".
The CSS is:
.hidden {
  display: none;
}

@media (min-width: 768px) {
  .md\:grid {
    display: grid;
  }
}

What are you actually trying to accomplish? Do you want it to be visible as a grid or do you want it to be hidden? Are you trying to dynamically toggle whether it is hidden or visible using JS?


add docker-compose file

Run command:
Check your device mount point before you run the container!
docker-compose up -d

Thank you. Not using compose myself. I have removed the "root" sub folder as it seems unnecessary.
Am I right to assume, the paths are relative to where you run the compose script?

Thank you for reviewing @rix1337 !

Thank you. Not using compose myself. I have removed the "root" sub folder as it seems unnecessary.
Am I right to assume, the paths are relative to where you run the compose script?

Sure

Thanks for the input. Am super happy someone else found this so useful, they contributed 🎉


[Improve] file upload check file type improvement

[Improve] file upload check file type improvement
Contribution Checklist
in FileHeaderCheckInterceptor judging the uploaded file type, all bytes of the uploaded file will be load to memory, which will causes OOM.
What changes were proposed in this pull request
Issue Number: close #2244

Brief change log

Verifying this change

This change is a trivial rework / code cleanup without any test coverage.
(or)
This change is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:

Does this pull request potentially affect one of the following parts

Dependencies (does it add or upgrade a dependency): no

cc @xinzhuxiansheng @1996fanrui PTAL. thanks


Update MapBox url according to MapBox documentation

https://www.mapbox.com/help/mapbox-with-openlayers/

@milafrerichs could you maybe take a look at this PR as you were the one that contributed the mapbox integration if I remember correctly? That'd be cool.
@jeffersonrpn thx for taking the time to submit this PR. Could you maybe just update it to fix the build issue and also to adjust your commit message to our guidelines.


Make project public

[x] https://github.com/njbbaer/simulacra/issues/29
[x] https://github.com/njbbaer/simulacra/issues/47
[x] https://github.com/njbbaer/simulacra/issues/51

:tada:


Updates flake.lock

Updates flake.lock, adding new sources to none-ls, and changing a module option in oil.nvim which has been depreciated in nixpkgs.

Obsoleted by https://github.com/nix-community/nixvim/pull/1275
Thank you nonetheless for this initiative !


task for brute force ftp attack

I would add the usage example for this attack to the other repository.

Please use Success and Failure return types, especially for exception handling


use custom emoji instead of link to image

If you set up a custom emoji in Slack for :hal9000: using the image linked to in the original plugin http://s29.postimg.org/en1j0grer/200px_HAL9000_svg.png  you'll have a nicer time.

If this isn't what software is about, I don't know what is.

lol


Check more packages with gofmt and go vet

Commits in this PR do:

Run go vet against packages mentioned in FORMATTABLE instead of TESTABLE_AND_FORMATTABLE and extend the list of packages in FORMATTABLE.
Fix issues in inspect and networking.
Replace the hardcoded list of packages with inexplicable magic.
Remove dead unit test file from networking - outdated, does not build, never executed.

cc @alban @eyakubovich for go vet fixes.

It looks good to me but I would like @eyakubovich's opinion about networking/net/net_test.go.

Rebased against current master - go vet caught yet another error in newly added code.

Travis build timing out?

I don't know why - backtrace is useless. Maybe travis was overloaded.

Kicked it again..
On Thu, Apr 23, 2015 at 12:34 AM, Krzesimir Nowak<EMAIL_ADDRESS>wrote:

I don't know why - backtrace is useless. Maybe travis was overloaded.
—
Reply to this email directly or view it on GitHub
https://github.com/coreos/rkt/pull/799#issuecomment-95477165.

Clearing assignment - I don't touch any networking anymore.

LGTM


Facing issue while importing this library.

I am using following dependencies in my project:
compile 'com.google.android.gms:play-services:+'
compile fileTree(dir: 'libs', include: ['*.jar'])
compile 'com.spotify.sdk:spotifysdk:1.0.0-beta8@aar'
// NOTE: Version must match!
compile 'com.android.support:appcompat-v7:21.0.3'
compile project(':spotify-web-api-android')
compile 'com.squareup.retrofit:retrofit:1.9.0'
compile 'com.squareup.okhttp:okhttp:2.2.0'
compile project(':facebookSDK')

FAILURE: Build failed with an exception.

What went wrong:
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxx
Error Code:
2
Output:
UNEXPECTED TOP-LEVEL EXCEPTION:
com.android.dex.DexException: Multiple dex files define Lcom/google/gson/JsonDeserializer;
at com.android.dx.merge.DexMerger.readSortableTypes(DexMerger.java:596)
at com.android.dx.merge.DexMerger.getSortedTypes(DexMerger.java:554)
at com.android.dx.merge.DexMerger.mergeClassDefs(DexMerger.java:535)
at com.android.dx.merge.DexMerger.mergeDexes(DexMerger.java:171)
at com.android.dx.merge.DexMerger.merge(DexMerger.java:189)
at com.android.dx.command.dexer.Main.mergeLibraryDexBuffers(Main.java:454)
at com.android.dx.command.dexer.Main.runMonoDex(Main.java:303)
at com.android.dx.command.dexer.Main.run(Main.java:246)
at com.android.dx.command.dexer.Main.main(Main.java:215)
at com.android.dx.command.Main.main(Main.java:106)

Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED
Total time: 17.223 secs
my skype id is : deepak.kumar3038

@deepakvvdn: This is isn't specific for this project, but is rather an error that can happen to any project using Gradle. There are plenty of suggestions to solve this on Stack Overflow, e.g.

http://stackoverflow.com/questions/7870265/unable-to-execute-dex-multiple-dex-files-define-lcom-myapp-rarray
http://stackoverflow.com/questions/21735493/com-android-dex-dexexception-multiple-dex-files-define
-http://stackoverflow.com/questions/22468700/unexpected-top-level-exception-com-android-dex-dexexception-multiple-dex-files

Please double-check that you're not importing the wrapper library multiple times. If you keep having issues and can't solve this using the solutions given above, I suggest you create a new question at Stack Overflow for it.

yaa thnks a lot for your help !!! i have resolved the issue

when I do ./gradlew on windows no build directory or .arr file is created throughout my windows files.

any ideas on what to do?


HP Notice:  fwrite(): send of 22 bytes failed with errno=32 Broken pipe in phpMQTT.php on line 214

I want to keep php mqtt script running on background but it is throwing this error

I am restarting this project so having a clear out of old issues (If you believe this is still valid please reopen)

Recent updates should be fix this

Getting the same error, with the recent version, after moving to a new server.
is this a port problem?  Worked just fine on the 2 old servers.

Getting the same error, with the recent version, after moving to a new server.
is this a port problem?  Worked just fine on the 2 old servers.

I've the same error after running a connection stable over more than a week.
PHP Notice:  fwrite(): send of 8192 bytes failed with errno=32 Broken pipe in /volume1/public/openHAB/conf/scripts/TD3511/phpMQTT.php on line 435
Line number is differnt. The error happens when using the publish API.

I have the same error - php notice from fwrite on line 435. :/
Makes the module completely unstable and unusable


Non-jQuery version

@bgrins I have been working on a version of this idea which does not depend on jQuery. It takes quite a different approach, but uses some of the ideas present in this repo.
Would you be happy for me to start a new repo with this new code and publish it to NPM?

Cool, it's up to you how you want to proceed - if it has similar features / browser support I'd wouldn't be opposed to taking the changes into master on this repo but don't have any problem if you'd rather start it as a new project.  I'm curious - what's the approach for the new version?

Thanks!

I'm curious - what's the approach for the new version?

A summary of the differences:

The API is new Expanding(textarea)

It separates the wrapper, textarea, and clone into separate classes

It uses ES2015 modules for each class, and is compiled into a single file with Rollup. (The source code is still written in ES5 to keep the build process simple and minimise the impacts of transpilation.)

It uses mocha + chai for running tests/assertions as I find useful the describe / context / it approach
dist/
  expanding.js
src/
  expanding.js
  helpers.js
  textarea-clone.js
  textarea.js
test/
  lib/
    chai.js
    mocha.css
    mocha.js
    sinon.js
  expanding-test.js
  index.html
  integration-test.js
  textarea-clone-test.js
  textarea-test.js

I started writing this with a pull request in mind, but wonder if the diff might be a little difficult to review?

The diff might be hard to review but feel free to send it over and we'll see what it ends up looking like.
What do you think about keeping a jQuery plugin registration that just calls out to new Expanding(textarea) to keep backwards compat?

OK, cool. I'll try and get something together that makes a bit more sense, perhaps by rewriting some of the internal methods to use vanilla JS/DOM.
Re: keeping the jQuery plugin, yeh sounds good, and it is kind of what we're doing already. There is a bit of jQuery code to prevent the plugin from being reinitialised, but it should be fairly straightforward.

Closed by https://github.com/bgrins/ExpandingTextareas/pull/64


Update MoneyFactory annotations

Hi 👋
the MoneyFactory has been updated to include int in its typehints in #533, but the updated generator hasn't been executed yet ;)
I ran it once. This change fixes IDE hints (like this one from PHPStorm):

Would be great to see this in the next patch!
Cheers 🍻 ,
Alex

Cool thanks


source of phishing website

Hi,
where are your phishing links from (some company or PhishTank)?
I want to look for some phishing websites in Chinese,  is there some available sites on your list?
Thanks!

We have our very own. + Others.
For a community based, see https://github.com/phishing-database/phishing.


Custom Validation

I have model with custom validation. I have a validator class and inside the class body I have:
validates_with MyValidator
How to use this validator with form model?
I put to the Form model:
extend ActiveModel::ModelValidations     copy_validations_from MyModel
But this is not working at all. It returns:
ArgumentError - Unknown validator: 'MyValidator'

Hi Tommy,
you don't need the ModelValidations module in your form. You have to make sure the MyValidator name sits in the right namespace: ActiveModel::Validations::MyValidator. This is not Reform's fault but an ActiveModel::Validations limitation.

Hi Tommy, I hope you got this sorted. I'm going to close this as it's been a while but do reopen if you need. 😄


Support for Json Type

Description
`
class Person(BaseModel):
names: Json[Any]
person_json = '''
{
"first_name": "john",
"last_name": "Doe",
"nick_name": "lost"
}
'''
p = Person.json_parse(person_json)
`
This automatically converts the person_json into pydantic Model. But I couldn't find the way to reverse this.

Hi, thanks for the issue. So the JSON is a dict - you mean you want to output the JSON as a string instead of a dict?

Yes. For more info, I am talking about this feature https://pydantic-docs.helpmanual.io/usage/types/#json-type

So, I am not sure what is required here to be honest. Can you show me a code snippet that is failing to do what you need?


Semantic Segmentation state

When doing semantic annotation in the segmentation layer, the state is not visible in the global state passed to the state server, making impossible for a user to save the changes.
For a large datasets and multiple users annotation we will probably want something else than a state server scripts, possibly a technology related to how graph operations are saved #97

Should be similar to what happening with equivalances
https://github.com/seung-lab/neuroglancer/blob/master/src/neuroglancer/segmentation_user_layer.ts#L186

Should we extend equivalences to have multiple values instead of just one?

Outdated, abandoned. Future Nico will maybe deal with semantic labels.


Support versionize on packed structs

Signed-off-by: Rob Bradford<EMAIL_ADDRESS>

@rbradford do you still need this? Is it building for you?

@acatangiu Yup, works grand. We use it in production in Cloud Hypervisor.


Add security context to containers running as non-root

@cgaege Can you take a look at the changes and merge?


Bulk status change appears not to be working

I've tried to update a submission status for a single submission using the batch status changer but I get a message telling me it's not possible to change the status for this combination of submissions.

Additionally when a status can be changed the message still displays.


Asynchronous version of Session.Get & Session.Set

As we know that there many session management techniques are present such as InMemory, Distributed or even Database, some of them may takes long time to get or set the values in the store, indeed we need to support async as first class

@hishamco : We did discuss about this in our team and felt that exposing async apis for Get and Set would not give a good user experience. More details below:

Session.Get("key1")  -> if the session is not loaded already, then its loaded and would be a synchronous call
Session.Get("key2") -> This call would not be over the network as all the session data has already been loaded into memory and would just be an in-memory lookup for this key.

To avoid making the the blocking call in step 1. above, you could do something like:

await Session.LoadAsync(); -> async call after which all the data related to current session is loaded into memory.
Session.Get("key1") -> in-memory lookup
Session.Get("key2") -> in-memory lookup

Thanks for clarification @kichalla, also I have seen a related issue #31 if i'm not wrong


Consider binarizing the weights in the splat decoder

Here, we have a scalar value which adjusts the weight of each Gaussian in the decoder:
https://github.com/alan-turing-institute/affinity-vae/blob/23349e544dc86cf335ab0e948ccea333f40ca3b4/avae/decoders/differentiable.py#L188-L195
We could consider using binarization of this linear layer to improve the reconstruction.

Some useful thoughts in this post:
https://discuss.pytorch.org/t/binary-activation-function-with-pytorch/56674/5


FEAT: customize fallback through props for lazy loading

Added props for providing customized fallback in case of lazy loading.

Nice work!

Nice work!


Process aborting with "Insufficient data to determine video format" error when fine-tuning

Hi, I am trying to fine tune the pre-trained(Kinetics) R2Plus1D model on my dataset. I created the train and test LMDB of my dataset like this:
#Creating Training LMDB
python /home/rahul/R2Plus1D/data/create_video_db.py --list_file=/home/rahul/Dataset/train_test_lists/train_list.csv --output_file=/home/rahul/Dataset/train_test_lists/LMDB_Training --use_list=1

#Creating testing LMDB
python /home/rahul/R2Plus1D/data/create_video_db.py --list_file=/home/rahul/Dataset/train_test_lists/test_list.csv --output_file=/home/rahul/Dataset/train_test_lists/LMDB_Testing --use_list=1

The format of the csv files is:
org_video("the path of videos"),label("integer label of the video")
And then I run this to train the model:
python /home/rahul/R2Plus1D/tools/train_net.py \
--train_data=/home/rahul/Dataset/train_test_lists/LMDB_Training \
--test_data=/home/rahul/Dataset/train_test_lists/LMDB_Testing \
--model_name=r2plus1d --model_depth=18 \
--clip_length_rgb=16 --batch_size=4 \
--pretrained_model=/home/rahul/R2Plus1D/pre-trained-models/r2.5d_d18_l16.pkl \
--db_type='pickle' --is_checkpoint=0 \
--gpus=0,1 --base_learning_rate=0.0002 \
--epoch_size=40000 --num_epochs=8 --step_epoch=2 \
--weight_decay=0.005 --num_labels=89

But the process is getting aborted with these errors:
E0612 23:00:22.106964  9409 video_decoder.cc:75] Insufficient data to determine video format
E0612 23:00:22.107067  9411 video_decoder.cc:75] Insufficient data to determine video format
E0612 23:00:22.106992  9412 video_decoder.cc:75] Insufficient data to determine video format
E0612 23:00:22.106918  9407 video_decoder.cc:75] Insufficient data to determine video format
E0612 23:00:22.107008  9413 video_decoder.cc:75] Insufficient data to determine video format
E0612 23:00:22.107035  9414 video_decoder.cc:75] Insufficient data to determine video format
E0612 23:00:22.106915  9408 video_decoder.cc:75] Insufficient data to determine video format
E0612 23:00:23.469903  9409 video_decoder.cc:75] Insufficient data to determine video format
E0612 23:00:23.469926  9411 video_decoder.cc:75] Insufficient data to determine video format
E0612 23:00:23.469921  9413 video_decoder.cc:75] Insufficient data to determine video format
E0612 23:00:23.469907  9410 video_decoder.cc:75] Insufficient data to determine video format
E0612 23:00:23.469923  9412 video_decoder.cc:75] Insufficient data to determine video format
E0612 23:00:23.469954  9414 video_decoder.cc:75] Insufficient data to determine video format
E0612 23:00:23.469923  9407 video_decoder.cc:75] Insufficient data to determine video format
E0612 23:00:23.469995  9408 video_decoder.cc:75] Insufficient data to determine video format

..........
..........
.........
.........
.........
 Encountered CUDA error: device-side assert triggered Error from operator:
input: "gpu_0/comp_4_spatbn_1" input: "gpu_0/comp_4_conv_2_middle_w" input: "gpu_0/__m1_shared" output: "gpu_0/comp_4_conv_2_middle_w_grad" output: "gpu_0/__m2_shared" name: "" type: "ConvGradient" arg { name: "no_bias" i: 1 } arg { name: "kernels" ints: 1 ints: 3 ints: 3 } arg { name: "ws_nbytes_limit" i: 67108864 } arg { name: "exhaustive_search" i: 1 } arg { name: "strides" ints: 1 ints: 1 ints: 1 } arg { name: "pads" ints: 0 ints: 1 ints: 1 ints: 0 ints: 1 ints: 1 } arg { name: "order" s: "NCHW" } device_option { device_type: 1 cuda_gpu_id: 0 } engine: "CUDNN" is_gradient_op: true
E0612 23:00:23.986739  9419 net_dag.cc:195] Secondary exception from operator chain starting at '' (type 'SoftmaxWithLoss'): caffe2::EnforceNotMet: [enforce fail at context_gpu.h:156] . Encountered CUDA error: device-side assert triggered Error from operator:
input: "gpu_1/last_out_L14" input: "gpu_1/label" output: "gpu_1/softmax" output: "gpu_1/loss" name: "" type: "SoftmaxWithLoss" device_option { device_type: 1 cuda_gpu_id: 1 }
F0612 23:00:23.990535  9416 context_gpu.h:107] Check failed: error == cudaSuccess device-side assert triggered
*** Check failure stack trace: ***
F0612 23:00:23.990537  9418 context_gpu.h:107] Check failed: error == cudaSuccess device-side assert triggeredF0612 23:00:23.990561  9420 context_gpu.h:107] Check failed: error == cudaSuccess device-side assert triggeredF0612 23:00:23.990689  9422 context_gpu.h:107] Check failed: error == cudaSuccess device-side assert triggeredF0612 23:00:23.990710  9421 context_gpu.h:107] Check failed: error == cudaSuccess device-side assert triggeredF0612 23:00:23.990717  9417 context_gpu.h:107] Check failed: error == cudaSuccess device-side assert triggeredF0612 23:00:23.991060  9419 context_gpu.h:107] Check failed: error == cudaSuccess device-side assert triggered
*** Check failure stack trace: ***
F0612 23:00:23.990537  9418 context_gpu.h:107] Check failed: error == cudaSuccess device-side assert triggeredF0612 23:00:23.990561  9420 context_gpu.h:107] Check failed: error == cudaSuccess device-side assert triggeredF0612 23:00:23.990689  9422 context_gpu.h:107] Check failed: error == cudaSuccess device-side assert triggeredF0612 23:00:23.990710  9421 context_gpu.h:107] Check failed: error == cudaSuccess device-side assert triggeredF0612 23:00:23.990717  9417 context_gpu.h:107] Check failed: error == cudaSuccess device-side assert triggeredF0612 23:00:23.991060  9419 context_gpu.h:107] Check failed: error == cudaSuccess device-side assert triggered
MyScripts/trainWithPretrainedModel.sh: line 10:  9358 Aborted

Here is what stdout dump that I passed to a file:
Ignoring @/caffe2/caffe2/contrib/nccl:nccl_ops as it is not a valid file.
Ignoring @/caffe2/caffe2/contrib/gloo:gloo_ops as it is not a valid file.
Ignoring @/caffe2/caffe2/contrib/gloo:gloo_ops_gpu as it is not a valid file

Can you please help me out of this?
Thanks,
Rahul Bhojwani

Looks like you failed at "SoftmaxWithLoss" layer. In your command, you have --num_labels=89, the layer name at the error is "gpu_1/last_out_L14", that mean you prediction layer has only 14 outputs at softmax, but you label goes out of scope e.g. label >=14. Then you hit this error. Check careful why you input --num_labels=89 why the net still create the last layer with 14 outputs.

The integer label of the video must start from 0, here happened some errors when I started from 1.

@dutran : Actually the --num_labels was set to 14 in the script I ran. I just copied the other script here by mistake. My bad. Will correct it now.
@murilovarges : I actually have my labels starting from 1. Let me try this.

Then, changing your labels to start from 0 as @murilovarges suggested will solve your issue.

Hey. So, I did that and the process is not getting aborted and the training is running. But still these warnings are being constantly being generated:
E0612 23:00:22.106964  9409 video_decoder.cc:75] Insufficient data to determine video format
E0612 23:00:22.107067  9411 video_decoder.cc:75] Insufficient data to determine video format
E0612 23:00:22.106992  9412 video_decoder.cc:75] Insufficient data to determine video format
E0612 23:00:22.106918  9407 video_decoder.cc:75] Insufficient data to determine video format
E0612 23:00:22.107008  9413 video_decoder.cc:75] Insufficient data to determine video format
E0612 23:00:22.107035  9414 video_decoder.cc:75] Insufficient data to determine video format

Can you help me understand the reason and whether it is a problem or not?
Thanks

It looks like, your video does not have some meta data, decoder does not know how to decode.

This message is in https://github.com/pytorch/pytorch/blob/master/caffe2/video/video_decoder.cc#L75.
I guess can happen when the video is very small too.

Similar issue in https://github.com/facebookresearch/video-nonlocal-net/issues/12

Oh great. This is very helpful. Thanks a lot @dutran and @murilovarges . 👍


feat: support for Quilt Mappings

Adds support for Quilt Mappings and Quilt Hashed

I'll remove Hashed from there then. Two things I had missed: namespaceFriendlinessIndex & historyNamespaces. I'll add Hashed after Intermediary for the first one, I'm not quite sure what the second one does though. Is Hashed relevant there?

the friendliness index is used when selecting a "friendly" name - the namespaces are iterated in order and the first available name is used, that's the one you see in the documentation links, overview, package overview, ...
the history (ancestry) namespaces are used for following a trail of an element (class or a member) through multiple namespaces, i.e. when one namespace stops mapping an element, but another one still does, the elements can still be connected to maintain the ancestry line


Added Jetpack keyword for hotspots

Newer Verizon hotspots are no longer branded internally as MiFi devices. The string 'Jetpack' is found throughout the http response instead.

Justin– I force pushed to my fork so it looks like I lost your previous comment. I've tested this new code while connected to an older MiFi, a newer Jetpack, and a wired connection. Everything appears to work as expected.


Change traceroute edst to string

The edst field of traceroute results contains a string (namely the IP
address in regular notation), same as the from field.
Example:

https://atlas.ripe.net/api/v2/measurements/1430077/results?start=1497231166&stop=1497231166&format=json

The API documentation incorrectly specifies this field to be an int.

Have this been reported to RIPE NCC?

@ogasser Ping? Would like to know if you have reported this to RIPE NCC before I close/merge/forget. I can do it if you haven't.

I did not yet report this to RIPE NCC. @jelu what is the appropriate channel to report RIPE Atlas API documentation bugs?

@ogasser you can email atlas at ripe.net

I reported the bug via email.


kotest task not generated

This is an Android project (uses the com.android.library plugin as well).
Gradle config:
Gradle version 7.2
Android Gradle Plugin version 7.1.1
Kotlin version 1.6.10
plugins {
    id("io.kotest") version "0.3.9"
}

dependencies {
    val kotestVersion = "5.1.0"
    testImplementation("io.kotest:kotest-framework-engine-jvm:$kotestVersion")
    testImplementation("io.kotest:kotest-assertions-core-jvm:$kotestVersion")
    testImplementation("io.kotest:kotest-property:$kotestVersion")
    testImplementation("io.kotest.extensions:kotest-extensions-koin:1.1.0")
}

Running ./gradlew tasks outputs under "Verification tasks":
Verification tasks
------------------
check - Runs all checks.
checkJetifier - Checks whether Jetifier is needed for the current project
connectedAndroidTest - Installs and runs instrumentation tests for all flavors on connected devices.
connectedCheck - Runs all device checks on currently connected devices.
connectedDevDebugAndroidTest - Installs and runs the tests for devDebug on connected devices.
deviceAndroidTest - Installs and runs instrumentation tests using all Device Providers.
deviceCheck - Runs all device checks using Device Providers and Test Servers.
lint - Runs lint on the default variant.
lintAnalyzeDevDebug - Run lint analysis on the devDebug variant
lintAnalyzePlayRelease - Run lint analysis on the playRelease variant
lintAnalyzeSnapshotRelease - Run lint analysis on the snapshotRelease variant
lintAnalyzeSprintRelease - Run lint analysis on the sprintRelease variant
lintDevDebug - Print text output from the corresponding lint report task
lintFix - Runs lint on the default variant and applies any safe suggestions to the source code.
lintFixDevDebug - Fix lint on the devDebug variant
lintFixPlayRelease - Fix lint on the playRelease variant
lintFixSnapshotRelease - Fix lint on the snapshotRelease variant
lintFixSprintRelease - Fix lint on the sprintRelease variant
lintPlayRelease - Print text output from the corresponding lint report task
lintReportDevDebug - Run lint on the devDebug variant
lintReportPlayRelease - Run lint on the playRelease variant
lintReportSnapshotRelease - Run lint on the snapshotRelease variant
lintReportSprintRelease - Run lint on the sprintRelease variant
lintSnapshotRelease - Print text output from the corresponding lint report task
lintSprintRelease - Print text output from the corresponding lint report task
test - Run unit tests for all variants.
testDevDebugUnitTest - Run unit tests for the devDebug build.
testPlayReleaseUnitTest - Run unit tests for the playRelease build.
testSnapshotReleaseUnitTest - Run unit tests for the snapshotRelease build.
testSprintReleaseUnitTest - Run unit tests for the sprintRelease build.

As you see we are using flavors like DevDebug, SnapshotRelease etc.

Please find a minimum reproducible project here:
https://github.com/lukasz-kalnik-gcx/KotestAndroid


Validation Set During Training

Would it be possible to use a validation set during training so that I can see if my model overfits?

Yes you could, but you need to fix a the code a little bit, in data.py and parametrs.py.

I need it too!


killDataSourceWhitelist cannot be set using Coordinator UI

error:"Can not deserialize instance of java.util.HashSet out of VALUE_STRING token↵ at [Source: HttpInputOverHTTP@7522095a; line: 1, column: 220]"

@pjain1 this looks like a bigger problem, it actually breaks the coordinator ui entirely if a whitelist has been set through the REST API. Looks like this was broken in https://github.com/druid-io/druid/pull/1990 when adding that field. @himanshug can you look into that?

@xvrl i have little/no experience with the coordinator ui, coffeen, node etc so far, can someone more knowledgeable point me to the place where problem lies and I can take a look at it. I see that most of it is written by @tarekrached , is he still maintaining it ?

@himanshug no he is not.
At some point we will rebuild the coordinator UI but for the time being we need to fix this

@himanshug the problem is with #1990 not the UI

If I had to guess, the UI probably doesn't support sending/receiving json arrays, which it would have to in order to deal with killDataSourceWhitelist.

@gianm that makes sense

@gianm that sounds true, @fjy POST api directly to coordinator works as expected.

actually deserialization of json array works fine, so if someone updated killDataSourceWhitelist using direct POST request and then viewing the config in the coordinator UI, it shows up as a comma separated list of dataSources fine.
however, if someone now wants to save updated config via the ui, then that does not work because request sent does not have a json array in killDataSourceWhitelist but a string.

I'm gonna fix it on the server by optionally accepting comma separated list of things in a string in addition to a json array... UI sends comma separated list as a string , so that would work fine.


SendSmtpEmail.ScheduledAt has type time.Time

Summary
The recent release v2.1.0 introduced a field ScheduledAt time.Time on the lib.SendSmtpEmail struct. This breaks existing code.
Background
Unfortunately, the Go JSON encoder is unable to omit time.Time zero values. Instead, zero values of time.Time are marshalled into "0001-01-01T00:00:00Z".
For example,
json.NewEncoder(os.Stdout).Encode(lib.SendSmtpEmail{})

will output
{"scheduledAt":"0001-01-01T00:00:00Z"}

Problem
The latest release of sendinblue/APIv3-go-library introduced the following field:
// UTC date-time on which the email has to schedule (YYYY-MM-DDTHH:mm:ss.SSSZ). Prefer to pass your timezone in date-time format for scheduling. There can be an expected delay of +5 minutes in scheduled email delivery. **Please note this feature is currently a public beta**.
ScheduledAt time.Time `json:"scheduledAt,omitempty"`

By solely updating the sendinblue Go SDK, previously working code to send emails will no longer work because

this field will have the zero value by default
will be marshalled to 0001-01-01T00:00:00Z
sendinblue will return a 400 error Scheduled date should not be less than current date

Side note
Other structs do have a ScheduledAt field already (e.g. CreateEmailCampaign or CreateSmsCampaign) but there this field is generated as a string, not a time.Time. I guess the reason was to circumvent the above problem. Probably this is the way to go for the SendSmtpEmail as well.

I had the exact same issue.
It is very surprising to see a feature which is supposed to be in beta from SendInBlue's own documentation (see screenshot below) breaking code which is in production.

After the previous issue (#23) having to do with a mistyped API exposed by this library, it looks like SendInBlue doesn't test its own SDK at all. To be honest, this doesn't instil confidence in your product and the way you maintain it. As I'm currently working on a prototype, I was hoping to integrate your SDK but I'm now much more likely to go for a competitor which takes care of its SDK. This is a shame since I really wanted to use your service.

Hello All,
Apologies for the inconveniences occurred.
This issue occurred because we generate our all SDKs through swagger codegen and omitempty will not work for this as its default zero value is 0001-01-01T00:00:00Z. We have created a task to resolve this issue and will update once it gets resolved meanwhile you can use scheduledAt.IsZero() to resolve the blocker.

same issue here .. had to remove scheduledAt from code manually

Hi @jain-raunak when can we expect this to be fixed?

Hello,
We will pick this task in the very next sprint (starting from 12th Dec). Thanks!

Hello,
This issue got fixed in latest updated version v2.1.2, please verify the same.
Thanks!


hide mpd module completely info when stopped or disconnected

Hi,
as the title says I'm trying to not show the mpd info when it's in state stopped or disconnected. My current setup achieves this by overriding the following format options in waybar/config with empty strings:
        "format-disconnected": "",
        "format-stopped": "",

this is done in combination with transparent `color and background-color in style.css:
#mpd {
    padding: 0 10px;
    margin: 0 4px;
    background-color: #475f94;
    color: #f3f4f5;
}

#mpd.disconnected {
    background-color: transparent;
    color: transparent;
}

#mpd.stopped {
    background-color: transparent;
    color: transparent;
}

#mpd.paused {
    background-color: #51a37a;
}

is there another way to do this? Apologies for the somewhat silly question, but it felt like a bit of a hack doing this, so I'm unsure if this is the correct method. I couldn't find any info in the waybar-mpd (5) that instructs me how to achieve this. (I thought this would be a fairly standard way of how people want to set this up but maybe it's just me ;))
Thanks for any pointers.
my version is:
Waybar v0.9.2-140-g759602a (branch 'master')
thanks,
~DA

I strongly agree with your point that implementing something like the "exec-if" field from the custom module for other modules would be tremendously useful.
In general, conditionally displaying/executing modules as implemented in #394 seems like an extremely useful capability, though, according to the pull request, this idea has been rejected by waybar's creator in favor of a different (also abandoned) concept. (#100).
~Non-dyslexic Messianic Jew who mirror-writes anyway
[ ^ Just toying with your handle @DyslexicAtheist ; no disrespect intended]


[WIP] Preserve queryParams

So far I've proven that this is doable by using the -routing service and observing changes in the state of that service.
Questions to be answered:

That service is not yet public. Ask the oracle (@rwjblue) wether or not this is a terrible practice
Measure performance penalty. If this brings this addon in the like of performance with a regular {{link-to}} that would defeat the purpose of using it.

@GavinJoyce May I take some advice on how to measure the perf of this change? I imagine that setting an observer won't affect performance that much (although all helpers will recompute when qps change while now it doesn't, but that is a bug to me)

@cibernox I just manually copy some code into a custom benchmark in a throwaway ember-performance branch and compare the results. I'm happy to do this later this evening if you want?
Thanks for doing this, this is a nasty bug and it's great that it'll be fixed

@GavinJoyce That would be great. That said, don't merge it yet. I need to test more edge cases.

Transition from /foo/bar?one=1&two=2 with {{href-to 'foo.qux'}}, where oneis a qp that belongs tofooandtwois a qp that belongs tobarbut not toqux. Expected: one=1is preserved,two=2` dissapears.
Transition from /foo/bar?one=1&two=2 with {{href-to 'foo.qux'}}, where oneis a qp that belongs tofooandtwois a qp that belongs tobarandqux`. Expected: Both remain.
Transition from /foo/bar?one=1&two=2 with {{href-to 'foo.qux'}}, where oneandtwoare qps that belongs tofoo`. Expected: Both remain.
Transition from /foo/bar?one=1&two=2 with {{href-to 'foo.qux'}}, where oneandtwoare qps that belongs tobarandqux`. Expected: Unsure.
Transition from /foo/bar?one=1&two=2 with {{href-to 'another'}}, where oneandtwoare qps that belongs tofooandanother`. Expected: None is preserved (I think).

@cibernox I didn't manage to get benchmark data, I was planning to modify https://github.com/GavinJoyce/ember-performance/pull/1 but ember-performance has changed significantly since then.
onQpsChange: Em.observer('_routing.currentState', function() {
  this.recompute();
}),

This seems like it will be expensive on any route transition. I wonder can we only recompute if a query param has changed?

I've been able to create a couple more complex scenarios that still fail. Over the weekend I'll try to figure out how to better do this.
I've also seen on the core notes that they agreed to proceed with your url-for RFC, so I might ping someone to give some guidance on this.

I was experimenting with this because I need it on my own project. Few observations:

observer on _routing.currentState fires even though routerJsState.fullQueryParams did not change. I've fixed it with my proxy routing service that abstracts currentQueryParams away.
fullQueryParams sometimes contained old value so I'm using routerJsState.queryParams instead. Not sure what's the difference between those two.
It only makes sense to preserve query params that belong to parent route handlers of the route being passed to href-to.
I'd maybe rather opt-in into this feature on every {{href-to}} than have it on by default because of performance penalty on route switching. On my (highly specific) use case this performance penalty is significant.

Is it worth investigating this with the new public router service?

@scalvert I believe it's not. I mean, its probably worth pursuing, but the approach would be totally different from different. Tests seem valuable tho


add gpups example

Hi, How to export model by FleetX?
I can't find the code to implement export model


restrict-seccomp policy does not comply with Pod Security Standards (Restricted)

The Restricted profile in the Pod Security Standards requires that the Seccomp profile be specified, and does not allow it to be unspecified, but the kyverno's restrict-seccomp policy allows it to be unspecified.

Hi @sotoiwa - can you point me to where PSS requires Seccomp profile?

@realshuting - please check here: https://kubernetes.io/docs/concepts/security/pod-security-standards/ (search for Seccomp)
Looks like there is Kubernetes version specific information now, which we will need to update to:
Seccomp profile must not be explicitly set to Unconfined.

Restricted Fields

spec.securityContext.seccompProfile.type
spec.containers[*].securityContext.seccompProfile.type
spec.initContainers[*].securityContext.seccompProfile.type
spec.ephemeralContainers[*].securityContext.seccompProfile.type
Allowed Values

Undefined/nil
RuntimeDefault
Localhost

Seccomp profile must be explicitly set to one of the allowed values. Both the Unconfined profile and the absence of a profile are prohibited.

Restricted Fields

spec.securityContext.seccompProfile.type
spec.containers[*].securityContext.seccompProfile.type
spec.initContainers[*].securityContext.seccompProfile.type
spec.ephemeralContainers[*].securityContext.seccompProfile.type
Allowed Values

RuntimeDefault
Localhost
The container fields may be undefined/nil if the pod-level spec.securityContext.seccompProfile.type field is set appropriately. Conversely, the pod-level field may be undefined/nil if _all_ container- level fields are set.

Thanks for bringing this to our attention @sotoiwa. I have built an updated version of this policy to conform to the latest standards. Please try it out and ensure it works correctly. A couple things:

It is set to enforce right now
ephemeralContainers is not being checked because this is still an alpha feature in v1.22
The test resources in kyverno/policies for this sample have a couple bugs. They will be updated with many more alongside this new policy.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-seccomp
  annotations:
    policies.kyverno.io/title: Restrict Seccomp
    policies.kyverno.io/category: Pod Security Standards (Restricted)
    policies.kyverno.io/severity: medium
    policies.kyverno.io/subject: Pod
    policies.kyverno.io/description: >-
      The runtime default seccomp profile must be required, or only specific
      additional profiles should be allowed. This policy, requiring Kubernetes
      v1.19 or later, ensures that only the `RuntimeDefault` or `Localhost` is
      used as a `type` and that it is not unset.
spec:
  background: true
  validationFailureAction: enforce
  rules:
  - name: restrict-seccomp
    match:
      resources:
        kinds:
        - Pod
    validate:
      message: >-
        Use of custom Seccomp profiles is disallowed. The fields
        spec.securityContext.seccompProfile.type,
        spec.containers[*].securityContext.seccompProfile.type, and
        spec.initContainers[*].securityContext.seccompProfile.type
        must be set to `RuntimeDefault` or `Localhost`.
      anyPattern:
      - spec:
          securityContext:
            seccompProfile:
              type: "RuntimeDefault | Localhost"
      - spec:
          containers:
          - securityContext:
              seccompProfile:
                type: "RuntimeDefault | Localhost"
  - name: restrict-seccomp-initcontainers
    match:
      resources:
        kinds:
        - Pod
    validate:
      message: >-
        Use of custom Seccomp profiles is disallowed. The fields
        spec.securityContext.seccompProfile.type,
        spec.containers[*].securityContext.seccompProfile.type, and
        spec.initContainers[*].securityContext.seccompProfile.type
        must be set to `RuntimeDefault` or `Localhost`.
      anyPattern:
      - spec:
          securityContext:
            seccompProfile:
              type: "RuntimeDefault | Localhost"
      - spec:
          =(initContainers):
          - securityContext:
              seccompProfile:
                type: "RuntimeDefault | Localhost"

All of these Pod resources should fail validation:
---
apiVersion: v1
kind: Pod
metadata:
  name: seccomp-pod-spec-undefined
spec:
  securityContext:
    runAsNonRoot: true
  containers:
  - name: seccomp
    image: busybox
    command: ["sleep", "9999"]
---
apiVersion: v1
kind: Pod
metadata:
  name: seccomp-pod-spec-unconfined
spec:
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: Unconfined
  containers:
  - name: seccomp
    image: busybox
    command: ["sleep", "9999"]
---
apiVersion: v1
kind: Pod
metadata:
  name: seccomp-pod-cont-undefined
spec:
  securityContext:
    runAsNonRoot: true
  containers:
  - name: seccomp
    image: busybox
    command: ["sleep", "9999"]
---
apiVersion: v1
kind: Pod
metadata:
  name: seccomp-pod-cont1-unconfined
spec:
  securityContext:
    runAsNonRoot: true
  containers:
  - name: seccomp
    image: busybox
    command: ["sleep", "9999"]
    securityContext:
      seccompProfile:
        type: Unconfined
---
apiVersion: v1
kind: Pod
metadata:
  name: seccomp-pod-cont2-inconsistent
spec:
  securityContext:
    runAsNonRoot: true
  containers:
  - name: seccomp00
    image: busybox
    command: ["sleep", "9999"]
    securityContext:
      seccompProfile:
        type: RuntimeDefault
  - name: seccomp01
    image: busybox
    command: ["sleep", "9999"]
    securityContext:
      seccompProfile:
        type: Unconfined
---
apiVersion: v1
kind: Pod
metadata:
  name: seccomp-pod-cont2-inconsistent1
spec:
  securityContext:
    runAsNonRoot: true
  containers:
  - name: seccomp00
    image: busybox
    command: ["sleep", "9999"]
  - name: seccomp01
    image: busybox
    command: ["sleep", "9999"]
    securityContext:
      seccompProfile:
        type: RuntimeDefault
---
apiVersion: v1
kind: Pod
metadata:
  name: seccomp-pod-initcont-undefined
spec:
  securityContext:
    runAsNonRoot: true
  initContainers:
  - name: initseccomp
    image: busybox
    command: ["sleep", "9999"]
  containers:
  - name: seccomp
    image: busybox
    command: ["sleep", "9999"]
    securityContext:
      seccompProfile:
        type: RuntimeDefault
---
apiVersion: v1
kind: Pod
metadata:
  name: seccomp-pod-initcont-unconfined
spec:
  securityContext:
    runAsNonRoot: true
  initContainers:
  - name: initseccomp
    image: busybox
    command: ["sleep", "9999"]
    securityContext:
      seccompProfile:
        type: Unconfined
  containers:
  - name: seccomp
    image: busybox
    command: ["sleep", "9999"]
    securityContext:
      seccompProfile:
        type: RuntimeDefault

@JimBugwadia and/or @realshuting would appreciate your review and vetting of this policy if you have time. As this is a PSS policy, it's quite important that it be rock solid.

Updated above comment with full array of test resources and test manifest. These should test every possible combination of fields to ensure the policy works as it should. Please validate and ensure there are no holes anywhere.

@sotoiwa please review and comment or else indicate if this issue can be closed.

I've seen the policy and the tests. I think it's all good. Thank you!

@chipzoller - I'm checking tests results against this policy, got 3 questions:

it seems like the type needs to be set to runtime/default if specified, while in all the above resources they are set to RuntimeDefault, did I miss anything?
for those unset resources, shouldn't the status be pass since they are unset?
the type is set to Localhost in the resource seccomp-good-pod-localhost, shouldn't the status be fail?

The original configuration of this policy which checked runtime/default was, I believe, a mistake. Upon checking the reference here, that value was applicable only for an annotation pre v1.19. I think in this version we (read: I) finally got it right.
When consulting the updated PSS for Seccomp (v1.19+), it specifies that unset should be denied. Quote:

Seccomp profile must be explicitly set to one of the allowed values. Both the Unconfined profile and the absence of a profile are prohibited.
Restricted Fields
spec.securityContext.seccompProfile.type
spec.containers[*].securityContext.seccompProfile.type
spec.initContainers[*].securityContext.seccompProfile.type
spec.ephemeralContainers[*].securityContext.seccompProfile.type

Allowed Values
RuntimeDefault
Localhost

The container fields may be undefined/nil if the pod-level spec.securityContext.seccompProfile.type field is set appropriately. Conversely, the pod-level field may be undefined/nil if all container- level fields are set.

See above bullet. Localhost is explicitly allowed. Note that the Kubernetes internal admission controller will perform the validation of any additional fields should type: Localhost. Therefore, Kyverno need only check that the value of the type key be one of two possibilities.

Thank you @chipzoller for the clarification, I somehow missed the policy you attached above. All make sense now!

See linked PR which contains the updates reflected during the course of this issue.


add h2-api plugin

Description
Upload permission for new plugin h2-api
Git repo: https://github.com/jenkinsci/h2-api-plugin
Hosting issue: https://issues.jenkins-ci.org/browse/HOSTING-767
Submitter checklist for changing permissions

Always

[x] Add link to plugin/component Git repository in description above

For a newly hosted plugin only

[x] Add link to resolved HOSTING issue in description above

For a new permissions file only

[x] Make sure the file is created in permissions/ directory
[x] artifactId (pom.xml) is used for name (permissions YAML file).
[x] groupId / artifactId (pom.xml) are correctly represented in path (permissions YAML file)
[x] Check that the file is named plugin-${artifactId}.yml for plugins

When adding new uploaders (this includes newly created permissions files)

[ ] Make sure to @mention an existing maintainer to confirm the permissions request, if applicable
[x] Use the Jenkins community (LDAP) account name in the YAML file, not the GitHub account name
[x] All newly added users have logged in to Artifactory at least once

for consistency with HOSTING-768 MySQL API Plugin and HOSTING-769 PostgreSQL API Plugin, I'll change the groupId of the h2-api from org.jenkins-ci.plugins.h2-api to io.jenkins.plugins.

You'll need to update this file as well to fix the path before releasing.


Add arm64 build support

This PR is still in progress but want get more feedbacks in advance.
To re-enable build support for arm,

Build on internal ARM instance in blossom cluster
Provide artifacts w/ classifiers like cuda11-arm64, cuda12-arm64
Use dockerhub/cuda rockylinux image for arm and update corresponding tools to support the build and test (since there is no official arm support for centos7)
Disabled faultinj build https://github.com/NVIDIA/spark-rapids-jni/issues/1368
Would not run sanitizer check as it failed core dump in arm ENV (currently do not have time to check the details)

In this case, our plugin jar build would also need to be updated to support release w/ -arm64 classifiers.
So the final releases would be like,
Default jar (maven required):
rapids-4-spark_2.12-<version>.jar (x86 cuda11 one)
X86 jars:
rapids-4-spark_2.12-<version>-cuda11.jar
rapids-4-spark_2.12-<version>-cuda12.jar
ARM64 jars:
rapids-4-spark_2.12-<version>-cuda11-arm64.jar
rapids-4-spark_2.12-<version>-cuda12-arm64.jar

@sameerz @GaryShen2008 @jlowe @revans2 @tgravescs @gerashegalov
Please help review and let me know if we are good to continue the current idea, thanks!
Last time (a year ago) we also discussed having another option that would provide different <cpu_arch> based artifacts instead of classifiers like
rapids-4-spark_2.12-<version>.jar
or/and
rapids-4-spark_2.12_amd64-<version>.jar
and
rapids-4-spark_2.12_aarch64-<version>.jar

this may require more refactoring works to our existing project structure and CICD for both JNI and plugin repos.

build

Approach looks OK to me. Ideally we would build an x86_64 for the non-ARM jar we have today, but I agree there's a lot of changes to CI and user education to accommodate that. Since the "cuda11" classifier implies it runs on all supported processor architectures, I'm OK in the short term with that being equivalent to the cuda11-x86_64 classifier since the former is a superset of the latter, but eventually we should educate users here.
Maybe we should target deploying both the cuda11 and cuda11-x86_64 artifacts in the short term, which will be identical, and sstart to migrate existing users to the latter artifact. Then we can, if desired, build an uber artifact in cuda11 that is all supported processors in one jar.

Thanks! For the short term I will just continue the work of classifiers,
also filed https://github.com/NVIDIA/spark-rapids-jni/issues/1387 which may require help from developers

build

@jlowe can you help review current changes?
Due to lack of arm resources, planning to merge and provide internal CI for developers fisrt, thanks


Lightgbm R installation

Hi, everyone,
I'm trying to install lightgbm in R (Windows 10, RStudio 3.6.3). I have already read and tried steps in #2995 and #2997 but I have a problem at a stage;
C:\Users\MGS\LightGBM>git fetch origin r/msvc-support
fatal: couldn't find remote ref r/msvc-support
Then I have tried;
C:\Users\MGS\LightGBM>git pull r/msvc-support
fatal: 'r/msvc-support' does not appear to be a git repository.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights and the repository exists.
But I got an error again. How can I solve this problem?
Thank you so much.

Thanks for using LightGBM.
Please follow the installation instructions in our official documentation: https://github.com/microsoft/LightGBM/blob/master/R-package/README.md#installation
If the instructions do not work, please comment here with some explanation of exactly the code you ran and what error message or other logs you are seeing and we will re-open the issue.

I've installed git and CMake(3.16.6)
I've installed Visual Studio Code (April 2020 version) (Actually, I want to use lightgbm in RStudio, so I have RStudio 3.6.3)
I've also installed RTools35.
Then, in cmd I wrote your instructions(both of them) and I get this;

Can you show me the value of your PATH environment variable? If make is not found, I suspect that you do not have Rtools directories on your PATH per https://github.com/microsoft/LightGBM/tree/master/R-package#windows-preparation.
By the way, the versions of R and RStudio are two different things. You can get your version of R by running R --version. That is probably 3.6.3.

Your PATH looks like it has the right things in it. Did you start a completely new CMD shell after setting those environment variables?
What branch of LightGBM are you on? Your git clone command in the first screenshot failed so I'm not sure what is in the LightGBM directory you're working from.
Could you please start a new CMD window, remove the existing LightGBM directory you have, and run the code from the install instructions?
git clone --recursive https://github.com/microsoft/LightGBM
cd LightGBM
Rscript build_r.R

I start a completely new CMD shell after setting environment variables but I got the same error again. Also, I've tried remove the existing LightGBM directory I have but I couldn't do that, so I wrote the command "dir LightGBM", it can be seen below;

Can you please show me the output of running these two commands from that LightGBM folder?
git branch -v

git log -n 5

My concern is that by following comments from the issues you referenced (#2995, #2997), you may have your local LightGBM in a state that doesn't work any more. Since those comments were only temporary fixes for those issues which have now been fixed on master.

Great, thanks! Can you please run git clone master to get the most recent changes, then try running Rscript build_r.R again?
If you get the same error about make not being found, then we will have to check if your Rtools entries in PATH are set up correctly. I could help with that if you show me a screenshot of Windows Explorer that shows the contents of C:\Program Files\Rtools.

I am really grateful for helping you so quickly but I got an error like this when I run "git clone master" so I couldn't try running Rscript build_r.R again.

ha you are fast! I was thinking about two things at once and made a mistake while typing. There was a typo in my comment and I edited it immediately.
It should have been git pull origin master, not git clone master.

Okay, so I changed and it has become like this, as a result I got the same error again.

So I also attach the screenshot of C:\Program Files\Rtools

aha! We are getting closer! I noticed that you have C:\Program Files\Rtools in your PATH. Can you please change it to C:\Program Files\Rtools\bin, close this CMD shell, open a new one, and re-run Rscript build_r.R from the root of the LightGBM repository?
I suspect that make is in Rtools\bin but not being found because your path ends in Rtools\.
If that's the case I'll make this clearer in our installation documentation, for those who aren't familiar with Rtools and/or setting their PATH.

OH MY GOD! It worked! I have lightgbm package now! You're amazing and I really really appreciate what you are doing and your help. Thank you so much @jameslamb  :)

WOO AWESOME! I am glad it's working for you! Thanks for your patience and for using LightGBM!
For those arriving at this issue from search engines:
If you try to build the R package on Windows and get an error like "warning in system(cmd): 'make' not found", be sure that Rtools\bin is on your PATH environment variable.


Fix hung domains and other VM lifecycle edge cases

Support terminationGracePeriodSeconds for VM pods.
Virtlet will try to wait for this number of seconds for VM
to shut down gracefully, then it will destroy the domain.
Update container statuses on StartContainer()/StopContainer().
Wait for domain to become running in StartContainer().
Lessen the dependency on ContainerStatus()/ListContainers() call
sequence.
Add a comment on why we're removing the domain upon
failed StartContainer() call.
Add libvirttools test for hung domains.

This change is

Reviewed 26 of 26 files at r1.
Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable

Review status: all files reviewed at latest revision, 1 unresolved discussion, some commit checks failed.

pkg/libvirttools/virtualization.go, line 507 at r1 (raw file):

	}

	if err == nil {

Why next if, instead of using else?

Comments from Reviewable

Review status: all files reviewed at latest revision, 1 unresolved discussion, some commit checks failed.

pkg/libvirttools/virtualization.go, line 507 at r1 (raw file):
Previously, jellonek (Piotr Skamruk) wrote…
Why next if, instead of using else?

Actually, in case if err != nil after the WaitLoop(), it may become nil if domain.Destroy() succeeds. StopContainer() is considered successful if it manages to kill the VM, even if it had to destroy the domain to do so.

Comments from Reviewable

Reviewed 26 of 26 files at r1.
Review status: all files reviewed at latest revision, 1 unresolved discussion, some commit checks failed.

pkg/libvirttools/virtualization.go, line 507 at r1 (raw file):
Previously, ivan4th (Ivan Shvedunov) wrote…
Actually, in case if err != nil after the WaitLoop(), it may become nil if domain.Destroy() succeeds. StopContainer() is considered successful if it manages to kill the VM, even if it had to destroy the domain to do so.

Yeah, right.

Comments from Reviewable

Reviewed 3 of 3 files at r2, 1 of 1 files at r3.
Review status: all files reviewed at latest revision, 1 unresolved discussion.

Comments from Reviewable


Updating Docker Traceability permissions

Added myself to all the docker-traceability components. Sorry I forgot
these two :(

https://github.com/jenkinsci/docker-traceability-plugin/tree/master/docker-java-shaded permissions are missing @alexanderrtaylor


Transpose dataset before running spark-perf decision tree timing if using "byCol"

Depends on jkbradley/spark/pull/27
Have not checked if this compiles so please do ping if there are problems
@fabuzaid21 @jkbradley

@feynmanliang I'm making a quick change to make TreeUtils public.  Could you update this to use TreeUtils, rather than copying the method?  That will make it easier to merge @fabuzaid21 's PR for improving TreeUtils.   Thanks!

@jkbradley done, also ran it locally to sanity check
@fabuzaid21 i ran into some problems with 1.6.0 vs 1.5.0. In case you run in to them as well, the fix is to publish the dt-features branch to your local maven repo (publishLocal in SBT), set MLLIB_VERSION=1.6.0 in config.py, and sbt/sbt -Dspark.version=1.6.0 when building the mllib-perf assembly.

Great, thanks @feynmanliang. I'm about to run some experiments on MNIST 8m, so, I'll be careful to use that

@feynmanliang That's the only issue I see.

LGTM, merging. Thanks!


Add bspwm module

There are a few things to discuss. The actual theme has been copied from base16-builder, which had two templates, one for dark schemes and one for light ones. I kept the distinction but is it really something we want ?
Furthermore, I find that using base08 as the focused window border color a bit too agressive, but I'd like other opinion.
Would close #39

Great, I will give it a try as soon as this nvidia build fixes itself.
in addition, can you help me understand each base and what is being used for?  as i know some are being used for alerts/warning as mentioned base08
I've collected the following below but not sure if this is accurate or not
Neutral Colors:
Darkest color, used for the background color of the application or text editor.
base00: ""
Slightly lighter color, used for borders, lines, and other subtle design elements.
base01: ""
Lighter color, used for syntax highlighting elements such as comments or preprocessor directives.
base02: ""
Lightest neutral color, used for plain text or background text.
base03: ""
Accent Colors:
First accent color, used to highlight important elements in the interface such as selected text or active buttons.
base04: ""
Lighter accent color, used for less important interface elements such as inactive buttons or secondary text.
base05: ""
Second accent color, used to highlight additional important elements in the interface such as warning or error messages.
base06: ""
Lightest accent color, used for less important highlighted elements such as less critical warning or status messages.
base07: ""
Bright Colors:
First bright color, used for syntax highlighting elements such as keywords or function names.
base08: ""
Slightly lighter bright color, used for syntax highlighting elements such as numeric constants or string literals.
base09: ""
Lighter version of base09, used for syntax highlighting elements such as escape characters or special symbols.
base0A: ""
Second bright color, used for syntax highlighting elements such as operators or punctuation.
base0B: ""
Color used for syntax highlighting elements such as function arguments or types.
base0C: ""
Color used for syntax highlighting elements such as variables or class names.
base0D: ""
Color used for syntax highlighting elements such as constants or enums.
base0E: ""
Last bright color, used for syntax highlighting elements such as macros or other preprocessor directives.
base0F: ""

also found these
Base00/Background: Used as a primary background color and can be used for subtle glow or light shadow effects; also used as a default color for text.
Base01/Current Line: Used as a highlight color for borders and shadows, or as a start color for subtle gradients; can also be used as a muted color for text.
Base02/Selection: Used as a color for borders, shadows, and drop shadows, or as an end color for gradients; can also be used for highlighting text.
Base03/Comment: Used as a secondary or muted color for borders, shadows, and gradients; can also be used for comments or annotations in text.
Base04/String: Used as a color for attention-grabbing or high-contrast borders, shadows, and gradients; can also be used to highlight or emphasize text.
Base05/Foreground: Used as a default color for borders, shadows, and text; can also be used as a neutral color for gradients.
Base06/Variable: Used as a color for subtle or secondary borders and shadows, or as a muted color for gradients; can also be used for highlighting or styling variable names in text.
Base07/Cursor: Used as a color for subtle or secondary borders and shadows, or as a highlight color for gradients; can also be used as a cursor color in text editors.
Base08/Red: Used as a color for borders, shadows, or gradients that indicate danger, criticality, or urgency; can also be used to indicate errors or warnings in text.
Base09/Orange: Used as a color for highlighting important information or elements in a GUI or application, as well as for styling text that requires attention or emphasis.
Base0A/Yellow: Used as a color for highlighting keywords or important information in various contexts, as well as for styling text that requires attention or emphasis.
Base0B/Green: Used as a color for indicating success, completion, or positive feedback in various contexts, as well as for styling text that indicates progress or status.
Base0C/Cyan: Used as a color for indicating methods, functions, or other related elements in various contexts, as well as for styling text that requires a technical or scientific tone.
Base0D/Blue: Used as a color for indicating links, navigation, or interactive elements in various contexts, as well as for styling text that requires a creative or imaginative tone.
Base0E/Purple: Used as a color for indicating numbers, data, or quantitative information in various contexts, as well as for styling text that indicates luxury or extravagance.
Base0F/Brown: Used as a color for indicating warnings, errors, or critical information in various contexts, as well as for styling text that indicates stability, reliability, or consistency.```

what is the syntax to point stylix.url to this commit?

two templates, one for dark schemes and one for light ones. I kept the distinction but is it really something we want ?

stylix.polarity doesn't necessarily tell us the lightness, as it defaults to either, and is not updated when choosing a premade theme. This is an issue with some existing modules too.

can you help me understand each base and what is being used for?

This table has some information on that, although its mainly focused on syntax highlighting.

what is the syntax to point stylix.url to this commit?

github:dwarfmaster/stylix/bspwm will fetch the latest commit from this branch.

I guess we can drop the distinction between light and dark themes, it should be symmetric anyway. I'll test it with a light theme to be sure (my default theme is dark).
You're right about the other settings not existing, I didn't notice the errors in my logs. If there are only 4 to set it makes it a lot easier to chose. I'll try to propose an update tonight or tomorrow.

What do you think now ? I removed the useless theming. We definitely want presel_feedback_color to have the same color as active_border_color. @realsnick If that looks good to you I'll merge it as is.

thanks! looks good


Extract snippet from file refactor.extract CodeAction

The LSP defines some CodeActionKinds that have special meanings in some Language Clients. One of those is the RefactorExtract action kind.
We could probably offer a RefactorExtract action when you highlight a large range of text inside a section|template|snippet.
When executed, we could do something like this:

For range Y in input file X:

Find all non-global variables in Y
Create a file snippets/Z.liquid with the contents of Y
Replace Y in X by the following:

  {%
    render '#{Z}',
    #{all_the_non_global_variables_as_arguments}
  %}

To be answered:

Can we prompt for a filename during that process?

Resources

https://microsoft.github.io/language-server-protocol/specifications/specification-3-17/#textDocument_codeAction

Sketching out the flow
sequenceDiagram
    actor A as User
    participant C as Client
    participant S as Server
    A-->>C: Select code and right click
    C->>+S: "textDocument/codeAction" request,
context.triggerKind: "invoked"
    S->>-C: CodeAction[]
kind: "refactor.extract"
command: "shopifyLiquid/extractSnippet"
args: uri, range
    C-->>A: Show options
    A-->>C: Select "Extract to new file"
    C->>S: "workspace/executeCommand" request
command: "shopify/extractSnippet"
args: uri, range
    activate S
    S->>S: Compute WorkspaceEdit for refactor
    S->>C: "workspace/applyEdit" request
    activate C
    C->>S: applied!
    deactivate C
    deactivate S

would “Select to refactor into a new file” be possible to take a “Section block” and move it to a /blocks/_blockName.liquid file, and then update the schema for that section to now accept that block?
-Ben


Issues/1525 add sort order consumer group page

[ ] Breaking change? (if so, please describe the impact and migration path for existing application instances)

What changes did you make? (Give an overview)

Implement Sorting to the Consumer Groups Page
Transform the Consumer Group Page to the SmartTable Structure
Styling fix for the Order indication

Is there anything you'd like reviewers to focus on?
How Has This Been Tested? (put an "x" (case-sensitive!) next to an item)

[ ] No need to
[ ] Manually (please, describe, if necessary)
[x] Unit checks
[ ] Integration checks
[x] Covered by existing automation

Checklist (put an "x" (case-sensitive!) next to all the items, otherwise the build will fail)

[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation (e.g. ENVIRONMENT VARIABLES)
[x] My changes generate no new warnings (e.g. Sonar is happy)
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[x] Any dependent changes have been merged

Check out Contributing and Code of Conduct
A picture of a cute animal (not mandatory but encouraged)

The Tests coverage include all the new Features that was added , but i think we need a separate issue to write test for the new tables.


Add threads to SMPT server

This PR adds threads. Threads used in the Client class to save messages into DB.

As I've mentioned before, I don't really know Ruby, but here's what I think these changes are doing:
A poll of worker threads is created, which watches a queue. Rather than create messages directly (one of the times the SMTP server makes a blocking IO call to the database) it's queuing that operation up to be handled by one of the worker threads. Basically, this leaves the asynchronous event loop in place, but handles creating messages in threads.
Assuming that my understanding is correct, I'm not sure this approach will be sufficient. There's a bunch of other areas which are potentially hiding blocking IO, for example: authentication (retrieving the user and password hash from the database), SSL negotiation, or maybe even logging. Queuing all of that stuff up for handling by the worker pool in this approach could prove difficult.
A potentially more complete approach would be to have a worker thread for each connection. So, the main thread would listen, and immediately spawn a new thread (or reuse a thread already in the pool) for any new connection that came in, then everything else would happen on that other thread. That way, anything at all that happens in that connection wouldn't prevent the main thread from accepting new connections (which is more-or-less what this bug is: the blocking IO is slowing down the event loop and preventing it from accepting incoming connections).

@dsnopek I thought about having a worker thread for each connection. Assume we have 4 threads. The first few connections will take all available threads. Each occupied thread should wait until the sender completes sending a message and the thread will be blocked by a slow sender or by a lost connection. So in my mind have a worker thread for each connection is not a good idea.
The good practice is using an event loop and put a slow function to thread https://speakerdeck.com/ledsun/test-asynchronous-functions-with-rspec?slide=10
We can't change Client in threads because it is stateful. So I decided to run in threads hawy part of Client which is creating a message in DB.

The number of threads in the pool (or ideally the min/max) will need to be adjusted depending on the load in order handle all connections, that's for sure. But that's something that already needs to be done in Postal for the REST API (which is thread-based, using a thread per connection) and a pretty normal part of server administration in general (you have to do the same thing for Apache, MySQL, etc). It should be possible to configure a thread pool to maximize a servers resources against the expected load.
But if we leave any blocking IO in the event loop, that just increases the runway until we again reach a point where the event loop starts failing to accept connections again - EVEN THOUGH the server still has available resources to handle those connections. Right now the problem is really extreme (our server has a load of 0.01 and has trouble handling a relatively small number of simultaneous connections) but fixing it only a little will just mean that this will start to happen with a higher number of simultaneous connection, but still leaving server resources unused.
I think the only way to fully solve this issue, is to either make this 100% asynchronous or 100% threading (ie. a thread for each connection).

@dsnopek Too many threads will consume too many resources. It isn't a good idea to create 100 threads if we have 100 connections. Also if one thread serves one connection the thread does nothing while waiting for commands and data from a sender.
I have an idea. We can put each connection in Client instance and store these instances in an array. So we can move all DB queries to Client. When Client instance gets command or data from a sender push a process into threads pool's queue. So threads will not wait for a certain connection. They will process command and data pushed to queue. Is it ok for you?
If you insist on each thread for each connection, let me know, I'll do as you want.

Too many threads will consume too many resources. It isn't a good idea to create 100 threads if we have 100 connections.

100 threads could be too many for a particular server, sure. But tuning the thread pool is the job of the server administrator. This same administrator will already be tuning the thread pool for Postal's API - which is already using a connection per thread - so doing the same kind of tuning for the SMTP server seems like no a big deal.
To assist in this tuning thread pools usually have a min/max rather than just a count. For example, you could set the min to 10 (so it's ready to handle 10 connections quickly without the overhead of initializing a new thread), and the max to 50 (because you know that's the most the server can handle with it's available memory and CPU). If the pool bursts up to 50, as those clients disconnect, it would destroy those threads until it gets back down to 10, so you only have the min thread count sitting around doing nothing.

Also if one thread serves one connection the thread does nothing while waiting for commands and data from a sender.

Yeah, but a thread that is doing nothing but waiting on IO won't be scheduled to run by the OS. So, it really just consumes memory.

If you insist on each thread for each connection, let me know, I'll do as you want.

Yes, I insist.

I didn't review the Postal's API code yet, but pretty sure it uses standard HTTP GET/POST, which handled by Puma server and Rails. So it doesn't need to implement threads for API, Puma and Rails does all work.
Ok, I'll implement one thread for each connection.

Right, Puma handles the thread pool for the API, and here we'll need some custom code to do it.
But from the perspective of the server administrator who's just trying to run Postal and make sure that it's handling as many requests as possible and using server resources efficiently, a thread pool is a thread pool. :-)

Thanks for your efforts gents

@dsnopek thread for each connection is done. Please review.

I think the overall approach used in lib/postal/smtp_server/server.rb looks good! There's loads of code to clean up, ex. all the commented out lines, making sure all the methods work right, maybe renaming some things (run_event_loop isn't a great name anymore).
I'm about to start doing some manual testing, so I'll let you know how that goes.
Also, this PR changes a whole lot of unrelated code. I don't know why most of it is changed, but I worry that all those unrelated changes may make the maintainers less willing to merge this PR. If some of those other things necessary clean-up for other reasons, it might make sense to split those changes into their own PR, and have this PR just be about the changes needed to make the SMTP server threaded?

Manual testing has been going pretty good!
I did hit one problem when testing with 10 simultaneous connections right away:
ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.000 seconds); all pooled connections were in use

This was basically ActiveRecord using up all the connections in its connection pool, because there were more threads to use them! The old event loop based SMTP server would only use a single connection, and now we'll use a connection per simultaneous thread.
So, to fix that, I increased the pool_size to 10 under each of the database connections in postal.yml:
main_db:
  # ... other stuff ...
  pool_size: 10

message_db:
  # ... other stuff ...
  pool_size: 10

I don't really understand the difference between the two connections, so it's possible I'd only need to increase one? But for which ever is used by the SMTP server, the pool_size will need to be at least as big as the maximum threads in the pool.

The app uses separates DB to store messages. I increased only main_db's pool_size.
The Rails app uses about 5 connections. So you need setting 5 + max_threads pool_size. Also, you can set a timeout in msec.

@andrew2net Could you add some debug logging that shows information about the number of threads, and the number of items in the thread pool's job queue? It'd be helpful for my manual testing to have some insight into what's happening in the thread pool. Thanks!

@dsnopek Added a number of threads and queue length messages to log.

It's worth noting that Postal used to have a thread-per-client SMTP server as an option alongside the event driven server.
See https://github.com/atech/postal/commit/b90632b525dab624b07f7b072c50a9c8773e3a6e
I removed it because the event driven server was dramatically more performant and reliable. The event driven server aims to avoid blocking calls but clearly these are sometimes used for internal connections (ie nothing that an external client could cause to block).
Can you tell me exactly which calls you are seeing blocking? My instinct would be that you have 2 simple options to improve things here:

Use a faster database server.
Run more event driven servers (separate processes) and TCP load balance them.

This would introduce a lot less potential new problems than trying to mix threads with the event loop.

@catphish Thanks for the response! Sorry for the delay in getting back to you, I've been crazy busy.

Can you tell me exactly which calls you are seeing blocking?

I wish had some better metrics on what exactly was blocking, but unfortunately, I don't know exactly.
Looking at the code, the other potential culprit is the SSL negotiation from STARTTLS, because I know there's back and forth required to do SSL negatiation, and this appears to be done in a blocking way (a single function call, where the SSL is already negotiated when the function returns).
See https://github.com/atech/postal/blob/master/lib/postal/smtp_server/server.rb#L193
What I can say, is that we tend to get "bursts" of connections to SMTP.  However, during those bursts, the load stays super low, and the database remains responsive. Which is why the problem seems to be that the event loop is just blocking for too long and not accepting those connections.
I don't think improving database performance will help - it seems to be fine.

Run more event driven servers (separate processes) and TCP load balance them.

This is an interesting idea! That also solves the problem with the event loop based approach that multiple CPUs (or core or threads) aren't used.
However, how would someone pull this off? Is it possible currently to configure the SMTP server to run multiple instances on different ports? If I could configure it, I would try it. :-)

This would introduce a lot less potential new problems than trying to mix threads with the event loop.

I don't think @andrew2net's implementation is actually mixing event loop and threads. It's still using nio4r, but I think that could easily be replaced with a normal select() system call. If you look at your old threading implementation, it's using select in a similar way:
https://github.com/atech/postal/commit/b90632b525dab624b07f7b072c50a9c8773e3a6e#diff-259e22219520520060f5c5bff5761fc4L233
The main difference between your old implementation and @andrew2net's is that it's using a thread pool, rather than creating a new thread for each request. And the thread pool operates in an interesting way, where it'll accept and queue up connections to be handled once a thread becomes free (where other thread pool implementations might not accept a connection if there are no available threads).
Returning to something you said earlier in your comment:

I removed it because the event driven server was dramatically more performant and reliable.

I suspect that using a thread pool, rather than always creating a new thread for each incoming connection, would also show dramatic improvements in performance and reliability over the the old threading implementation. :-)

I've continued to do some testing on this. I created a simple Python script that creates 100 simultaneous threads which 10 messages each. Postal is running on a relatively resource constrained VM, so the situation is even worse than in production. Here's some of my findings...
Running against the unmodified 'master' branch without using STARTTLS:
Success:         635
Failure:         365
Min timing (ms): 1167.0
Max timing (ms): 12903.0
Avg timing (ms): 10095.3826772

The failures counted above are connect timeouts. The 'timing' is the milliseconds from starting the SMTP connection, sending message, through to disconnecting.
Running with @andrew2net's PR (again without STARTTLS) and using the default configuration (2 thread min, 10 thread max):
Success:         1000
Failure:         0
Min timing (ms): 302.0
Max timing (ms): 5463.0
Avg timing (ms): 4394.327

So, going from a few hundred timeout failures to zero seems like a pretty good improvement!
However, trying STARTTLS with this PR fails with the following error in the smtp_server.log:
SSL Negotiation Failed: read would block

@andrew2net Can you check that out?

@dsnopek I think you should use a certificate files in your test script.

@andrew2net My test script works fine with STARTTLS on the 'master' branch - I don't see why switching it to your PR means that I have to supply the certificate files in my test script? Ideally, all code that works with Postal on the 'master' branch should work unaltered with your PR.

@dsnopek I fixed the issue with STARTTLS

Seems to be working in my testing! And now for some more statistics...
Here's the 'master' branch with STARTTLS:
Success:         653
Failure:         347
Min timing (ms): 4089.0
Max timing (ms): 11882.0
Avg timing (ms): 10133.6753446

And here's this PR with STARTTLS:
Success:         1000
Failure:         0
Min timing (ms): 338.0
Max timing (ms): 4619.0
Avg timing (ms): 3824.236

Again, when using this PR, there were no connection timeouts, whereas there are several hundred on the 'master' branch.
In these results (and the previous ones), performance also seems to be quite a lot better in general with the threaded approach. I'm a little surprised that the "Min timing (ms)" is so much better - I would have thought it'd be about the same, because the first connection should have no interference in getting connected, but I guess the synchronous IO is causing slowdowns when having to multitask with multiple connections at all.
I'm going to continue to do some testing on my local test VM, and then we'll start trying these changes in production soon. We have a whole bunch of monitoring in production to detect the issues we've been having, so we should see relatively quickly whether or not this actually fixes our original issue. :-)

FYI, we just deployed these changes (on the 965d024 commit, but the following commit is just updating tests) to our production Postal instance. So far so good! I'll keep an eye on our monitoring and logs and let everyone know how it goes. :-)

Just a quick update here: things seem to be working! We have spent weeks mitigating this problem (ie. coming up with ways to smooth out our traffic or moving more stuff from SMTP to the API) so the improvement isn't quite as dramatic as it would have been had we had these changes available right way, but the problem hasn't recurred since switching this PR, so that's a good sign. :-)

Just a quick update here: things seem to be working! We have spent weeks mitigating this problem (ie. coming up with ways to smooth out our traffic or moving more stuff from SMTP to the API) so the improvement isn't quite as dramatic as it would have been had we had these changes available right way, but the problem hasn't recurred since switching this PR, so that's a good sign. :-)

Yes, I started using your PR (All 9 commits) 4 days ago and I have seen good results and improvement. I have never encounter any issue for now. In fact, there are few new features seen.
Keep up the good work bruh.

@catphish Any chance you could give this PR a second look? I responded to your original feedback in this comment https://github.com/atech/postal/pull/751#issuecomment-469425393 and did a bunch of profiling/testing afterwards which I think is pretty compelling. My test script is available, so you can see exactly how I got those results and are certainly welcome to try and reproduce them. Thanks!

Yes I'll definitely have another look at this. The concept worries me, but all reports seem to be good!

I've been using this patches in a test environment and even when it improve smtp sessions something really odd happens under certain stress (high speed/volume tests) circumstances.
Postal from time to time begin to send duplicate emails as shown below:

root@postal:/opt/postal/log# grep testmail@---- smtp_server.log
[smtp.1:6154] [2019-04-26T08:42:44.456] DEBUG -- : [JNKBLO] <= RCPT TO:<testmail@---->
[smtp.1:6154] [2019-04-26T08:42:44.457] DEBUG -- : [JNKBLO] Added external address 'testmail@----'

root@postal:/opt/postal/log# grep testmail@---- smtp_sender.log
[worker.4:18616] [2019-04-26T08:42:44.828] INFO -- : [QXKIIGXZ] Sending message 11::38756322 to testmail@----
[worker.4:18616] [2019-04-26T08:42:44.849] INFO -- : [QXKIIGXZ] Message sent #38756322 to smtp-relay.gwreactor.com (<IP_ADDRESS>) for testmail@----
[worker.4:18616] [2019-04-26T08:42:44.914] INFO -- : [QXKIIGXZ] Sending message 11::38756322 to testmail@----
[worker.4:18616] [2019-04-26T08:42:44.934] INFO -- : [QXKIIGXZ] Message sent #38756322 to smtp-relay.gwreactor.com (<IP_ADDRESS>) for testmail@----

Test environment:

1 single server running postal+rabbit+mariadb (64gb ram + 24 cpu + 100gb)
test email list (25000 emails) sent in 100 batches with 250 emails each using smtp
1 single remote smtp server using mail-sink from postfix (just for logging purposes)

I was able to confirm those duplicates in remote smtp server logs.
Maybe someone else could test different uses (batches, sessions/s, active smtp connections, etc).

I haven't seen the duplicate emails issue noted in the last comment.
But I've now twice seen an issue when using this PR in production: the queue can get "stuck", where all 10 threads are consumed doing nothing, and there's some number of items in the queue that keeps going up. The two times this happened, it was right after a big spike of messages coming through. Most of the messages did get through, but afterward, the problem started.
This is pretty serious issue, and may lead to us switching away from this PR in production, at least for a bit.

@dsnopek if a client doesn't close a connection then a thread waits until the connection is closed by timeout. I said about the issue at the beginning.

Each occupied thread should wait until the sender completes sending a message and the thread will be blocked by a slow sender or by a lost connection.

Hm. I think the timeout, whatever it is, is set way too high. In the logs I was watching, the queue was never going down and the threads were never being freed.

Could you provide the log?