Can't speed up model when pruning mT5 model

Describe the issue:
I use TaylorFOWeightPruner to prune mT5_base model, but the errors happened when speed up model.
pruner = TaylorFOWeightPruner(attention_pruned_model, ffn_config_list, evaluator, taylor_pruner_steps)
_, ffn_masks = pruner.compress()
renamed_ffn_masks = {}
# rename the masks keys, because we only speedup the bert.encoder
for model_name, targets_mask in ffn_masks.items():
renamed_ffn_masks[model_name] = targets_mask
pruner._unwrap_model()
attention_pruned_model.load_state_dict(check_point)
m_Speedup = ModelSpeedup(attention_pruned_model, (a.to(device), b.to(device), c.to(device), d.to(device)), renamed_ffn_masks)
m_Speedup.speedup_model()
optimizer = Adam(attention_pruned_model.parameters(), lr=init_lr)
logg error:
[2023-02-03 11:29:50] start to speedup the model
2023-02-03 11:29:50 - INFO: start to speedup the model
2023-02-03 11:29:54 - INFO: {}
2023-02-03 11:29:54 - WARNING: no multi-dimension masks found.
2023-02-03 11:29:54 - INFO: Dectected conv prune dim" 0
[2023-02-03 11:29:55] infer module masks...
2023-02-03 11:29:55 - INFO: infer module masks...
[2023-02-03 11:29:55] Update mask for encoder.aten::size.519
2023-02-03 11:29:55 - INFO: Update mask for encoder.aten::size.519
[2023-02-03 11:29:55] Update mask for encoder.aten::slice.521
2023-02-03 11:29:55 - INFO: Update mask for encoder.aten::slice.521
[2023-02-03 11:29:55] Slice dim:0, Slice obj:slice(0,<PHONE_NUMBER>854775807, 1)
2023-02-03 11:29:55 - INFO: Slice dim:0, Slice obj:slice(0,<PHONE_NUMBER>854775807, 1)
[2023-02-03 11:29:55] Model has Slice operation, and the operand size=torch.Size([8, 10]), Slice object:(slice(0,<PHONE_NUMBER>854775807, 1),)
2023-02-03 11:29:55 - INFO: Model has Slice operation, and the operand size=torch.Size([8, 10]), Slice object:(slice(0,<PHONE_NUMBER>854775807, 1),)
[2023-02-03 11:29:55] Model has Slice operation, and the operand size=torch.Size([8, 10]), Slice object:(slice(0,<PHONE_NUMBER>854775807, 1),)
2023-02-03 11:29:55 - INFO: Model has Slice operation, and the operand size=torch.Size([8, 10]), Slice object:(slice(0,<PHONE_NUMBER>854775807, 1),)
[2023-02-03 11:29:55] Update mask for decoder.aten::size.808
2023-02-03 11:29:55 - INFO: Update mask for decoder.aten::size.808
[2023-02-03 11:29:55] Update mask for decoder.aten::size.809
2023-02-03 11:29:55 - INFO: Update mask for decoder.aten::size.809
[2023-02-03 11:29:55] Update mask for decoder.aten::slice.825
2023-02-03 11:29:55 - INFO: Update mask for decoder.aten::slice.825
[2023-02-03 11:29:55] Slice dim:0, Slice obj:slice(0,<PHONE_NUMBER>854775807, 1)
2023-02-03 11:29:55 - INFO: Slice dim:0, Slice obj:slice(0,<PHONE_NUMBER>854775807, 1)
[2023-02-03 11:29:55] Model has Slice operation, and the operand size=torch.Size([8, 10]), Slice object:(slice(0,<PHONE_NUMBER>854775807, 1),)
2023-02-03 11:29:55 - INFO: Model has Slice operation, and the operand size=torch.Size([8, 10]), Slice object:(slice(0,<PHONE_NUMBER>854775807, 1),)
[2023-02-03 11:29:55] Model has Slice operation, and the operand size=torch.Size([8, 10]), Slice object:(slice(0,<PHONE_NUMBER>854775807, 1),)
2023-02-03 11:29:55 - INFO: Model has Slice operation, and the operand size=torch.Size([8, 10]), Slice object:(slice(0,<PHONE_NUMBER>854775807, 1),)
[2023-02-03 11:29:55] Update mask for decoder.aten::slice.833
2023-02-03 11:29:55 - INFO: Update mask for decoder.aten::slice.833
[2023-02-03 11:29:55] Slice dim:0, Slice obj:slice(0,<PHONE_NUMBER>854775807, 1)
2023-02-03 11:29:55 - INFO: Slice dim:0, Slice obj:slice(0,<PHONE_NUMBER>854775807, 1)
[2023-02-03 11:29:55] Model has Slice operation, and the operand size=torch.Size([8, 10]), Slice object:(slice(0,<PHONE_NUMBER>854775807, 1),)
2023-02-03 11:29:55 - INFO: Model has Slice operation, and the operand size=torch.Size([8, 10]), Slice object:(slice(0,<PHONE_NUMBER>854775807, 1),)
[2023-02-03 11:29:55] Model has Slice operation, and the operand size=torch.Size([8, 10]), Slice object:(slice(0,<PHONE_NUMBER>854775807, 1),)
2023-02-03 11:29:55 - INFO: Model has Slice operation, and the operand size=torch.Size([8, 10]), Slice object:(slice(0,<PHONE_NUMBER>854775807, 1),)
[2023-02-03 11:29:55] Update mask for encoder.aten::view.520
2023-02-03 11:29:55 - INFO: Update mask for encoder.aten::view.520
[2023-02-03 11:29:55] WARNING: throw some args away when calling the function "view"
2023-02-03 11:29:55 - WARNING: throw some args away when calling the function "view"
[2023-02-03 11:29:55] WARNING: throw some args away when calling the function "view"
2023-02-03 11:29:55 - WARNING: throw some args away when calling the function "view"
[2023-02-03 11:29:55] Update mask for encoder.aten::unsqueeze.522
2023-02-03 11:29:55 - INFO: Update mask for encoder.aten::unsqueeze.522
[2023-02-03 11:29:55] Update mask for decoder.aten::view.810
2023-02-03 11:29:55 - INFO: Update mask for decoder.aten::view.810
[2023-02-03 11:29:55] WARNING: throw some args away when calling the function "view"
2023-02-03 11:29:55 - WARNING: throw some args away when calling the function "view"
[2023-02-03 11:29:55] WARNING: throw some args away when calling the function "view"
2023-02-03 11:29:55 - WARNING: throw some args away when calling the function "view"
[2023-02-03 11:29:55] Update mask for decoder.aten::arange.811
2023-02-03 11:29:55 - INFO: Update mask for decoder.aten::arange.811
Traceback (most recent call last):
File "attention_pruned_model.py", line 446, in
main(config)
File "attention_pruned_model.py", line 387, in main
m_Speedup.speedup_model()
File "/home/dhnan/anaconda3/envs/nemo/lib/python3.8/site-packages/nni/compression/pytorch/speedup/compressor.py", line 546, in speedup_model
self.infer_modules_masks()
File "/home/dhnan/anaconda3/envs/nemo/lib/python3.8/site-packages/nni/compression/pytorch/speedup/compressor.py", line 383, in infer_modules_masks
self.update_direct_sparsity(curnode)
File "/home/dhnan/anaconda3/envs/nemo/lib/python3.8/site-packages/nni/compression/pytorch/speedup/compressor.py", line 237, in update_direct_sparsity
_auto_infer = AutoMaskInference(
File "/home/dhnan/anaconda3/envs/nemo/lib/python3.8/site-packages/nni/compression/pytorch/speedup/infer_mask.py", line 80, in init
self.output = self.module(*dummy_input)
File "/home/dhnan/anaconda3/envs/nemo/lib/python3.8/site-packages/nni/compression/pytorch/speedup/jit_translate.py", line 245, in call
result = self.func(*self.positional, **self.keyword)
TypeError: arange() received an invalid combination of arguments - got (Tensor, pin_memory=bool, device=torch.device, layout=NoneType, dtype=NoneType), but expected one of:

(Number end, *, Tensor out, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)
(Number start, Number end, *, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)
(Number start, Number end, Number step, *, Tensor out, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)

Could you please tell me how to solve it?
Thanks very much!
Environment:

NNI version: 2.10
Training service (local|remote|pai|aml|etc):
Client OS:
Server OS (for remote mode only):
Python version: 3.8.0
PyTorch/TensorFlow version: Pytorch 1.13.0
Is conda/virtualenv/venv used?: conda
Is running in Docker?: No

Configuration:

Experiment config (remember to remove secrets!):
Search space:

Log message:

nnimanager.log:
dispatcher.log:
nnictl stdout and stderr:

How to reproduce it?:

Hello, would u please provide the details of ur model?

I have been working on this issue recently. However, I encountered another bug during ModelSpeedup(). Let me reply to you as soon as possible!


fix(api-server-lib): Fix /server/restart failing in api-server-lib

Overview
When /server/restart is provided by api-server-lib, there's a bug that was fixed in the
update-server version of /server/restart: a new thread has no asyncio event loop running and a new
one must be created.
Testing
The issue can be seen when the robot fails to restart when /server/update is hit with current edge running on system 3.0:
From Resin logs, immediately after hitting restart
 main  Exception in thread Thread-1:
 main  Traceback (most recent call last):
 main    File "/usr/local/lib/python3.6/threading.py", line 916, in _bootstrap_inner
 main      self.run()
 main    File "/usr/local/lib/python3.6/threading.py", line 864, in run
 main      self._target(*self._args, **self._kwargs)
 main    File "/data/packages/usr/local/lib/python3.6/site-packages/ot2serverlib/endpoints.py", line 134, in __wait_and_restart
 main      loop = asyncio.get_event_loop()
 main    File "/usr/local/lib/python3.6/asyncio/events.py", line 676, in get_event_loop
 main      return get_event_loop_policy().get_event_loop()
 main    File "/usr/local/lib/python3.6/asyncio/events.py", line 584, in get_event_loop
 main      % threading.current_thread().name)
 main  RuntimeError: There is no current event loop in thread 'Thread-1

The fix is to switch to creating a new event loop, and after updating to the fixed server lib wheel and restarting via resin, the restart now succeeds.
From resin logs, immediately after hitting restart with new version
Killing application 'registry2.resin.io/v2/230045d40d12596d526cd5c71518f9bc'
Application exited 'registry2.resin.io/v2/230045d40d12596d526cd5c71518f9bc'
Killed application 'registry2.resin.io/v2/230045d40d12596d526cd5c71518f9bc'
Installing application 'registry2.resin.io/v2/230045d40d12596d526cd5c71518f9bc'
Installed application 'registry2.resin.io/v2/230045d40d12596d526cd5c71518f9bc'
Starting application 'registry2.resin.io/v2/230045d40d12596d526cd5c71518f9bc'
Started application 'registry2.resin.io/v2/230045d40d12596d526cd5c71518f9bc'

Optimistically merging as this is a copy of the implementation in Update Server that is already tested


[ V1.9.0 ] Testing

@irinelenache, @tudor-themeisle I have launched some new features for Logo Maker:

Page lazy loading ( when you go to a page for the first time, you will see an intermediary animation - a blinking text )
Slider for scaling the logo

I want you to test if the app is working ok, especially on Apple products.

Augusto and I tested this and on his Mac is working but on my windows machine is not. Here is a video of the behavior in my case ( https://www.loom.com/share/494a0034d3fb43beb08e189bbffbcf00 )

@tudor-themeisle , sorry that I forgot to mention that the testing is only for https://themeisle.com/logo-maker/logo-wizzard/#/start
LogoMaker is a final product and not a plugin like Otter to be used by folks in WordPress. You need to test only the instance that is hosted on our site.

Ok, I understand, sorry to @Soare-Robert-Daniel.
Tested on my windows machine, on explorer browser, and I got this https://prnt.sc/116ein5. I put the link, access it, click Start Designing Your Logo, and it goes here ( https://themeisle.com/logo-maker/logo-wizzard/#/ ). If I put that link in chrome, it works fine, I can start creating the logo.

No need to worry about Internet Explorer.

@Soare-Robert-Daniel tested it and found some things:

[ ] The slider doesn't work properly on macbook, both chrome and safari. In the video the slider doesn't move as the cursor (the video hides my cursor) and if i just click it, it goes back with 0.5x
https://www.loom.com/share/4caac725a5af4580b51d56c65f98d7e8

[ ] The font that is present in the font size select options is different than what is used for the other elements
https://prnt.sc/117xtd6

[ ] Similar to first, but it happens in the color selector:
https://www.loom.com/share/17557dba9573460694c83713783a3d94

Tested it with an Iphone 11 too:

[ ] The slider leaves some blue lines where it was stopped
https://user-images.githubusercontent.com/67897586/114142385-8fc2b100-991b-11eb-85a7-0029fa9931eb.jpeg and it remains selected even if i stopped using it. After i click somewhere else, it appears like this
https://user-images.githubusercontent.com/67897586/114142752-fd6edd00-991b-11eb-8d34-8065a9f12272.jpeg

[ ] In the color selector, when i try sliding to find a color, the page is scrolling (made a video on browserstack, i can try to make a screen recording on the phone if you want)
https://www.loom.com/share/30879142a8b0452885e025ecfeb25bd2

I saw the scrolling problem is present in android too, but tested only on browserstack

@irinelenache, I have made some changes to the scroll behavior and the font family.
The version is posted on the stagging website. Also, for the iOS test, is it possible to test on a real device? BrowserStack worked sluggishly on iOS.
CSS problems for sliders are not a priority at the moment.
LINK TEST STAGING: https://staging-frontend.themeisle.com/logo-maker/logo-maker-editor/#/creator

@Soare-Robert-Daniel i marked as done the issues that are fixed. tested with macbook pro - macOS Big Sur and iPhone 11 - iOS 14.4.2 (not browserstack)

@irinelenache, I have refactored the size's slider. This should solve the problem.
LINK TEST STAGING: https://staging-frontend.themeisle.com/logo-maker/logo-maker-editor/#/creator

@Soare-Robert-Daniel The slider works fine now. I don't know if this is relevant but i couldn't access the link on Iphone 11/Safari until i cleared the website data


mangal: 3.14.2 -> 4.0.1

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/metafates/mangal/releases.
meta.description for mangal is: "A fancy CLI app written in Go which scrapes, downloads and packs manga into different formats"
meta.homepage for mangal is: "https://github.com/metafates/mangal"
meta.changelog for mangal is: ""
Updates performed

Golang update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found 4.0.0 with grep in /nix/store/x9822fy6nda2qh78j1jxabqwav6rd4fs-mangal-4.0.0
found 4.0.0 in filename of file in /nix/store/x9822fy6nda2qh78j1jxabqwav6rd4fs-mangal-4.0.0

Rebuild report (if merged into master) (click to expand)

1 total rebuild path(s)

1 package rebuild(s)

First fifty rebuilds by attrpath
mangal

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/x9822fy6nda2qh78j1jxabqwav6rd4fs-mangal-4.0.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A mangal https://github.com/r-ryantm/nixpkgs/archive/1769c0c8b5a7777063a7bac1079a8779c8535ee4.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/x9822fy6nda2qh78j1jxabqwav6rd4fs-mangal-4.0.0
ls -la /nix/store/x9822fy6nda2qh78j1jxabqwav6rd4fs-mangal-4.0.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

mangal

Maintainer pings
cc @bertof for testing.

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/metafates/mangal/releases.
meta.description for mangal is: "A fancy CLI app written in Go which scrapes, downloads and packs manga into different formats"
meta.homepage for mangal is: "https://github.com/metafates/mangal"
meta.changelog for mangal is: ""
Updates performed

Golang update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found 4.0.1 with grep in /nix/store/jgvsy73i3fdiy0g2pd24l2n1w8x3x15p-mangal-4.0.1
found 4.0.1 in filename of file in /nix/store/jgvsy73i3fdiy0g2pd24l2n1w8x3x15p-mangal-4.0.1

Rebuild report (if merged into master) (click to expand)

1 total rebuild path(s)

1 package rebuild(s)

First fifty rebuilds by attrpath
mangal

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/jgvsy73i3fdiy0g2pd24l2n1w8x3x15p-mangal-4.0.1 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A mangal https://github.com/r-ryantm/nixpkgs/archive/b78bc38c1c05f26711b7682523826394289b50c5.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/jgvsy73i3fdiy0g2pd24l2n1w8x3x15p-mangal-4.0.1
ls -la /nix/store/jgvsy73i3fdiy0g2pd24l2n1w8x3x15p-mangal-4.0.1/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

mangal

Maintainer pings
cc @bertof for testing.


[AutoPR cdn/resource-manager] add premium_chinaCdn to CDN sku

Created to sync https://github.com/Azure/azure-rest-api-specs/pull/5897

(message created by the CI based on PR content)

This PR has been merged into https://github.com/Azure/azure-sdk-for-python/pull/4759


Old version(s) of SpFx Extension Code Displays Intermittently

Category

[ ] Question
[ ] Typo
[x] Bug
[ ] Additional article idea

Expected or Desired Behavior
Sites will display the latest version of a SpFx extension after the latest version is deployed and the app is updated on the site.
Observed Behavior
Functionality intermittently changes between the latest versions and prior versions of the SpFx extension.  Clearing local browser cache does not seem to have an effect.  Clearing the session may fix the issue temporarily.  Navigating through contents of the modern site almost always causes the extension to regress to a previous version
I figured this was a CDN caching issue at first.  After several days with no changes to the SpFx extension we are still seeing an old version intermittently
Steps to Reproduce

Start with a Modern site collection which has multiple pages/libraries
Deploy a SpFx Application extension to SharePoint using the CDN instructions at https://dev.office.com/sharepoint/docs/spfx/extensions/get-started/hosting-extension-from-office365-cdn.  In this case my application displays some content in the "Top" placeholder
Make updates to the code base and deploy a new version of the SpFx app.  Update the version of the app on the site
Click through the site and watch the extension change functionality between different versions of the SpFx app.  Clear browser cache and experience the same
Wait several days (or a weekend) for the CDN to presumably flush the cache.  Experience the same issue.

Can you include a fiddler trace of a working and non working request (ie 1 where you get an old version and one where you get a new version)?  If you would like to email it to grahamc at Microsoft we'll be able to take a look at this.

@mcmynn83 I sent the traces your way.  Thanks for your response.

This might be related 876. In this case, office CDN is not even used, SP is serving randomly old published versions of the JS file stored in Style Library. Also recorded video about it.

This week we are getting intermittent 406 errors on SpFx Extension loading (below).
{"HttpStatus":"NotAcceptable","ErrorMessage":"Requested asset is not found or is not located in the Public CDN enabled location","ErrorCode":406012}

@mcmynn83 Any status update?  In the meantime, do you have any suggestions for workarounds?

had a similar error message "No 'Access-Control-Allow-Origin' header is present" a full browser cache reset fixed the issue in my case

@pdemro Are you still experiencing this issue?

Recently experienced the same issue.
Hitting F5 sometimes loads old scripts from CDN and sometimes from app catalog (we changed CDN base url to app catalog client assets library). Sometimes it's not loading at all - error 406.
We've discovered it was caused by "*" in webpart manifest version.
{
    "$schema": "https://developer.microsoft.com/json-schemas/spfx/client-side-web-part-manifest.schema.json",
    "id": "<Your_Id>",
    "alias": "<Your_Alias>",
    "componentType": "WebPart",
    "version": "1.0.1", // used to be * (star)
    "manifestVersion": 2,
    "requiresCustomScript": false,
    ...
}

Can anybody confirm this actually causes it? Thanks.


Highlighted resources

Admins should have the ability to choose which resources appear on the front page
[ ] Add highlight flag to resource model and display that list on home page
[ ] Allow admin to upload picture to resource to use as highlight image OR choose from a list of 10-20 pre-existing options.
[ ] Gray out existing photo option if it is already in use on a highlighted resource so that there are never duplicate images on the front page
[ ] Add highlight status as a weight in search so that highlighted resources appear higher in search results

In consultation with the team we have opted for highlighting currently enrolling studies on the home page, so this feature is no longer needed at this time.


feat: add resource_class parameter for docker executor

Add resource_class parameters for docker executor.
According to the official documentation, if you do not specify a size, resource_class will be set automatically.

For new projects created after September 1, 2021 that do not specify a resource class, CircleCI will try to find the right default value for your organization. To avoid using a default, explicitly specify a resource class size in your config for each job.
https://circleci.com/docs/2.0/configuration-reference/#resourceclass

I want to add an option so that I can select resource_class size.

Hello @kynefuk Apologies for the day. I would like you add your change but I believe it may be best to use a generic string rather than an enum in this case, so that there is the potential for users of CircleCI Runner to specify their resource class as well. As this branch is now out of date as well (on us, sorry about that), I will take it on to re-incorporate your change in a new PR if that is alright with you.
I will close this PR for now, if you would like to collaborate on the new changes please feel free to let me know. I expect we'll see these changes live in about 2 weeks max.

@KyleTryon
Thanks!
I'll make a new PR based on your comments!


Enhance API to query the runtimes available

This is related to #618
We support the --kind for action to specify the runtime version like nodejs:6
As we add more runtimes we could have multiple nodejs versions (i.e. 6.2.1, 6.3.4, 7.0.1, etc..)
After implementing #618
It will be useful to have a client API to query the available runtimes versions for all languages
It can be part of the object already return in /v1/info or a new API /v1/info/runtimes
Related to #547
This will allow client side libraries or tools (client-js, CLI) to query runtime kinds and present to the user.
cc @sjfink @jasonpet

+1 on including it in /api/v1. Or go super fancy: Include it in OPTIONS /api/v1/namespaces/_/actions

+1 on including it in /api/v1. Or go super fancy: Include it in OPTIONS /api/v1/namespaces/_/actions

I'm fine with either as long whiskswagger.json get's updated 😹

This is actually related to #335 which goes a step further and also enumerates limits. Close this as dup? @csantanapr

@markusthoemmes yes we can close as dup as long we copy the requirements and add them to #335

closing as duplicate of #335.


[FR] integration for home assistant

Hello I find your project very nice neat and useful. But I'm using hassos which does not allow me to install packages.but with integration it would work. Are U interested in making an integration out of this ? Would be awesome ☺

I am not sure how to create an app for HASSOS, but theoretically it could be installed on this OS because it's a Linux system.
I've never used it, so I don't know how to do this integration.
Maybe I could create a home assistant integration in the future so I will leave this feature request open.

This has been done.


add daily, monthly conus404 to osn

Closes #304

@rsignell-usgs - all CONUS404 datasets are on the OSN pod now. I can't remember who you said needed to use them, once you merge this PR, you can pass the correct intake catalog entries on to them.


Problems when losing kafka connection

I created a minimal example for a kafka consumer. Maybe I am doing something wrong, but in case the application loses the connection to kafka the consumer does not come up again properly.
Here is my minimal example:
https://github.com/XOfSpades/minimal_brod_example
Steps to reproduce:

start kafka (maybe adopt the port)
produce a message which will create a console output of the message
restart kafka (e.g. with a docker-compose down && docker-compose up -d)
produce again a message

The last message won't be shown.
I already recognized that the main process of brod seems to come back again:
iex(1)> GenServer.whereis(:brod_kafka_base_server)
#PID<0.244.0>

If this is anyhow important I am using:

Elixir 1.7.2
docker image confluentinc/cp-kafka:4.1.0
docker image confluentinc/cp-zookeeper:4.1.0

Takk for hjelped :-)

btw I used a one node kafka here

Hi @XOfSpades
could you provide some logs especially the restart logs after kafka restart.
e.g. consumer group partition assignment logs

your application is down and I can't seem to find the start log:

14:18:41.973 [info] Application minimal_brod_example exited: shutdown

you'll need to restart the group subscriber after bord client is back.
in practice, you may want to manage your brod_group_subscriber worker under supervisor3
which can be configured to retry restart indefinitely.

Thank you. Found it


IT stays in a white screen

IT stays in a white screen .
ubuntu 18.04
it was all okay until today

I am facing the same issue as well.

It works with this solution:
https://github.com/klaussinani/tusk/issues/308


do not remove container after stop

Is there a way to not remove the container after it is stopped?

Heya!
This could be easily a command line flag! what is your use case for this?

@yuvipanda and @snizzleorg I'm going to close this. I'm happy to reopen if asked. Thanks.


python requirements missing in master

I was following guide for manual installation but when trying to run
pip install -r LiuAlgoTrader/liualgotrader/requirements/dev.txt
I got error that file does not exist, apparently the folder "requirements" is not present in master.

Oh It was moved to pdm, sorry i didn't catch that.


Printing on Star TSP100III

Hi, I have the wifi version of this printer but despite the discovery is working, printing is not. What should I do to make printing work?
Using your discover_printers project example code I can find the printer but when I tap to print the demo receipt nothing is printed although it says success on the snack bar and the printer ready light blinks once.
I also tried to comment all text, image and barcode lines leaving just feed and paper cut trying to test if any command would work but nothing happened.
I appreciate if you can give me some direction on printing with your package to this printer.

Hi,
Check this please:
https://github.com/andrey-ushakov/esc_pos_printer/issues/57#issuecomment-602114078
You probably need to adjust the dip switches and choose the ESC/POS mode.

Hi Andrey, thanks for your reply.
As I could figure out the TSP100iii can switch to ESC/POS mode through software, i.e., some command you send to it.
In the manual below, I found how to configure it to ESC/POS, but I also realized that it is something that works only for computer software as it saves on the driver that all printing will be using ESC/POS.
https://www.starmicronics.com/faq/Resources/FaqDocuments/131/1203a_StarMicronics_ESCPOS_TSP100.pdf
Following these steps changed nothing on the app when trying to print, and there is no switch like on #57 comment on this model.
https://www.starmicronics.com/absolutefm/absolutefm/afmviewfaq.aspx?faqid=72
Is there some kind of command that we have to send on every printing that will inform that it is ESC/POS mode?

@RosilBelli
since the software. required to change the TSP100 print mode is a Star produced software, I do not see a feasible way to have this package inform the star printer that it is to accept ESC/POS commands.
As suggested in the thread of #57 , Profiles would seem like the best way to go.
If this is something with enough demand, perhaps we open a ticket / feature for adding profiles?
As far as I can tell (more research required), the profiles would simply be a mapping of the ticket methods, to the proper byte structure.
Specifically for StarMicronics, we would have a class which would convert the Ticket bytes according to the information found in this manual
@andrey-ushakov What are your thoughts on this? Should we open a ticket and begin some preliminary work? I would benefit greatly from a StarMicronics profile, and would be eager to do much of the work myself.

Profiles describe a printer's capabilities. They are not converters. Some examples could be found here
esc_pos_printer was created to generate the ESC/POS byte code. It shouldn't convert it to Star format either.
Instead, the most natural way would be to create a new dart/flutter package that will generate Star bytecode. It's also technically possible to create a separated package to convert ESC/POS bytecode to Star bytecode but it's harder to do and I think there are fewer use cases.

@andrey-ushakov Gotcha!
I need to get Start compatibiliy in the next day or two. So with that being the case, I will be trying to build a StarMicronics printer package tomorrow.
It would be really cool to have a single package that is something like receipt_printer that could grow, and eventually have the ability to communicate and work with most all printers out there.
And with tree-shaking built in, the extra code would not cause bloating for those apps who only use one or two features.

Hi @AirborneEagle ! This's really good news, and yes, Star Printers are really very used on the market.

@AirborneEagle Hi, i have exactly the same problem, but i have usb Star printer TSP143IIIU and should make it work in next few days. if you have any progress in this please let me know, probably here.

Hey all.
here is a fork/branch where the utils now support Star emulation.
This is a breaking change seeing that it adds an EmulationType parameter to the ticket constructor. but other than that, it is all the same
it is working for me so far though. although I have not yet put in the star commands for printing images, or barcodes etc. but the basics work. (text, alignment, bold, underline, cut, feed, columns, etc.)
side note, @andrey-ushakov  What was the motivation behind splitting this package into 3 repos?

Hi,
do you have any news to merge the branch created by AirborneEagle ?
Thanks in advance for you reply,

@AirborneEagle  I tried using your fork for Star TSP100III printers. Though my code is able to establish connection with the printer,  the actual print doesn't work. Are there any additional settings needed to get this working ?


OptionGroups types definition may be incorrect

Reproducible in:
Version Bolt 3.12.2
Node.js runtime version: v14.19.0, Typescript target: ES2020
Steps to reproduce:
The type definition for OptionGroups does not appear to match the Slack documentation for OptionGroups. The type definition defined in types/options/index.ts requires the label to be a string, whereas the official API appears to require label to be a PlainTextElement.
My specific use case is an external_select within a dialog:
// Modal dialog served to the user
{
  "type": "modal",
  "callback_id": "someCallbackId",
  "private_metadata": "",
  "title": { "type": "plain_text", "text": "Choose an External Option", "emoji": true },
  "submit": { "type": "plain_text", "text": "Choose", "emoji": true },
  "close": { "type": "plain_text", "text": "Cancel", "emoji": true },
  "blocks": [
    {
      "type": "input",
      "block_id": "someBlockId",
      "label": { "type": "plain_text", "text": "Event Selection" }
      "element": {
        "type": "external_select",
        "action_id": "someActionId",
        "min_query_length": 0,
        "placeholder": {
          "type": "plain_text",
          "text": "Choose an Option"
        }
      }
    }
  ]
}

import { App } from '@slack/bolt';
const app = new App();

// Bolt middleware to provide options to the dialog above
app.options('someCallbackId', async ({ ack }) => {

  // Example option_groups copied directly from Slack API documentation
  // @see: https://api.slack.com/reference/block-kit/composition-objects#option_group
  const option_groups = {
    option_groups: [
      {
        label: {
          type: 'plain_text',
          text: 'Group 1'
        },
        options: [
          {
            text: {
              type: 'plain_text',
              text: '*this is plain_text text*'
            },
            value: 'value-0'
          },
          {
            text: {
              type: 'plain_text',
              text: '*this is plain_text text*'
            },
            value: 'value-1'
          },
          {
            text: {
              type: 'plain_text',
              text: '*this is plain_text text*'
            },
            value: 'value-2'
          }
        ]
      },
      {
        label: {
          type: 'plain_text',
          text:'Group 2'
        },
        options: [
          {
            text:{
              type:'plain_text',
              text:'*this is plain_text text*'
            },
            value:'value-3'
          }
        ]
      }
    ]
  };

  await ack(option_groups);
});

The contents of option_groups is copied directly from the option_groups documentation.
Expected result:
The typescript project compiles with no errors.
Actual result:
await ack(option_groups); yields the following typescript error:
Argument of type '{ option_groups: { label: { type: string; text: string; }; options: { text: { type: string; text: string; }; value: string; }[]; }[]; }' is not assignable to parameter of type 'BlockOptions | OptionGroups<BlockOptions>'.
  Type '{ option_groups: { label: { type: string; text: string; }; options: { text: { type: string; text: string; }; value: string; }[]; }[]; }' is not assignable to type 'OptionGroups<BlockOptions>'.
    Types of property 'option_groups' are incompatible.
      Type '{ label: { type: string; text: string; }; options: { text: { type: string; text: string; }; value: string; }[]; }[]' is not assignable to type '({ label: string; } & BlockOptions)[]'.
        Type '{ label: { type: string; text: string; }; options: { text: { type: string; text: string; }; value: string; }[]; }' is not assignable to type '{ label: string; } & BlockOptions'.
          Type '{ label: { type: string; text: string; }; options: { text: { type: string; text: string; }; value: string; }[]; }' is not assignable to type '{ label: string; }'.
            Types of property 'label' are incompatible.
              Type '{ type: string; text: string; }' is not assignable to type 'string'.ts(2345)

Attempts to use a label of type string rather than type { type: string; text: string; } (PlainTextElement) results in no options being displayed to the user.
Requirements
For general questions/issues about Slack API platform or its server-side, could you submit questions at https://my.slack.com/help/requests/new instead. :bow:
Please read the Contributing guidelines and Code of Conduct before creating this issue or pull request. By submitting, you are agreeing to those rules.

Hi @npnicholson, thank you so much for taking the time to report this issue. You are right that this is a type bug in this SDK. We'll fix it in the next release.


Incomplete documentation: admin permissions are required to connect to a BitBucket Cloud repo

admin permissions are required for the account used to connect to the BitBucket Cloud repo, even it the pipeline does not use PR and CI integrations:
trigger: none
pr: none

[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: cf09cc7f-cd4d-6647-e47b-82744bd3d2dd
Version Independent ID: 75e80c27-0e2e-41d1-272f-e3da856f5071
Content: Build Bitbucket Cloud repositories - Azure Pipelines
Content Source: docs/pipelines/repos/bitbucket.md
Product: devops
Technology: devops-cicd-repos
GitHub Login: @vijayma
Microsoft Alias: vijayma

Updated the doc to state that user needs admin access to Bitucket since that identity is used to register webhook. This is required since we always register a webhook even if you do not configure a ci or pr trigger in the YAML file. Azure Pipelines needs to inspect the YAML file after every change since that change can enable triggers in the yaml file.


Order transaction error: Cannot read property 'currency' of undefined

For transaction 04DB1D743926DF6F2C3855C95FBB877186F55E9A3B25B1B491F55CB84594FC88 the REST server returns the error below.
{
  "success": false,
  "error": "Cannot read property 'currency' of undefined",
  "error_type": "transaction"
}

While https://ripple.com/build/websocket-tool/#tx shows an apparently valid transaction.
Is this a problem with the Ripple REST server?

Looking into it

This should fix it, https://github.com/ripple/ripple-rest/pull/385

Yes, that has fixed it. Thanks very much.


support search issues pagination

This adds pagination support to search results, next PR will include template changes to allow for UI to utilize this pagination.

Stale?

After finished this PR, I realize we should not support pagination for a search engine. The next page maybe better.

For anyone coming to this PR in the future as in the linked issue, this PR is not suitable for pagination due to other issues.

I'm writing a new PR which will replace this one and try to fix #24662


Tiles dont show up on watch

Hi, I tried to build a quick demo app using your library and I am struggling using the tiles. I followed your docs and to confirm this is what my code looks like:
# main.dart
import 'package:flutter/material.dart';
import 'package:wear_plus/wear_plus.dart';
import 'package:yak/tile.dart';
import 'package:wearthat/tiles.dart' as wear_tiles;

@pragma('vm:entry-point')
void maintile() {
  wear_tiles.runTiles(
    [
      wear_tiles.TileService(
        name: "WeatherTile",
        routes: {
          '/': () => WeatherTile(),
          // If needed more routes
          // '/secondary': () => SecondaryWeatherTile(),
        },
      ),
    ],
  );
}

void main() {
  runApp(const MyApp());
}
...

#tile.dart
import 'package:wearthat/tiles.dart';
import 'package:weather/weather.dart';

Future<Weather> getWeather() async {
  WeatherFactory wf = WeatherFactory('<KEY>');
  Weather w = await wf.currentWeatherByLocation(55, 15);
  return w;
}

class WeatherTile extends Tile<Weather> {
  @override
  Future<Weather> createState() {
    return getWeather();
  }

  @override
  TileWidget build(TileContext context, Weather? state) {
    return Column(
      children: [
        SizedBox(
          width: 32,
          height: 32,
          child: Image("loading_weather"),
        ),
        Text("I will show my weather here, if i want to...DONT PRESSURE ME !"),
      ],
    );
  }

  @override
  Map<String, TileResourceProvider> resources(
          TileContext context, Weather? state) =>
      {
        'loading_weather': TileResources.asset("assets/img.png"),
      };
}

pubspec.yaml
...
tiles:
  WeatherTile:
    label: Quick Weather
    preview: assets/img.png

I also ran the dart command to update the tile.
❯ dart run wearthat:synctiles
Building package executable...
Built wearthat:synctiles.

Clearing tiles...
  Removing tiles from manifest
  Removing tiles classes
  Removing tile previews

Adding tiles...
  Loading tiles from pubspec.yaml
  Added WeatherTile

Done in 0:00:00.000050.

When I run on the emulator however I cannot see the tile. My app starts but there is no custom tile there, only the default ones. Am I doing something wrong?

Quite embarrassingly it seems that I was just forgetting to actually add the widget on the watch after creating it :facepalm: This can be closed

Will open an issue to add the "Add widget on the watch" step.
Glad you figured it out 👍


Is  Elasticsearch-Loader-Setup for Snowplow Depricated in favour of Rdb loader

I am trying to implement the Es loader , but confused if it has been depricated in favour of rdb loader

Hey, please redirect your support requests to our Discourse, GitHub is only for code-related issues.


WeTEE Network

Project Abstract
WeTEE is a decentralized application deployment platform integrated with a Trusted Execution Environment (TEE).
WeTEE consists of blockchain networks and multiple confidential computing clusters, collectively providing an efficient decentralised solution for confidential computing.
Thread-level confidential container service providers need to provide hardware devices that support Intel SGX, and they are required to use IPv4 / IPv6 to access the Internet.

At this stage, Substrate is mainly used as the blockchain framework to implement application deployment and billing-related smart contract functions.
After the application is deployed, its workload will be safeguarded by hardware protection to prevent data leakage. Even confidential computing providers will not be able to access the data.
It integrates with the existing cloud native toolchain for developers, typically requiring no code modifications, and in special cases, only a small amount of code modification is needed.
Developers can view information such as the resource usage and health status of applications in the Web interface provided by WeTEE.

Grant level

[ ] Level 1:  Up to $10,000, 2 approvals
[x] Level 2:  Up to $30,000, 3 approvals
[ ] Level 3:  Unlimited, 5 approvals (for >$100k: Web3 Foundation Council approval)

Application Checklist

[x] The application template has been copied and aptly renamed (project_name.md).
[x] I have read the application guidelines.
[x] Payment details have been provided (bank details via email or Polkadot (USDC & USDT) or BTC address in the application).
[x] The software delivered for this grant will be released under an open-source license specified in the application.
[x] The initial PR contains only one commit (squash and force-push if needed).
[x] The grant will only be announced once the first milestone has been accepted (see the announcement guidelines).
[ ] I prefer the discussion of this application to take place in a private Element/Matrix channel. My username is: @_______:matrix.org (change the homeserver if you use a different one)

I have read and hereby sign the Contributor License Agreement.

Thank you very much, I studied Acurast's and integritee's solutions very carefully. I feel our solution is different in a few ways:

Our solution is more like an Amazon cloud-like solution. We hope users can deploy their original web2 programs to WeTEE without modifying code, changing programming languages, or altering user habits. Users can run programs like mysql, golang webserver, VUE/REACT web apps, etc. This is an aspect that existing solutions have not covered yet.
Also, we are focused on B-end users. We provide governance mechanisms similar to openGov for each organization, enabling decentralized organizations to better manage web2 programs.

@Noc2  Thank you for pointing that out. We have done some further research on these two projects and have come to the following conclusions:
According to the description of the Processors section in the official Acurast documentation:

Acurast Processors offer their computational capacity to Consumers. In return for the confidential execution and verifiable fulfillment of jobs, processors are rewarded by the Consumers. At this stage, processors utilize dedicated Android smartphones as the off-chain infrastructure behind the Acurast Secure Hardware Runtime (ASHR). Become a processor within minutes.

Acurast uses the processors of Android devices at the hardware layer and provides the Acurast Secure Hardware Runtime (ASHR). Currently, WeTEE adopts the TEE of Intel SGX/AMD SEV, which is for x86 architecture devices, and the application runtime of WeTEE is a self-developed product.
According to the description of the Application Layer section in the official Acurast documentation:

Acurast's Zero Trust architecture transforms the way applications are designed and deployed.

According to the description in this part of the Acurast documentation and the use cases provided in the Acurast Console, after our preliminary attempt, we found that Acurast primarily provides JavaScript Functions to acquire Web2 interface data and synchronize cross-chain data. The computation demand of such JavaScript Functions is typically measured in seconds, and the operational stress on physical devices is relatively low, so Acurast's overall computational model is more similar to the AWS Lambda service. This indirectly verifies why Acurast can choose the mobile SoC as its computing unit.
As for WeTEE's use of x86 architecture devices, after achieving decentralized privacy computing technologically, the types of applications it serves are more "universal" than those of Acurast. This is also why WeTEE supports C, Python, Go, Rust, Javascript, and all program codes that support Gramine.
From a technical reuse perspective, Zero Trust architecture is a very good security solution. WeTEE will also attempt to integrate the zero-trust solution with existing solutions in the future. However, since Acurast's business attributes greatly differ from those of WeTEE, we are currently not considering reusing this project's code.
Like Integritee, WeTEE has adopted a solution based on TEE. However, WeTEE is not only a highly scalable, privacy-enabling network in the Polkadot ecosystem, but it has also built a decentralized application deployment infrastructure based on decentralized servers.
According to the information provided by the Integritee official document Sidechain SDK, Custom Business Logic / STF, and the integritee-worker repository code, application developers must use its SDK to build applications when developing applications in Integritee. The business code will also be programmed in Rust, and it needs to integrate with Integritee's Sidechain. During the application deployment phase, the TEE environment of Integritee comes from its subsidiary company, Securitee. Securitee's infrastructure is hosted in the open, convertible, scalable, and reliable cloud infrastructure of OVHcloud, which is Europe's leading cloud provider.
WeTEE adopts different technical routes and strategies for privacy computing. Application developers can use their preferred programming languages for application development without the need for excessive code modification. WeTEE's TEE environment is provided by miners and matches application deployment requests with TEE computing nodes based on an algorithm.
While WeTEE and Integritee's worker are similar in terms of business logic, the worker of WeTEE is developed in Golang. Reusing the code related to Integritee's worker would introduce unnecessary business code modules and may cause some degree of project delay.

Project
Technical Solutions
Programming Languages
Execution Hardware
Supported Programming Languages

WeTEE
• Kubernetes and Docker as computing cluster solutions • Gramine or Ego as confidential container solutions• Worker with TEE（run as K8s Operator） provides sgx remote attestation, key management, program Confidentiality Injection and uploads sgx remote attestation as part of proof of work to blockchain network consensus.
golang
x86 Server with Intel SGX
C, Python, Go, Rust, Javascript and all program codes that support Gramine

Integritee
Woker with TEE provides Sidechain、Off-Chain Worker、Oracle as confidential solutions
rust
x86 Server with Intel SGX
Rust（Fork integritee worker and add own Rust code）

Acurast
Acurast  Zero-Trust Execution Layer as End-to-End Zero Trust Job Execution as confidential solutions
rust
mobile device
Javascript

@takahser  Thanks for your response
Q1: Can you talk a bit more about how your governance model will work? For example, how will you incentivise people to vote?
WeTEE aims to provide a decentralized application deployment platform that integrates a trusted execution environment for decentralized applications, allowing the deployment environment of applications to gradually break free from the constraints of centralized data centers.
And as a startup product, there will inevitably be a huge gap with the current mature Web2 public cloud, which also means that there is a huge amount of development work and demand for developers. Looking back at the growth process of the public cloud, the development roadmap of each public cloud’s cloud business is centered around the needs of various applications in the market environment at that time for the cloud, but the demand side of the application is mostly large customers of the public cloud, and the development priority of the public cloud functions is also tilted towards large customers.
This also means that some functions with problems in the public cloud, but with less impact on large customers, cannot be promptly fixed or improved. Functions of this type, which have high demand among public cloud users but are not in line with the interests of public cloud vendors, may not even receive development support.
And the WeTEE DAO organization hopes to change this situation, with the following main strategies:

Make WeTEE's R&D more community-driven and transparent
The Core Team is mainly responsible for the development of the WeTEE base, but will open the authority to modify functions to the community for voting.

The WeTEE DAO hopes to evolve naturally and form an open Domain Driven Design pattern under the DAO model, where each domain can collaborate and develop relatively independently, promoting the progress of WeTEE together.
The overall style of the WeTEE DAO will practice the K.I.S.S. principle, so the current organizational structure of the WeTEE DAO is as follows:

Core Group: WeTEE team, composed of R&D, capital, and marketing personnel
Advisory Group: external assistance team, composed of advisors, sponsors, and partners
R&D Group: composed of individuals who submit product suggestions, code, and designs to WeTEE.

The expected result after a period of natural evolution is the birth of two groups:

Domain: naturally formed research domains in WeTEE
Domain core: naturally formed opinion leaders in the domain, consisting of one person or multiple people.

WeTEE is organized and operates in the form of DAO, which has greater advantages in financial and development transparency, allowing developers and service providers to establish trust with the project more quickly.
In terms of details, WeTEE has created sudo, gov, guild, project, asset, treasury, and other modules based on Substrate for on-chain governance. We have also created the DTIM tool  based on the Matrix protocol for work collaboration and organizational governance.
In order to make the community more active, WeTEE DAO has adopted the following strategies:

Attract more like-minded people through the Social DAO approach, increasing community activity.
Create on-chain governance tools through Substrate, as well as instant messaging and work collaboration and organizational governance tools specifically designed for Web3, DTIM, simplifying the voting process and reducing participation difficulty.
Each group and main project adopts an internal governance approach, allowing participants to focus on their own areas.

In order to attract more developers to participate in the development of WeTEE, WeTEE DAO has adopted the following strategies:

Individuals who contribute new features or bug fixes to WeTEE will receive rewards.
Individuals engaging in white hat activities and submitting them to WeTEE will receive rewards.
Individuals who involved in adapting applications for WeTEE will receive rewards.
Personnel who become core in their domain will be rewarded.
If an application is integrated or used by other WeTEE applications, the original developer will be rewarded.

In order to attract more developers to participate in the voting of WeTEE, WeTEE DAO has adopted the following strategies:

The workflow, on-chain parameters, and addition or deletion of functions of WeTEE can all be adjusted through voting.
Regular proposal discussion meetings or conferences will be held to
allow members to express their opinions and make suggestions.
Each vote must be preceded by a reasonable discussion.
Voting itself will set a certain participation threshold, but the threshold will not be high.
For a passed vote, the proposer of the vote will receive rewards such as identity and credibility value.
For a passed vote, voters who voted in favor will receive rewards.
WeTEE will try to judge the actual value of a vote based on the actual final result generated by its content, and feedback adjustments to the "Fair Points" of voters, which will affect their rewards.
"Fair Points" will influence the growth of the voter's identity. Upon reaching a certain accumulation threshold, they can apply to join the Core Team.

Q2: When moving a web2 project to your platform, what adjustments are necessary, if any?
We currently support two encryption schemes, with the following deployment scenarios:

If the user does not use any tool for container migration, but directly deploys to the wetee platform, the program will run in normal mode, but there will be no protection ofconfidential computing on the program and data.
If developers use golang to develop programs, they need to follow the ego document to use ego-go to compile the program, package it into a docker image, and then run it on the platform. Almost no code needs to be changed.
If developers use other programming languages, they can use Gramine’s Docker image conversion tool, Gramine Shielded Containers (GSC), to automatically convert their programs into confidential containers. Developers do not need to perform any additional operations, and most programs do not require code modification.

Q3: Can web2 users deploy their apps on your K3/K8 systems or are you using K3/K8 only to run the mn b platform in the background, thus requiring web2 users to deploy their apps differently?
Web2 users can directly deploy programs on the K3/K8 cluster connected to WeTEE. Users can use Wetee in the same way as they use k8s/k3s. In the future, we will also provide one-click deployment tools, where users only need to execute one command or click one button to complete the deployment. We will also provide decentralized CI/CD services, allowing users to modify the code and then deploy it with just one click.
Q4: Are there any other technical limitations to be expected when compared to using web2 infrastructure, e.g. performance?
Different confidential schemes will have some limitations.

Ego programs will consume about 10% more machine performance than normal golang programs, and there are also some limitations.
Gramine programs also consume about 10% more machine performance than normal lower programs, and there is an official machine performance adjustment document.

Q5: It'd be nice if you could include one or two demo apps in the milestones that can be directly deployed on your infrastructure. Doesn't have to be anything sophisticated, a hello-world type app would do, just as a proof-of-concept that and how web2 integrations with your platform work.
In Milestone 2 Delivery  "01 App Example" will provide a program example that can be run directly on wetee.
Because we adopted two relatively mature solutions, Ego and Gramine, in theory, all Gramine cases and Ego cases can be run directly on our platform.
Q6: How does your solution compare to ankr?
According to the official documentation provided by Ankr, as stated in Ankr Docs:

Ankr is the leading Web3 infrastructure company. It has a set of different products for building, earning, gaming, and more — all on blockchain.

Its main functions are as follows:
RPC Service

RPC Service (https://www.ankr.com/rpc/) — a platform that provides access to our top-class nodes infrastructure to query the vast list of supported chains, monitor requested data telemetry, and test the RPC API methods you require before actually using them.

Ankr Staking

Ankr Staking aims to bring the benefits of DeFi to the masses with Liquid Staking, Delegated Staking, DeFi, Bridge, Switch, and Parachain Crowdloan.

From the main structure of Ankr, Ankr is a Hub that integrates and encapsulates various blockchain RPC/REST APIs. Application developers can quickly develop applications by calling the APIs provided by Ankr. For business data that needs to be persisted in the application, you can choose IPFS or STORJ for storage according to actual situations.
This makes Ankr closer to a SaaS-like platform that provides integrated services related to blockchain, and provides gateway functions for blockchain-related APIs and storage services. However, the official documentation of Ankr and STORJ does not explain the distribution of their servers, nor do they clearly explain how they manage their physical servers, so it is impossible to speculate how their backend business processes customer business data.
Therefore, based on the above two points, the main differences between WeTEE and Ankr are as follows:

For application development, developers have complete autonomy over the development paradigm of their applications and are not necessarily limited to API-oriented development.
WeTEE not only supports the deployment of blockchain-based DApps, but also supports the deployment of Web2 applications.
WeTEE strives to be closer to native K3S/K8S during the application deployment phase.
WeTEE adopts a technical approach and strategy for privacy computing, allowing application developers to use their familiar programming languages and development paradigms without having to modify code excessively.
The TEE environment of WeTEE is provided by miners and matched with application deployment requests and TEE computing nodes based on algorithms, making it more transparent compared to Ankr.

From a business scenario perspective, Ankr is more focused on areas such as Gamefi/Defi/parallel chain construction, where users are currently unable to freely deploy applications. Moreover, from the perspective of project operation, Ankr adopts a corporate operation model, which is the same as the public cloud company model mentioned in the first question, and thus has the same potential problems and risks.
Q7: Expanding on the future roadmap beyond the grant would give more confidence in the long-term vision.
When you express interest in the future development of WeTEE, our team is deeply encouraged, as your support leads us to believe that our code and products will achieve meaningful and valuable results.
This grant will allow us to focus on developing core features, laying a reliable business foundation for WeTEE. Upon reviewing the outcomes of our submitted grant, you will be able to easily verify WeTEE's core business using the documentation guidance and Docker examples provided by WeTEE.
The R&D of WeTEE can be divided into three major stages, each of which contains several small R&D stages or iterative cycles:

Core R&D stage
User experience optimization stage
Ecosystem construction stage

During the current core development stage, WeTEE will concentrate all research and development resources on WeTEE's own development, including on-chain workers, apps, tasks, as well as K3S/K8S operators relevant to physical servers, app deployment models, task deployment models, and worker attestation models.
Once this part is completed, WeTEE will spend a small amount of time to reorganizing the code and conducting a retrospective on the completed work, after which the development will move into the next phase of core research and development.
At this stage, the R&D content mainly includes the WeTEE test/main network, the WeTEE Dapp SDK, accessing the Polkadot mainnet using Coretime, as well as the WeTEE monitoring system and the WeTEE Web user interface.
After completing this stage of development, WeTEE will invite seed users to conduct usability testing and user acceptance testing, and operational work will be carried out in accordance with the requirements of WeTEE DAO. Following initial user feedback, targeted fixes and optimizations will be implemented to address any issues within WeTEE.
The following will enter the third phase of core R&D, which is also the last core R&D phase in the current planning. In this phase, WeTEE will dynamically allocate R&D resources to the ‘blockchain-related’ or ‘hardware-related’ fields based on user and market feedback.

Blockchain-related:

Integrate https://github.com/paritytech/frontier and support wallet applications such as Metamask
Improve the multi-language version of the SDK to help users better integrate accounts and business between Web2 apps and Web3 DApps
Improve the decentralized collaboration system of WeTEE, allowing WeTEE users to view the status of application clusters and application deployment events at any time

Hardware-related:

Compatibility with AMD SEV confidential solution
Compatibility with Intel TDX confidential solution
Performance optimization of confidential computing, distributed storage, and network for server K3S/K8S.

@BurnWW thanks for your very detailed and helpful answers.
I have a few final questions, but I'm already going to mark your proposal as ready for review.

Ego programs will consume about 10% more machine performance than normal golang programs, and there are also some limitations.
Gramine programs also consume about 10% more machine performance than normal lower programs, and there is an official machine performance adjustment document.

While the Gramine doc mentions "1-10% overhead", I didn't find such information for Ego. Did you get this number (10%) from their resources or did you evaluate it yourself, e.g. by testing it?

If the user does not use any tool for container migration, but directly deploys to the wetee platform, the program will run in normal mode, but there will be no protection ofconfidential computing on the program and data.

Does that mean that anybody in the network can theoretically read their data?

If developers use golang to develop programs, they need to follow the ego document to use ego-go to compile the program, package it into a docker image, and then run it on the platform. Almost no code needs to be changed.
If developers use other programming languages, they can use Gramine’s Docker image conversion tool, Gramine Shielded Containers (GSC), to automatically convert their programs into confidential containers. Developers do not need to perform any additional operations, and most programs do not require code modification.

Would it be possible to maintain a "soft-fork mentality" that doesn't require the user to do any changes on their containers (I.e. no hard fork of the relevant code, so to speak), while still being protected? While I'm not sure if it's technically possible, this would be the ideal scenario because it'd significantly lower the barriers of entry for web2 users. I imagine that converting their containers or using golang with ego would discourage a lot of web2 teams. My personal experience shows that these kind of conversion tools often are not as simple as advertised (not in the containerisation space in particular, but in general).

Q1:While the Gramine doc mentions "1-10% overhead", I didn't find such information for Ego. Did you get this number (10%) from their resources or did you evaluate it yourself, e.g. by testing it?
During the WeTEE technology selection phase, we carried out evaluations across multiple dimensions such as hardware types, operating systems, virtualization, encryption methods, performance overhead, software ecosystem, and user learning costs.
Ultimately, we decided to adopt libos + K3S/K8S + Intel SGX (AMD SEV) as the architectural direction for the WeTEE technology stack.
The libos, including Gramine and Ego as mentioned, strikes a balance between the ease of use for application developers and the performance requirements of WeTEE.

libos is an operating system design pattern that moves the implementation of application program interfaces (APIs) from the operating system kernel to user space libraries.
The goal of this design pattern is to provide a more flexible and lightweight operating system solution by reducing the functionality of the operating system kernel and offering higher-level application program interfaces to improve performance and applicability.

We have selected several different yet representative applications to run on the WeTEE technical stack and observed the performance overhead of the applications. Ultimately, we arrived at performance overhead conclusions similar to those in the Gramine official documentation, namely an overhead of "1-10%".
According to our analysis, the performance overhead can be roughly divided into two categories: the overhead brought by libos and the overhead brought by cryptographic computations. The performance overhead is also influenced by various factors such as CPU model, operating system kernel version, system component versions, and current system load. Of course, different types of applications also exhibit significant differences in performance overhead.
Gramine or Ego, acting as libos, provides applications with a consistent interface compared to the kernel by intercepting and simulating system calls (syscalls), allowing applications to run on libos without awareness. Leveraging the interface abstraction layer provided by libos, similar to glibc or musl, applications can directly perform the required syscall operations in the system user space, resulting in minimal performance overhead.
The primary impact on performance comes from the overhead of Intel SGX and the type of system calls required during application runtime. Due to the differences in applications, in general, the performance loss of single-process programs in SGX mode ranges from 5% to 10%. This aligns with the results of our testing after migrating applications to Gramine and Ego.
To test performance overhead, we used a simple program implemented in Golang that calculates prime numbers up to 10,000.
package main

import (
	"fmt"
	"time"
)

func main() {
	firstDate := time.Now()
	defer func() {
		fmt.Println("Time-consuming：", time.Since(firstDate))
	}()
	// Task Queue Channel
	intChan := make(chan int, 1000)
	// The output channel, all the calculation results are placed here.
	primeChan := make(chan int, 2000)
	// Exit marked pipe
	exitChan := make(chan int, 4)
	// Distribute tasks
	go putNum(intChan)
	// Start four coroutines to calculate prime numbers and put them into the result channel.
	for i := 0; i < 4; i++ {
		go cal(intChan, primeChan, exitChan)
	}
	// Start coroutine, continuously get end flag from exitChan, when the number of acquisitions reaches 4
	go closeWork(primeChan, exitChan)
	// The main thread traverses the result set in range.
	for i := range primeChan {
		fmt.Println(i)
	}
	fmt.Println("Traversal ends")
}

/*
*
PutNum：Coroutine is responsible for putting all the numbers that need to be calculated into the intChan channel. Note: After everything is put in, close the intChan channel so that consumers don’t get stuck in an infinite loop when iterating through it with for-range
*/
func putNum(intChan chan int) {
	for i := 1; i <= 100000; i++ {
		intChan <- i
	}
	close(intChan)
}

/*
*
Determine whether all work coroutines have ended. If they have ended, close primeChan to notify the main thread.
*/
func closeWork(primeChan chan int, exitChan chan int) {
	for i := 0; i < 4; i++ {
		<-exitChan
	}
	close(primeChan)
	close(exitChan)
}

/*
*
for-range loop traverses intChan and calculates whether it is a prime number. For-range will traverse to the closed unknown of the channel. When the range loop ends, put an identifier into exitChan indicating that the current coroutine has ended
*/
func cal(intChan chan int, primeChan chan int, exitChan chan int) {
	for v := range intChan {
		flag := true
		for i := 2; i < v; i++ {
			if v%i == 0 {
				flag = false
				break
			}
		}
		if flag {
			primeChan <- v
		}
	}
	exitChan <- 0
}

Compile and run using Golang:
### Program Compilation
$ go build -o wetee_prime_test_native wetee_prime_test.go

### Run
$ ./wetee_prime_test_native

Compile and run using Ego with the same source code:
### Program Compilation
$ ego-go build wetee_prime_test.go

### Sign
$ ego sign wetee_prime_test

### Run
$ ego run wetee_prime_test

Our test hardware is as follows:
### CPU
Intel(R) Core(TM) i7-9700F

### Operating System
ubuntu 22.04

### Golang version
go1.21.0 linux/amd64

### Ego version
EGo v1.4.1 (8b99356398dd3bcb5f74e5194d20ce421f607404)

The test results are as follows:

test
go
ego
overhead

1
861.689517ms
887.994298ms
3%

2
871.182404ms
879.994342ms
1%

3
869.130757ms
863.994439ms
-1%

4
819.968487ms
875.994353ms
6%

5
854.097488ms
887.994268ms
3%

6
779.094409ms
819.9947ms
5%

7
865.685818ms
879.994304ms
1%

8
869.595235ms
879.994298ms
1%

9
790.125787ms
875.994316ms
10%

10
864.461084ms
895.99418ms
3%

Q2: Does that mean that anybody in the network can theoretically read their data?
If the application itself provides unprotected data interfaces (APIs) or program listening ports, then anyone in the network can retrieve data through these interfaces or ports. However, if we understand it correctly, your question should mainly focus on the following scenario :

When a third party (an individual or the owner of a physical server in the network) illegally accesses the data in the container through technical means, what are the disposal measures of WeTEE?

WeTEE primarily focuses on container security by implementing defense measures in two aspects: remote access and physical access.
WeTEE primarily focuses on container security by implementing defense measures in two aspects: remote access and physical access.

Remote Access to Physical Machines :
◦ WeTEE provides network security reinforcement guidelines to block privileged port access.
Remote Access to K3S/K8S or Containers :
◦ The Agent will set RBAC for the local K3S/K8S to manage user and service account access to resources.
◦ Fine-grained control of Pod network traffic is achieved through Network Policies.
◦ Container data protection is implemented in reference to Pod Security Standards provided by K8S.
◦ Each Pod is assigned an appropriate ServiceAccount to restrict its access to other resources in the cluster.
◦ Miners are prompted and guided to update K3S/K8S when necessary.
Remote Access to Applications :
◦ Inbound requests are only allowed for the specific ports designated during application deployment.
Physical Access to Servers :
◦ WeTEE provides physical server security reinforcement guidelines, guiding miners to implement basic security measures for physical servers.
◦ Periodic system checks are conducted, and miners are prompted to update the system to mitigate potential zero-day vulnerabilities.
◦ At the current stage, the Agent must run in a TEE environment with hash verification.
◦ Applications running in the TEE environment encrypt their data, rendering it inaccessible to third parties.

Security reinforcement settings are relatively complex, and due to space limitations, it is not possible to provide detailed elaboration. In summary, data protection for applications running in non-TEE environments relies on the data protection policies inherent to K3S/K8S, while applications in TEE environments benefit from strict data protection measures.
Furthermore, WeTEE continuously tracks relevant CVE events to further optimize security reinforcement settings.
Q3: Would it be possible to maintain a "soft-fork mentality" that doesn't require the user to do any changes on their containers (I.e. no hard fork of the relevant code, so to speak), while still being protected? While I'm not sure if it's technically possible, this would be the ideal scenario because it'd significantly lower the barriers of entry for web2 users. I imagine that converting their containers or using golang with ego would discourage a lot of web2 teams. My personal experience shows that these kind of conversion tools often are not as simple as advertised (not in the containerisation space in particular, but in general).
Currently, a large number of Intel SGX-enabled hardware assets require support. In this regard, Gramine stands out as a promising libos, providing an abstract interface layer for glibc/musl without intrusive modifications to programs. This allows developers to smoothly migrate existing applications to the Intel SGX hardware platform. The emergence of Gramine offers developers a more convenient way to leverage the advantages of Intel SGX hardware, easing the process for them.
Meanwhile, we will continue to optimize the functionality of WeTEE, continuously improving and reducing the difficulty for developers in application migration.
• In the first stage
WeTEE will provide base images with Gramine based on Ubuntu Server / Rocky Linux. Here is an example of the base image code using Ubuntu Server.
ARG UBUNTU_IMAGE=ubuntu:20.04

FROM ${UBUNTU_IMAGE}

# ARGs cannot be grouped since each FROM in a Dockerfile initiates a new build
# stage, resulting in the loss of ARG values from earlier stages.
 An example is shown below.
FROM wetee/ubuntu:22.04
// Developer-defined workflow
.....

// run
gramine-sgx redis-server

Developers can complete the construction of confidential containers without needing to focus extensively on the details of Gramine. Additionally, developers have the option to directly use Gramine Shielded Containers (GSC) to generate the image.
• In the second phase
WeTEE will provide decentralized image building services, enabling developers to upload their code to Github or image compilation nodes. The system will then perform automated image building based on the Dockerfile. This initiative will liberate developers from the tedious task of image building, allowing for fully automated application deployment and upgrades.
Once WeTEE provides support for AMD SEV and Intel TDX, users will no longer need to modify their code, prepare Dockerfiles, or worry about compatibility issues. SEV and TDX, in the form of confidential virtual machines, provide a confidential computing environment for programs.

@takahser Thanks for your response  We have updated the descriptions of all Milestones.

@BurnWW thanks for another very helpful reply. I can see that you have a good plan here, and you seem to have thought of everything and since you made the deliverables clearer, I'm happy to approve it. The decentralized image building services you mention for the next phase sound promising as well - if it works, that could help driving adoption to this kind of platform significantly. BTW, another question that popped up in my mind is, whether current K8 Tooling would be compatible with your platform, e.g. ArgoCD, OpenShift, etc.

@takahser Thanks for your response.
In the grant application, the following business was mentioned:

WeTEE incorporates an intelligent compiler that possesses auto code analysis capabilities. This compiler can deeply parse the codes of Web2 applications and astutely identify the compilable sections related to business logic. WeTEE will extract these parts of code and automate its compilation into TEE application containers.
When application developers need to update their applications to provide new features or security patches, they can fully leverage the hot update mechanism provided by WeTEE for rapid application updates.

To achieve this business goal, the WeTEE team has researched common CI/CD software in the current market environment, such as those listed in List of Continuous Integration Services. These are all excellent works of software engineering, but in order to meet WeTEE's demand for ease of use for application developers, we are still looking for an appropriate open-source CI/CD solution, striving to strike a balance between lightweight and ease of use for WeTEE.
Currently, the requirements of WeTEE for CI/CD are as follows:

Interchangeability: The CI/CD should be lightweight and atomically replaceable.
Encryption: CI/CD needs to run in the TEE environment of WeTEE.
Ecological Compatibility: CI/CD system needs to have a well-designed API for easy invocation by K3S/K8S and WeTEE.
Component Independence: Minimize invasive modifications of the K3S/K8S system by the CI/CD.
Security: CI/CD needs to have certain access control policies, which can achieve
security reinforcement through configuration modification.

When you mentioned ArgoCD, we decided to put ArgoCD at the top of our CI/CD candidate list because currently we know that ArgoCD runs well in TEE environments.

The core R&D of WeTEE's CI/CD is expected to begin in the 'User experience optimization stage':
The R&D of WeTEE can be divided into three major stages, each of which contains several small R&D stages or iterative cycles:

Core R&D stage
User experience optimization stage
Ecosystem construction stage

The function of CI/CD is expected to be provided in sync with the WeTEE main network, WeTEE Dapp SDK, and WeTEE monitoring system and WeTEE web user interface.
Sorry for clicking the wrong close button just now. It's been corrected.


Use CURLOPT_CUSTOMREQUEST properly

Fixes #227
This PR implements the proper use of CURLOPT_CUSTOMREQUEST. Apparently, Curl does not like when you set CUSTOMREQUEST directly for POST and HEAD requests. For example, setting CUSTOMREQUEST=POST will cause POST redirects to keep using POST instead of switching to GET like they should.
Similar issues exist for HEAD request as well.

I'm using 5.16.6 on 'Ubuntu and it still keeps POST after redirects.

@justtb can you please make another issue for this?


[SPARK-31020][SQL] Support foldable schemas by from_csv

What changes were proposed in this pull request?
In the PR, I propose to replace matching by Literal in ExprUtils.evalSchemaExpr() to checking foldable property of the schema expression.
Why are the changes needed?
This should improve Spark SQL UX for from_csv. Currently, Spark expects only literals:
spark-sql> select from_csv('1, 3.14', replace('dpt_org_id INT, dpt_org_city STRING', 'dpt_org_', ''));
Error in query: Schema should be specified in DDL format as a string literal or output of the schema_of_csv function instead of replace('dpt_org_id INT, dpt_org_city STRING', 'dpt_org_', '');; line 1 pos 7

Does this PR introduce any user-facing change?
Yes, after the changes users can pass any foldable string expression as the schema parameter to from_csv(). For the example above:
spark-sql> select from_csv('1, 3.14', replace('dpt_org_id INT, dpt_org_city STRING', 'dpt_org_', ''));
{"id":1,"city":" 3.14"}

How was this patch tested?
Added new test to CsvFunctionsSuite.

Test build #119231 has started for PR 27771 at commit df02a09.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23970/
Test PASSed.

@HyukjinKwon @cloud-fan Please, take a look at this PR. It is kind of follow up of https://github.com/apache/spark/pull/22666/files#r228785835

Test build #119231 has finished for PR 27771 at commit df02a09.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119231/
Test FAILed.

Test build #119239 has started for PR 27771 at commit a72568e.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23978/
Test PASSed.

Test build #119244 has started for PR 27771 at commit e95e8a7.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23983/
Test PASSed.

Test build #119239 has finished for PR 27771 at commit a72568e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119239/
Test FAILed.

Test build #119244 has finished for PR 27771 at commit e95e8a7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119244/
Test PASSed.

Oh, I just noticed you open separate PRs for JSON and CSV. Let's merge because they will likely be reverted or merged together with the same reason and same review. The changes are not too big either.

@dongjoon-hyun WDYT of mixing all changes into one PR?

+1 to put them together. They are the same fix, for different functions.

Let's put it back together. Piggy-backing isn't the standard approach, of course but this one specifically will be discussed, reverted or merged together logically. I think it's fine to combine.

Let's just close :-).


Drag and Drop Image for Editing

I would like to add a drag and drop functionality. I want users to be able to drag an image to the canvas for editing. It'll make the app more convenient to use. Please assign me this task. Thank you

@Tolu-Mals You can work on it.


Uses okio/moshi natively in Elasticsearch everywhere

Before, we used a combination of our buffer and Okio's Buffer, and a
combination of our json encoder and Moshi's. This changes all the code
in Elasticsearch to use Okio buffers which reduces the amount of array
allocations and allows us to do cheap content hashes in the future.
The content hashes will be used to obviate duplicate document indexing,
for example, when reporters backoff and retry with the same span. This
will help simplify analysis.

I ran a 5m load test against before and after. As it is, this doesn't make a big difference, likely because by default under load senders will produce 3MiB or so json per message. This will still result in big arrays. OTOH, we now have a chance to optimize.
the left hand side is before and the right hand side is with this change. The test is brave-webmvc-example against zipkin+elasticsearch 6.7 for 5 minutes

the above load test result was before MD5 was added.. I'll run it again

with hashing the load is up a bit from the base case (based on my not entirely scientific scenario). I will try to buy back some performance from elsewhere to pay for the hashing.

ok fixed some memory overhead

OK I'm personally happy with this. the line corresponding to GC time (100ms) and heap used (<100MiB) is stable throughout the load (8.5Kspans/second indexing into ES in my laptop)


feat(variables): link store variables data & refactor dropdown

To Reviewers

[ ] Skip (style, chore ONLY)
[ ] Not that difficult

Type of Change

[x] New feature
[x] Bug fixes
[x] Feature improvement
[x] Refactor
[ ] Others (performance improvement, refactoring, CI/CD, etc.)

Checklist

[x] Error / Warning / Lint / Type

Description
SSIA

Link store data.
Refactor dropdown, now dropdown accesses store directly without using props from customize page.
Not that difficult other changes.

Thank you for reviewing illegible code..!
I reflect your reviews.


Cert manager stuck at attempting to acquire leader lease

Describe the bug:
Have been running with cert manager for a year now, but suddenly, it is being stuck at "attempting to acquire leader lease", with no error messages.  It is working fine on a similar cluster, with the same settings.
Environment details::

Kubernetes version (e.g. v1.10.2): 1.11.8-gke.6
Cloud-provider/provisioner (e.g. GKE, kops AWS, etc): GKE
cert-manager version (e.g. v0.4.0): v0.6.2
Install method (e.g. helm or static manifests): helm

/kind bug

After reinstall of helm chart, and some patience, then it worked.


Add support for S3 resources

S3 resources are comprised of a bucket name, and an optional object key (or wildcard): see https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-arn-format.html
I chose to expose bucket name and object key as text values instead of using amazonka's BucketName and ObjectKey, since this lib does not depend on amazonka.

Correction: for consistency with the existing Lambda module, we'd have one Network.AWS.ARN.S3 module, which contains a Bucket and an Object type, along with their corresponding parse/unparse functions and their prisms.
Please also do a cabal outdated check, update changelog, and add @since annotations to the things you add (should be <IP_ADDRESS>).

(i left the @since annotations out, since i imagine those would be added upon release). Will add them.
Something strange with the nix setup is that nothing seems to be in the path in the dev shell, just haskell-ci

Correction: for consistency with the existing Lambda module, we'd have one Network.AWS.ARN.S3 module, which contains a Bucket and an Object type, along with their corresponding parse/unparse functions and their prisms.

That's what i did at first, but reading the S3 ARN doc, i felt the model was really a single S3 resource entity, with the object key part being optional. I can separate the two use-cases, but i think having the (bucket x Maybe object) representation is useful

In case this is useful: I was working with a function yesterday that accepted as input a ARN S3Resource and required an object ARN. The function had to have a failure mode for when it was passed a bucket ARN; it wouldn't have needed that if we could specify that it needed an object ARN.

We should debug your Nix trouble at some point, because my nix develop is definitely downloading and building the other libraries needed here.
The problem experienced by @lrworth is exactly why I want to distinguish bucket and object ARNs. We could provide functions in Network.AWS.ARN.S3 to make moving between them a little easier (lens can lift them to working on ARNs instead of just the resource-part):
objectInBucket :: Text -> Bucket -> Object
containingBucket :: Object -> Bucket

Do you think they'd be useful? I'm not wedded to the names.

Yeah, it's always good to have faithful representations anyway. I can handle it when I'm back from vacay

I think i have handled everything

Couple more nits.
I was thinking maybe we provide functions to parse/render Either Bucket Object, but they're probably easy enough to write using existing tools and instance Alternative Maybe, so maybe they're beneath the Fairbairn Threshold?

that was my initial plan, but yeah before writing them i realized that would be one-liners that did not use anything specific to this lib, so better handled at use site.

(i ran cabal outdated, but it didn't find anything out of date)


Unused dimension 0 or 1

We were looking at the nifti specification to know if we should write
dim = [N, w, h, d, 0, 0, 0]
OR
dim = [N, w, h, d, 1, 1, 1]

but it's not entirelly clear! The standard says
dim[i] = length of dimension #i, for i=1..dim[0]  (must be positive)

Is zero positive or negative? Such is the question :) A quick research returns a math.stackexchange page. It seems that there a difference between "positive" and "strickly positive", but the standard doesn't use the clear term "strickly positive"... 0 seems to be both "positive" and "negative" AND neither, depending on who you ask, the context and the position of Venus.
What do we have in nifti-rs?

The default NiftiHeader creates a dim [1, 0, 0, 0, 0, 0, 0]
The Dim and Idx structs both receive a [u16; 8] without modifying it. It's validated for its length and negative values, but not for unused values. It currently accepts [3, 100, 100, 100, 0, 42, 1].

What do we do with this? It's  not a bug, nifti-rs still works as intended, but... do we "fix" it?

Unless I overlooked something, I think that we would be safe to keep using dim as we are now. Here is my rationale:

That snippet from the standard only specifies dim[i] where i is in a "valid" dimension range, which is from 1 to dim[0] inclusive. So an unused dim[6] would not need to be positive.
The official FAQ, in Q4, states that at least dim[0] and dim[1] are required, by their words. It continues with dim[2] being required for images and dim[3] being required for volumes. As one cannot omit writing or reading any bytes in dim, they ought to mean that setting non-required parts to some value is not necessary.
This would clash with the default value of dim in NiftiHeader, but to be fair, this information can only be fully specified alongside a volume. This default set up is mostly done for convenience. The header can still be malformed in other ways anyway. It has been up to reader and writer functions to validate the header.

I believe that was originally intended in NIfTI-1 from the start was that implementers:

Must only read the dimensions in dim from dim[i] up to dim[0], and reading beyond that is a bug that could lead to ill-formed behavior;
May choose to initialize the remaining unused values to 0 or 1, but only for convenience. Even the header file at some point suggests that "It is best to initialize ALL fields in the NIFTI-1 header to 0 (e.g., with calloc()), then fill in what is needed."

We are good with the first rule, and for the creation of fresh new volumes we should be good with the second one too. In a similar fashion, it's faster to just copy a [u16; 8] into a Dim or Idx without cleaning up the unused values. The only exception I can think of from the second rule would be if we stumble upon a... creative implementation that relies on these unused parameters to be set to something reasonable. Should something like that come around, we would then assess the implications of making the library more compatible with that implementation.

Ok, perfect, thank you for your input. It was useful at least to discuss it and to make things clearer.


Fix/Verify the TLS half close behavior in s2n

S2N does not handle half close connections.
S2n marks the conn->closed = 1; if s2n_connection_recv_stuffer returns 0.
https://github.com/awslabs/s2n/blob/master/tls/s2n_recv.c#L63
the s2n send functions does not send any data if conn->closed is set to 1.
https://github.com/awslabs/s2n/blob/master/tls/s2n_send.c#L93
As a result, half close TLS connections are not processed like TCP half close connections. i.e. if a client sends a http request and performs half close by closing the write side of the socket, the server running s2n would stop sending the HTTP response once it reads the EOF.
As per TLS protocol,
https://tools.ietf.org/html/rfc5246#section-7.2.1
"
The client and the server must share knowledge that the connection is
ending in order to avoid a truncation attack. Either party may
initiate the exchange of closing messages.
close_notify
This message notifies the recipient that the sender will not send
any more messages on this connection. Note that as of TLS 1.1,
failure to properly close a connection no longer requires that a
session not be resumed. This is a change from TLS 1.0 to conform
with widespread implementation practice.
Either party may initiate a close by sending a close_notify alert.
Any data received after a closure alert is ignored.
Unless some other fatal alert has been transmitted, each party is
required to send a close_notify alert before closing the write side
of the connection. The other party MUST respond with a close_notify
alert of its own and close down the connection immediately,
discarding any pending writes. It is not required for the initiator
of the close to wait for the responding close_notify alert before
closing the read side of the connection.
If the application protocol using TLS provides that any data may be
carried over the underlying transport after the TLS connection is
closed, the TLS implementation must receive the responding
close_notify alert before indicating to the application layer that
the TLS connection has ended. If the application protocol will not
transfer any additional data, but will only close the underlying
transport connection, then the implementation MAY choose to close the
transport without waiting for the responding close_notify. No part
of this standard should be taken to dictate the manner in which a
usage profile for TLS manages its data transport, including when
connections are opened or closed.
Note: It is assumed that closing a connection reliably delivers
pending data before destroying the transport.
"
What would be the ideal behavior for half close connections in s2n?

TCP half-close can happen when a client sends a FIN, but the server still has data to send.
So the server finishes sending it's response, then sends it's own FIN and completes the
closure. For example:
(SYN, SYN-ACK, ACK)
client: GET index.html HTTP/1.1
client: FIN                     (client done writing)
server: ACK                     (acknowledge the FIN)
server: HTTP/1.1 200            (responds with index.html)
	        [index.html]
server: FIN
client: ACK

(Note: TLS close-notify is not in the example and is not sent by the client or server)
However, server-side s2n interprets the presence of a FIN from the client is taken as a
closure of both read and write, so the response will not be sent. Openssl does allow
for half-close behavior. To allow half-close behavior in s2n, state could be added to keep
track of closure for read and write ends separately. When getting a close-notify,
s2n would set closed state for both ends.

This was fixed in #3932


Add docs for configuring jetty http port

Suppose I want to serve my app at a port of my choosing. Surely this is supported — how can I configure this with Gretty?

Found it here, the example has port =  but this config option seems to have been removed. Please update the examples.


updated to 6.9; added GroovyScriptBindingSupplier; handle tab-key

Coverage remained the same at 68.421% when pulling a94463a524f996817a91037ceb4fa36d9196f240 on 6-9-support into 1dd75fb8d85ab09b8637d55babb7e30e99ae6c9b on master.


Manual in miniting page is still outdated

Description
A clear description of the bug is:
The support manual link on the miniting page is still outdated.

Must include any relevant identifiers, like:

Network: Testnet

Version: 2.3.0-rc3

Work Completed:
Update minting support link to threefoldfaq

Blocked on https://github.com/threefoldtech/tfgrid-sdk-ts/pull/2376#issuecomment-1987174096

Blocked on #2376 (comment)
verify link is outdated too
@samaradel

please ignore this link for now, update the other links and merge the branch,
will check keep this link till tomorrow

Work Completed:
Update minting support link to threefoldfaq

support link is pointing to https://threefold.io/ not https://threefoldfaq.crisp.help/en/

@A-Harby comment updated

Verified, Devnet d965ce4.
The support link is now directing to https://threefold.io/.

TC2295 - Grid Minting Explorer