add tadej to rookie list

Description
Welcome! We are excited to have you on board. This is your first formal step in making it official that you are a part of the Polkadot Heroes program as either a hero or a rookie. Please fill out your profile details in this pull request. Remember to place your profile in alphabetical order in the list, add your profile image to the ./images subfolder of either the Heroes/ or Rookies/ folder, and fill out all the informational fields in your card. You can copy and paste someone else's card and replace their information with your own.
Make sure to preview your additions in the Heroes/README.md or the Rookies/README.md file to confirm it looks as expected.
Checklist:

[x] My name is alphabetically ordered by first name in the README
[x] My profile picture is added to the /images folder of either the Heroes/ or the Rookies/ directory
[x] All the fields are filled out for my profile (see the other profiles for examples)

Hi @tad3j,
Welcome! 🎉
Can you please move your profile addition to be in alphabetical order by first name, which currently is between Suraj and Osama?

Hi @tad3j,
Welcome! 🎉
Can you please move your profile addition to be in alphabetical order by first name, which currently is between Suraj and Osama?

Done.

Great, thanks! Now you just need to resolve the merge conflict (you can do this in the GitHub UI in this pull request) and then we'll be ready to merge.

@hummusonrails I think it should be ready now. Sorry, I didn't see the conflict before.


test

e

test

1

2

3

4

5

6

7

8

9

10

11

12

@tgolen I don't think you'll mind if I close this lol


need improvements to primary versus secondary highlighting on reflection view

Note that 118a and 1192 are supposedly different highlight colors but it's very subtle.
Once again in dark theme which is a little better, but not great if they aren't contiguous:

More general dup of #2506, feel free to close mine.

Nope, definitely a dupe.


Provide a Virtual Method Anchor for Classes in Headers

🦟 Bug fix
Based on https://github.com/gazebosim/gz-rendering/issues/702#issuecomment-1217673602
Summary
We have some classes defined in headers that need to have at least one virtual method implemented in a cc file.
Checklist

[ ] Signed all commits for DCO
[ ] Added tests
[ ] Updated documentation (as needed)
[ ] Updated migration guide (as needed)
[ ] Consider updating Python bindings (if the library has them)
[ ] codecheck passed (See contributing)
[ ] All tests passed (See test coverage)
[ ] While waiting for a review on your PR, please help review another open pull request to support the maintainers

Note to maintainers: Remember to use Squash-Merge and edit the commit message to match the pull request summary while retaining Signed-off-by messages.

@osrf-jenkins retest this please


Can I extract 3D features per frame?

Thanks for the great work!
Regarding the question, I did try to change the frame rates within the files and it somehow worked with the 2D features, but for the 3D features, it still show result as per second. Can you show me which value I should change? Thanks!

Did you try to change https://github.com/antoine77340/video_feature_extractor/blob/master/extract.py#L34
and replace 24 by another frame per second value?

Thanks for the response!
Yes, I tried to change the frame per second value. But the thing is that I found the dimensions of the output seems smaller than what I expected for frame extraction.
After reading again your paper and the instructions, I found that

The 3D model is a ResNexT-101 16 frames (https://github.com/kenshohara/3D-ResNets-PyTorch) pretrained on Kinetics. The 3D features are extracted at 1.5 feature per second at the resolution of 112.
So as far as I understand, it might be because of using the ResNexT-101 16 frames model that you have to adjust the shape in here?
https://github.com/antoine77340/video_feature_extractor/blob/master/preprocessing.py#L35


Bug: Cannot create a new user when SAML and ABLC rule is enabled

Environment
Host

item
version

GROWI
6.2.x ~

How to reproduce? (再現手順)

Enable SAML and ABLC rule
Login with SAML

The login user to be used must not satisfy ABLC rules.

What happens? (症状)

/passport/saml/callback endpoint doesn't respond anything

When this situation occurs, no error message has been output

What is the expected result? (期待される動作)

/passport/saml/callback endpoint responds valid content.

対応ストーリー：https://redmine.weseek.co.jp/issues/144138


Module Not Found Error with @janus-idp/backstage-plugin-topology v1.22.1

Describe the bug
When using the @janus-idp/backstage-plugin-topology  with my Backstage application, I encounter the following error:
[0] ERROR in<EMAIL_ADDRESS>1:0-75
[0] Module not found: Error: Can't resolve '../plugins/topology/src/imgs/logos/defaulthub.svg' in '/Users/xyz/develop/backstage-demo/node_modules/@janus-idp/backstage-plugin-topology/dist/utils'
[0]
[0] ERROR in<EMAIL_ADDRESS>2:0-70
[0] Module not found: Error: Can't resolve '../plugins/topology/src/imgs/logos/django.svg' in '/Users/xyz/develop/backstage-demo/node_modules/@janus-idp/backstage-plugin-topology/dist/utils'
[0]
[0] ERROR in<EMAIL_ADDRESS>3:0-70
[0] Module not found: Error: Can't resolve '../plugins/topology/src/imgs/logos/dotnet.svg' in '/Users/xyz/develop/backstage-demo/node_modules/@janus-idp/backstage-plugin-topology/dist/utils'
[0]
[0] ERROR in<EMAIL_ADDRESS>4:0-70
[0] Module not found: Error: Can't resolve '../plugins/topology/src/imgs/logos/drupal.svg' in '/Users/xyz/develop/backstage-demo/node_modules/@janus-idp/backstage-plugin-topology/dist/utils'
[0]

This error occurs when I try to load my application at localhost:3000. However, when I revert to version 1.21.11 of the plugin, everything works fine without any issues.
Expected Behavior
What are the steps to reproduce this bug?

Follow the documentation to register the Topology plugin.
Run yarn dev
Navigate to localhost:3000 in a browser.
Observe the error message described above.

Versions of software used and environment
"@janus-idp/backstage-plugin-topology": "^1.22.1",

OS:   Darwin 23.5.0 - darwin/arm64
node: v20.12.1
yarn: 1.22.22
cli:  0.26.10 (installed)
backstage:  1.28.0

Hi @youngyol I have tried to reproduce this issue locally, but the svg icons are loaded correctly in http://localhost:3000/static/ in Sources. Can you try with backstage 1.26.5 once? In package.json it says
  "backstage": {
    "role": "frontend-plugin",
    "supported-versions": "1.26.5"
  },

Hi @ciiay,  thank you for your response.
I tried downgrading to Backstage 1.26.0, but still encounter the same issue.
Here are the steps I followed to set up:
# Created a new Backstage app version 1.26.0
npx<EMAIL_ADDRESS>
# Added the Kubernetes plugin (frontend)
yarn --cwd packages/app add @backstage/plugin-kubernetes

# Added the Kubernetes plugin (backend)
yarn --cwd packages/backend add @backstage/plugin-kubernetes-backend

# Added the Topology plugin
yarn workspace app add @janus-idp/backstage-plugin-topology

# Enabled the TOPOLOGY tab in packages/app/src/components/catalog/EntityPage.tsx:
import { TopologyPage } from '@janus-idp/backstage-plugin-topology';

const serviceEntityPage = (
  <EntityLayout>
    ...
    <EntityLayout.Route path="/topology" title="Topology">
      <TopologyPage />
    </EntityLayout.Route>
  </EntityLayout>
);

# To start the app
yarn install
yarn dev

I also checked the<EMAIL_ADDRESS>file located at the project root, and it contains the following imports:
import defaultImg from '../plugins/topology/src/imgs/logos/defaulthub.svg';
import djangoImg from '../plugins/topology/src/imgs/logos/django.svg';
import dotnetImg from '../plugins/topology/src/imgs/logos/dotnet.svg';
...

However, I noticed that the actual images are located at<EMAIL_ADDRESS>This discrepancy might be causing the issue, but I am not entirely sure.

I want to mention that I am new to Backstage and TypeScript, so I might be missing something obvious.
I would appreciate further assistance. Is there a specific configuration or additional step that I might be missing?
Thank you for your help.

Same issue here, using Backstage version 1.29.1. Issue not present in v 1.21.11 of @janus-idp/backstage-plugin-topology

Hi @youngyol @mikkeschiren , I have tested again with the latest @janus-idp/backstage-plugin-topology 1.25.0 version and not able to reproduce the issue. The svg image references are updated with the correct relative path now as shown below. Could you please confirm again with the latest @janus-idp/backstage-plugin-topology 1.25.0 version?

Hi @ciiay, I tested with version 1.25.0, and everything works fine now.
I'm closing the issue.
Thank you.

I think there's a regression on this side.
Version 1.27.x present same issue described above.
Downgrading to 1.25.0 as suggested works fine with Backstage 1.29.2.

Using 1.25.0 as a workaround for now with backstage 1.31, but please re-open this underlying issue.


fix bug uncovered in LDV testing of address deletes

@njjc build passes and ready to merge.  this is one line fix I found I needed to make while doing LDV test writing.

Nice timing. Adding it to hedap now. :-)

love that this was picked up by the LDV suite


Initial color and builder flag

A brute version of colors and a flag to make content more readable if you are just exploring

There is also a bug if a player's nickname and realname were the same, when looking at username sent you a duplicated description text so I added an if not found_something conditional to the relevant part.
Code quality is probably close to non-existent, these are more like just ideas I guess.


Inquire about a question

Hello, I've been working on replicating your job recently and have encountered some issues. I'd like to ask for assistance.In the Transcription module, after using the Whisper base model for transcription, I have 'train_scraped.csv,' but the generated 'test_scraped.csv' is empty, containing only 'addressfname' and 'transcript.' Is it because the dataset I downloaded only includes training data without testing data, or do I need to manually split it? Or is there another reason?"

Hello, sorry for my delayed response. I'll gladly assist you with your encountered issue. Which dataset are you using? For this project I used the dataset of the ADReSSo challenge which you can download from DementiaBank-ADReSS-2021. Then you should have both training and test data available and your directory tree should look like this:


Updated polyfill to fix IE10 issue

Fix description: https://github.com/webcomponents/webcomponentsjs/commit/feef1883d19c80072c411b332cf97e3e99c11941

LGTM /cc @coreh @mattsherman

Any doubts about this PR?

LGTM

+1


File size increses 10 times when using imgpdf

The output file increases from 500KB to 10MB when I use the 'imgpdf'.
This doesn't happen with the 'pdf' option.
Is there any way to fix this?

I'm starting to work with the library.
I guess it depends first on the complexity of your original PDF...
Did you try to change the scale (default to 0.2) factor and antialiasing (defaut false) properties? What optimisation could you get from it?
Scale factor defaul means 360dpi image per pages (72pdi / 0.2). It probably explains your situation.
A scale of 0.72 would give you 100dpi.
ref:
https://github.com/tpisto/pdf-fill-form/blob/master/src/NodePoppler.cc#L106
https://people.freedesktop.org/~aacid/docs/qt5/classPoppler_1_1Page.html#ae37d9b14a50231ed92260552964a1ed2


Demo app does not display DICOM P10 files

Hi all,
I've a cornerstone demo app installed on linux, working with no issue.
I've also some files, all P10 compliant (verified with dcmftest from dcmtk).
Modifying a json file for a study with these lines
    {
        "seriesUid" : "5",
        "seriesDescription": "PATHOLOGY",
        "seriesNumber" : "5",
        "instanceList" : [
            {"imageId": "http://localhost:8000/DICOM/WRIST_RIGHT/T1/IM-0001-0012.dcm"},
            {"imageId": "http://localhost:8000/DICOM/WRIST_RIGHT/T1/IM-0001-0006.dcm"},
            {"imageId": "http://localhost:8000/DICOM/WRIST_RIGHT/T1/IM-0001-0001.dcm"},
            {"imageId": "http://localhost:8000/DICOM/WRIST_RIGHT/T1/IM-0001-0004.dcm"},
            {"imageId": "http://localhost:8000/DICOM/WRIST_RIGHT/T1/IM-0001-0008.dcm"},
            {"imageId": "http://localhost:8000/DICOM/WRIST_RIGHT/T1/IM-0001-0011.dcm"},
            {"imageId": "http://localhost:8000/DICOM/WRIST_RIGHT/T1/IM-0001-0005.dcm"},
            {"imageId": "http://localhost:8000/DICOM/WRIST_RIGHT/T1/IM-0001-0003.dcm"}
        ]
    }

the viewer only display a black square. All files are accessible and can be downloaded with a browser.
Chrome developer tool console display no errors, but in "network" tab, on the first image, I've a "failed to response data" message. Any ideas?
Is there a way to use plain apache file serving as a store for dicom images and cornerstone demo app? If not, can you give me some hints so I can write a php "wrapper" to serve dicom files without install orthanc or other wado gateways?
Many thanks,
Francesco

Hi Francesco,
Yes you can use Apache to serve DICOM files.  Please try following the troubleshooting guide here:
https://github.com/chafey/cornerstoneWADOImageLoader/wiki/troubleshooting

hi chris,
I setup apache with cors (http://enable-cors.org/server_apache.html) and also with corsproxy,
but nothing changed. same dicom files served from same apache (with or without corsproxy)
are usable with dwv viewer, but I think cornerstone is better suited for my project.
I'll try to debug the http transfer with tshark or some other low level network tools...
thank you and have a nice day,
francesco

If the images display fine in DWV then the problem is not with your apache configuration.  I suspect your ImageIds are not correct - to load DICOM images you must use the cornerstoneWADOImageLoader which uses the 'wadorui' scheme in the imageId.  You are using the http scheme which maps to the cornerstoneWebImageLoader which is used for non DICOM images like PNG, JPEG, etc.

I've tried with "wadouri:http://...." with no luck.

now it works!
raw dicom files from a php script (with ob_clean) from the same web server,
with "dicomweb:/path/to/dicom/file" in json list.
thank you for this great programs!


Added license information.

Type: Missing
Summary:
Added license information.
Details:
The license for SpecRun.Runner was not included.
Resolution:
The license was found for SpecRun.Runner (and other SpecRun packages):
https://github.com/SpecFlowOSS/SpecFlow.Rider/blob/master/LICENSE
Affected definitions:

SpecRun.Runner 1.6.0

License link points to http://www.specflow.org/plus/eula/, which is a EULA. Updated to OTHER.

Thank you for your contribution!


⚠ Samsung has degraded performance

In 4ccd5d2, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 2205 ms

Resolved: Samsung performance has improved in 0cae2c4.


Wrong redirection from analytics page

Actions before raising this issue

[X] I searched the existing issues and did not find anything similar.
 As I remember, in older versions, the "Back" button took you to the last project pages.
Environment
Server version: 2.16.1

Core version: 15.1.0

Canvas version: 2.20.6

UI version: 1.64.0

Hi, thanks for the report. We will look into this


Update rubyinstaller-devkit-2.6.6-1-x64.exe SHA

This just fixes the windows functional verify tests to use the correct sha for rubyinstaller-devkit-2.6.6-1-x64.exe`
Signed-off-by: Christopher A. Snapp<EMAIL_ADDRESS>

Thanks @christopher-snapp


Node.js bindings: Throw a proper error object in single-file test

@zqzhang, @wanghongjuan this should fix the NaN response when a test is torn down. The invasive tests need sudo to run, so if you don't have that in your environment, the test will fail.
Can you please confirm that this produces an informative error message for you?

Thanks a lot. The issue doesn't reproduced when using the modified code.The Ostro image initial role is root. Is it viable that run the invasive tests by using root role without using sudo command?

@wanghongjuan I was thinking about that. I need to learn how to detect whether a process is running as root, and prepend sudo contingent upon that.


why use Repository

Why do you use the Repository ?
In most of the articles I read, it is said that the Net Framework itself uses the repository and the UOW pattern, so it is not correct to use it again in a clean architecture.

It's purely a preference. It's useful to create an abstraction over EF Core. It doesn't create a disturbance in the architecture if neatly executed.


Introduce Alpine Docker build

Signed-off-by: Damien DUPORTAL<EMAIL_ADDRESS>This Pull Request introduce an alternative docker build based on Linux Alpine.

A special GlibC build was needed
LibWebSockets had to be built from source

The resulting image is ~68 Mb (~ 28 Mb when compressed from a docker registry).
You can find a pre-built version here: https://hub.docker.com/r/dduportal/ttyd/ to test

Nice thanks !
Now that the tag have been built on the DockerHub, I've deleted my own image version, and my fork (don't need them: everything is on your own image).
Have a nice day, thanks for your work .


Uploading file directly to the cloud storage with fileWriteStreamHandler, real benefit?

According to the documentation, «if you are looking to write the file uploaded in other types of cloud storages (AWS S3, Azure blob storage, Google cloud storage) or private file storage, the fileWriteStreamHandler is the option you're looking for.»
So, I want to upload a file directly to the cloud storage without storing it in between on the application server and ideally saving the application server RAM, CPU, and storage on temporary keeping the uploading file.
This is especially critically when we're talking about uploading big files, which application server can't handle in the RAM in whole.
The dropbox-sdk-js expects receiving a file buffer:
await dbx.filesUpload({
	contents: fileBuffer,
	path: filePath
});

while the fileWriteStreamHandler gets a file path and if I understand it correctly, to be able to forward it to Dropbox, I still need to generate a file buffer with fs.readFile on the application server:
const fileBuffer = await fs.readFile(file.filepath);

So what's the real benefit of using fileWriteStreamHandler comparing to uploader.parse()? In both cases, I need to generate a buffer on a server side and only after to forward it to the storage. Or do I miss something important here?

Ideally every cloud storage has an API that supports Node streams for upload.
If that is not the with dropbox, then don't use fileWriteStreamHandler option.
I think you are right unfortunately.


Http or Https

Are you using running this as http or https service that can be accessed from the main excalidraw app.. I tried this code.. unfortunately does not work for me.. My s3 bucket is setup and I have the access keys and other variables also set.. not sure where is the problem

@thecloudgarage sorry, only just saw this. I've only ever tried HTTPS.


Make it i18n friendly

@satazor implemented this for js.ipfs.io using react-intl and it is super easy to work with. @satazor, wanna shine some light here?

Yes we used react-intl in js.ipfs.io, mainly because it's made for react and supports:

messages with placeholders
date & time
relative dates (xxx ago)
number formatting, including currency

The strategy we went for was to have a intl folder that container everything related to i18n. Contributors just need to follow a few steps to add a new language, which are detailed in the README.

I think that's all, feel free to ask more questions and I will happily provide help.

cc https://github.com/ipfs/ipfs-gui/issues/50

Some relevant notes: https://github.com/ipfs/i18n#adding-i18n-to-your-project
I think we won't have a lot of string in revamped version.
As soon initial UI lands,  may be a good idea to prioritize i18n and do it sooner than later, that way translation team will have more time to work on it before shipping :)

I can add it on the PR I'm working on. It's not that much 😄

@hacdias as long the change set is not too big.
(Would be cool to have it in separate PR, so we can link to it as an example for other projects that want to add i18n)

@lidel it would probably introduce a lot of changes to the code I wrote so far, but basically just adding translation functions. We can leave it for another PR just to keep things separated. #662 is already getting really big, but it couldn't be other way since it introduces really deep changes.


WPF: switching display device while app is running causes crash

Issue:
When using multiple screens, switching the display device causes the program to crash.

With two monitors connected and the projection mode set to "Extend," the program runs normally.
However, when changing the screen projection mode to "PC screen only," the program freezes and crashes.

ScottPlot Version: 5.0.43
The type of control used: ScottPlot.WPF.WpfPlotGL.
WpfPlotGL does not have this issue.

Thanks for reporting this @kwonganding!
I think there is a typo in the last line of your message... Can you clarify whether this affects the normal control or the opengl control?
Thanks!

... also if the crash has an error message (or better yet the full message of an exception), if you could paste that here it would be really helpful 👍

The type of control used: ScottPlot.WPF.WpfPlotGL.
WpfPlot does not have this issue.
-- sorry，I remember modifying it， I might have forgotten to commit.

Hi @kwonganding, thanks for the follow-up!

When switching displays, the program freezes (no response) with no error message.

In my case it just went black, but clicking and dragging the control to pan made it appear again. Is this the same behavior you experience? If so, perhaps you can detect the display change then force a redraw.
This page may be helpful for that?
https://stackoverflow.com/questions/11983111/detect-when-new-display-is-connected
I think this may be out of the scope of "ScottPlot" since the behavior in question is SKGLControl and is part of the SkiaSharp package managed by Microsoft. I'll close this issue for now, but if it turns out there's something concrete I can do to make this better let me know and we can open this back up and implement a fix 👍


Avoid crash when fail to parse fort name

Short Description:
Traceback (most recent call last):
  File "pokecli.py", line 126, in <module>
    main()
  File "pokecli.py", line 119, in main
    bot.take_step()
  File "/Users/sinistance/Documents/pokemon/PokemonGo-Bot-dev/pokemongo_bot/__init__.py", line 33, in take_step
    self.stepper.take_step()
  File "/Users/sinistance/Documents/pokemon/PokemonGo-Bot-dev/pokemongo_bot/stepper.py", line 56, in take_step
    self._work_at_position(position[0], position[1], position[2], True)
  File "/Users/sinistance/Documents/pokemon/PokemonGo-Bot-dev/pokemongo_bot/stepper.py", line 104, in _work_at_position
    self.bot.work_on_cell(cell, position, pokemon_only)
  File "/Users/sinistance/Documents/pokemon/PokemonGo-Bot-dev/pokemongo_bot/__init__.py", line 97, in work_on_cell
    hack_chain = worker.work()
  File "/Users/sinistance/Documents/pokemon/PokemonGo-Bot-dev/pokemongo_bot/cell_workers/seen_fort_worker.py", line 48, in work
    fort_details = response_dict['responses']['FORT_DETAILS']
KeyError: 'FORT_DETAILS'

Fixes:

avoid crash when fail to parse fort name
remove userdata.js from git replace it with example file

oops conflicts....

@solderzzc updated

Thanks , I hate this issue for long....


Changed Sorter trait to be generic

This will make the trait object safe, and also allow implementations of
sorting algorithms that do not work for all Ord types.

I also for some reason had to change the input arguments to fn bench(). Don't know why it worked before but didn't now. Hope I didn't break anything with that change.


Updated to react 15.2.1, and adjusted readme's build process

I've updated the package to use the newest react. The project we used it with pulled in react 0.14.
Also a small bit on the build process was missing in the readme.md

Updated all dependences to the latest.
Remove node 4 support
Added node 6 support

Seems I have some things to fix


RocketMQ consume with number of consumeThreadMin, consumeThreadMax value doesn't work.

The issue tracker is ONLY used for bug report(feature request need to follow RIP process). Keep in mind, please check whether there is an existing same report before your raise a new one.
Alternately (especially if your communication is not a bug report), you can send mail to our mailing lists. We welcome any friendly suggestions, bug fixes, collaboration and other improvements.
Please ensure that your bug report is clear and that it is complete. Otherwise, we may be unable to understand it or to reproduce it, either of which would prevent us from fixing the bug. We strongly recommend the report(bug report or feature request) could include some hints as the following:
BUG REPORT

Please describe the issue you observed:

What did you do (The steps to reproduce)?
when I consume the message, increase the value of consumeThreadMax doesn't work.

What did you expect to see?
I expert consume speed will be faster when increase the value of consumeThreadMax

What did you see instead?
the consume speed  keep stable even I increase the value of consumeThreadMax

Please tell us about your environment:
4.3.0, I checked the lasted version, it still happens.
Other information (e.g. detailed explanation, logs, related issues, suggestions how to fix, etc):

I found some in the class ConsumeMessageConcurrentlyService:
public ConsumeMessageConcurrentlyService(DefaultMQPushConsumerImpl defaultMQPushConsumerImpl,
MessageListenerConcurrently messageListener) {
this.defaultMQPushConsumerImpl = defaultMQPushConsumerImpl;
this.messageListener = messageListener;
    this.defaultMQPushConsumer = this.defaultMQPushConsumerImpl.getDefaultMQPushConsumer();
    this.consumerGroup = this.defaultMQPushConsumer.getConsumerGroup();
    this.consumeRequestQueue = new LinkedBlockingQueue<Runnable>();

    this.consumeExecutor = new ThreadPoolExecutor(
        this.defaultMQPushConsumer.getConsumeThreadMin(),
        this.defaultMQPushConsumer.getConsumeThreadMax(),
        1000 * 60,
        TimeUnit.MILLISECONDS,
        this.consumeRequestQueue,
        new ThreadFactoryImpl("ConsumeMessageThread_"));

    this.scheduledExecutorService = Executors.newSingleThreadScheduledExecutor(new ThreadFactoryImpl("ConsumeMessageScheduledThread_"));
    this.cleanExpireMsgExecutors = Executors.newSingleThreadScheduledExecutor(new ThreadFactoryImpl("CleanExpireMsgScheduledThread_"));
}

This constructor take unbounded queue(new LinkedBlockingQueue())  as  workQueue，it has a risk that when new task comes, if the number equals to consumeThreadMin, it will be pushed to the unbounded queue, the consume thread number will NEVER be increased. so the consumeThreadMax value will NEVER work。 so consumeRequestQueue should be assigned as a bounded queue as workQueue, like this.consumeRequestQueue = new LinkedBlockingQueue(128);

yep, so it would be better to set minThread equals with maxThread, but if you want to replace with a  bounded queue as workQueue,  it will lead to the message lost in the condition of this queue is full :).


installation trouble

Hi, i have a fresh install of JW on my raspberry pi home server, but i having trouble to use JW.
i have another script like rss reader hosted, work perfectly.
[my nginx config]
(https://www.zerobin.net/?324c89e0aa94e894#nzukmVwmH7Z4Zv0J4BRpf8Bw+qeDn4QhB5zItlQxJcg=)

Thanks Tom.

what's your trouble? style lost?

No style, postes redirect to error
Maybe caused by nginx conf ?

It's my nginx config.
https://gist.github.com/hjue/647dc694dc3b67994202

i have almost the [same]
(https://www.zerobin.net/?324c89e0aa94e894#nzukmVwmH7Z4Zv0J4BRpf8Bw+qeDn4QhB5zItlQxJcg=)


feat: tag 组件

描述：仿 antd tag 组件，组成: background + marker + text
目的：尝试熟悉 G 5.0
待办

[ ] 宽度限制，溢出省略
[x] 单测
[ ] 文档

形态
激活形态

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

65 of 68 (95.59%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+1.2%) to 68.56%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/ui/tag/index.ts
63
66
95.45%

Totals

Change from base Build<PHONE_NUMBER>:
1.2%

Covered Lines:
1225

Relevant Lines:
1669

💛  - Coveralls


Create a ref doc for pod security policy

@kubernetes/docs @janetkuo Initial documentation for PSP.
Feature issue: https://github.com/kubernetes/features/issues/5

This change is

Looks good. A couple of minor comments.
However, since this feature isn't live until release 1.4, can you please retarget this PR to the release-1.4 branch?

Duplicate of #1206. I'll go ahead and close this one.


CLOUDP-266544: Support local resource credentials

Add common infrastructure for adding local per CRD credentials at pkg/api/credentials.go.
Use the new infra from AtlasDatabaseUser's CRD and Controller.
Solve whether the credentials are taken from the current resource, the project or render nil.

Testing:

Unit tests on new ComputeSecret.
Conversion test fix: credentials must be ignored on comparisons between the Atlas and Kubernetes Spec.
Users e2e test: Second user takes locally defined credentials. Also if local credentials do not exist, reconciliation fails as expected.

All Submissions:

[X] Have you signed our CLA?

One nit about the global secret fallback handling. The logic is quite confusing to the reader. Why not make it explicit, where:
type Provider interface {
...
	GlobalSecret() client.ObjectKey
}

and then just invoke:
credentialsSecret, err := customresource.ComputeSecret(r.AtlasProvider.GlobalSecret(), project, databaseUser)
...

where the fallback logic is explicit in ComputeSecret:
func ComputeSecret(globalSecret client.ObjectKey, project *akov2.AtlasProject, resource api.ResourceWithCredentials) (*client.ObjectKey, error) {
	if globalSecret.Name != "" {
		return &globalSecret, nil
	}
	return resolveConnectionSecret(project, resource)
}

There is less indirection and makes the fallback explicit.

One nit about the global secret fallback handling. The logic is quite confusing to the reader. Why not make it explicit, where:
type Provider interface {
...
	GlobalSecret() client.ObjectKey
}

and then just invoke:
credentialsSecret, err := customresource.ComputeSecret(r.AtlasProvider.GlobalSecret(), project, databaseUser)
...

where the fallback logic is explicit in ComputeSecret:
func ComputeSecret(globalSecret client.ObjectKey, project *akov2.AtlasProject, resource api.ResourceWithCredentials) (*client.ObjectKey, error) {
	if globalSecret.Name != "" {
		return &globalSecret, nil
	}
	return resolveConnectionSecret(project, resource)
}

There is less indirection and makes the fallback explicit.

In the end I removed the global fallback from this part of the resolution. That will still work as before.
The only loss is we do not have precise errors when there is no global fallback, but that scenario was problematic to open up so soon anyway. We should decide if the global fallback will be optional separately.

rerunning e2e gov, let's evaluate if this is still a blocker.

Our nightly test suite saw recovery today at midnight on the gov tests:
✅ cloud-tests / test-e2e-gov / E2E Gov tests

@s-urbaniak all green again, I closed CLOUDP-270948


feat: add a selection list when deleting auth tokens

Manta can manage multiple auth tokens depending on the site the user has worked with.
In order to improve the auth token cleaning process, we will provide the user with a list of local auth token to delete
Please chose the site token to delete from the list below::
> alpsm_auth
  alps_auth

solved in v1.28.13


Investigate using native DOM events

Rather than callbacks

User can access value (formatted) and rawValue from input object


VSCode, Clean up: Remove classic context status indicator

Remove the classic context status indicator. This has been replaced by the enhanced context settings widget. Also clean up some app-less onboarding properties.
Test plan
CI, specifically
pnpm test:integration
pnpm test:e2e

QQ: Do we still need https://github.com/sourcegraph/cody/blob/main/vscode/src/services/LocalAppDetector.ts at this point?

It's not used right now, but the last bit of #1966 is to toast users to remove it, so I want to use it for that. We can delete it and I can pull it back out of history though if we want.


Testing problem

Thanks for publishing the codebase.I have a question about testing.I set it up exactly as you mentioned in the README documentation.The rest arguments of testing are same as training.For example,'load_model=1'  and 'model_load_dir="/output/worksfor/b2b4_3_0.05_100_0.05/model/model.ckpt"'.But I got the following error:
INFO:tensorflow:Restoring parameters from /output/worksfor/b2b4_3_0.05_100_0.05/model/model.ckpt
INFO:tensorflow:Restoring parameters from /output/worksfor/b2b4_3_0.05_100_0.05/model/model.ckpt
07/06/2019 03:42:44 PM: [ Restoring parameters from /output/worksfor/b2b4_3_0.05_100_0.05/model/model.ckpt ]
2019-07-06 15:42:44.906846: W tensorflow/core/framework/op_kernel.cc:1318] OP_REQUIRES failed at save_restore_tensor.cc:170 : Invalid argument: Unsuccessful TensorSliceReader constructor: Failed to get matching files on /output/worksfor/b2b4_3_0.05_100_0.05/model/model.ckpt: Not found: /output/worksfor/b2b4_3_0.05_100_0.05/model; No such file or directory
Traceback (most recent call last):
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call
    return fn(*args)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Unsuccessful TensorSliceReader constructor: Failed to get matching files on /output/worksfor/b2b4_3_0.05_100_0.05/model/model.ckpt: Not found: /output/worksfor/b2b4_3_0.05_100_0.05/model; No such file or directory
	 [[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "code/model/trainer.py", line 574, in <module>
    trainer.initialize(restore=save_path, sess=sess)
  File "code/model/trainer.py", line 144, in initialize
    return  self.model_saver.restore(sess, restore)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1802, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 900, in run
    run_metadata_ptr)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1135, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
    run_metadata)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Unsuccessful TensorSliceReader constructor: Failed to get matching files on /output/worksfor/b2b4_3_0.05_100_0.05/model/model.ckpt: Not found: /output/worksfor/b2b4_3_0.05_100_0.05/model; No such file or directory
	 [[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

Caused by op 'save/RestoreV2', defined at:
  File "code/model/trainer.py", line 574, in <module>
    trainer.initialize(restore=save_path, sess=sess)
  File "code/model/trainer.py", line 138, in initialize
    self.model_saver = tf.train.Saver(max_to_keep=2)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1338, in __init__
    self.build()
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1347, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1384, in _build
    build_save=build_save, build_restore=build_restore)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 835, in _build_internal
    restore_sequentially, reshape)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 472, in _AddRestoreOps
    restore_sequentially)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 886, in bulk_restore
    return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1463, in restore_v2
    shape_and_slices=shape_and_slices, dtypes=dtypes, name=name)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3392, in create_op
    op_def=op_def)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1718, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Unsuccessful TensorSliceReader constructor: Failed to get matching files on /output/worksfor/b2b4_3_0.05_100_0.05/model/model.ckpt: Not found: /output/worksfor/b2b4_3_0.05_100_0.05/model; No such file or directory
	 [[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

I still have a question I would like to ask you. Does the training part also include testing(I found that the test() function in trainer.py is also executed when only training)? Are the results of your papers calculated by training or by testing? Thanks again.

Can you train the model from scratch? I think there's a mismatch in tf versions.
The trainer class has the testing function. You could call it outside the train method also.
The results reported in the paper are using the test function.

Thank you for your answer.I have already solved the problem of "Testing". It needs to add a "/" symbol, like this: /output/worksfor//b2b4_3_0.05_100_0.05/model/model.ckpt.
What you mean is that when you execute the "sh run.sh configs/xxx.sh" command, you have already included a test, right?
By the way,hat is the reason for each test result being different?

There's no constant seed used which results in different results after every run.

Appreciate your help!

Hi,
@DR73 How do you solve the problem? What do you mean by "add a '/' symbol"? According to the readme, we need to modify "model_load_dir="saved_models/countries_s2/model.ckpt"", but it doesn't work. How to solve it?

Like this: /output/worksfor//b2b4_3_0.05_100_0.05/model/model.ckpt.Double '/' after relation name ‘worksfor’. But I don't know whether it applies to your situation.


!sid

display player steam ids

Alright I'll take a look at this as it it labeled important. But I have some questions:

Should we save the SID to the database? This way we can see the steamID of everyone who has played on the server.
Should this command admin only?
If we would save the SID to the database should we change every command that uses the name as an indicator of the player to the steamID? This way users can change their usernames but would still be recognized by the server This step could and should also be split into smaller parts so we don't need to change everything at once.

SteamIDs should already be available in the player objects, we're not saving them yet just to keep database compatibility. I doesn't need to be admin only, I'm going to implement a config option for admin only commands. AKeep using usernames in commands for now, we're going to use the player-key (iirc this should be available too) for identification instead.
I'll do a single large update to the database schema before the next release.


Adding --iree-hal-substitute-executable= flag.

This allows for specifying one or more executable_name=file.xxx pairs that each replace a hal.executable op with the executable_name with the contents of file.xxx. .mlir/.mlirbc files are loaded and a hal.executable with the matching name is used as a replacement while any other file type will cause the original executable to be externalized and linked with the specified file (.ptx/.spv/etc).
The additional --iree-hal-substitute-executable-*-from= flags allow for scanning a directory for executables by name to build the substitution mapping. Only files in the executable_name or module_executable_name form will be substituted but we can extend this in the future to support variant naming.
Because of phase ordering constraints around where codegen is able to mutate host-related code such as workgroup count calculations there are two flag sets:
--iree-hal-substitute-executable-source=name=file.xxx
--iree-hal-substitute-executable-sources-from=path/
and
--iree-hal-substitute-executable-object=name=file.xxx
--iree-hal-substitute-executable-objects-from=path/
Sources are substituted immediately prior to benchmark generation and when substituting target objects (.ptx, .spv, etc) require that it's ok to skip codegen (workgroup count calculation is not dependent on root op detection, etc). Objects are substituted immediately after codegen for use in cases where codegen is to generate the host code. There are uses for both depending on what the input IR is and what the developer wants to modify (host code, device code, or both).
The primary developer workflows this covers are:

dump executable sources via --iree-hal-dump-executable-sources-to= and modify them, potentially running any number of iree-opt passes, before linking them back in to the original program they came from
author custom implementations ala the custom_dispatch sample in target toolchains (.cu -> .ptx, .glsl -> .spv, etc) and use those in full programs without needing to modify the compiler
do either of the above and use the substituted executable for microbenchmarking via --iree-hal-dump-executable-benchmarks-to= (so one can easily microbenchmark handwritten kernels)

Example usage:
# dump sources for a program
iree-compile ... \
  --iree-hal-dump-executable-sources-to=~/sources/
# <modify some of the sources>
# recompile with the new changes and substitute 2 executables
iree-compile ... \
    --iree-hal-substitute-executable-source=_main_dispatch_0=~/sources/modified_dispatch_0.mlir \
    --iree-hal-substitute-executable-source=_main_dispatch_1=~/sources/modified_dispatch_1.mlir
# same thing with search paths
iree-compile ... \
    --iree-hal-executable-object-search-path=~/sources/ \
    --iree-hal-substitute-executable-source=_main_dispatch_0=modified_dispatch_0.mlir
# same thing but matching all files by name as from dump-sources-to:
iree-compile ... \
    --iree-hal-substitute-executable-sources-from=~/sources/

This works with ptx/spv as well:
# dump ptx binaries
iree-compile ... \
    --iree-hal-dump-executable-binaries-to=~/binaries/
# <modify dispatch ptx>
# replace @_main_dispatch_0 with the external ptx file
iree-compile ... \
    --iree-hal-substitute-executable-object=_main_dispatch_0=~/binaries/modified_dispatch_0.ptx

It's also possible to iterate on microbenchmarks using the custom sources/objects:
# dump all benchmarks with the substitution active
iree-compile ... \
    --iree-hal-substitute-executable-source=_main_dispatch_0=~/sources/modified_dispatch_0.mlir \
    --iree-hal-dump-executable-benchmarks-to=~/benchmarks/
# inspect benchmark for the dispatch and see the hello.world attr
# can use iree-compile to build the benchmark and then iree-benchmark-module

Progress on #12222 (the rest for linking alternative formats is orthogonal).

Locally I've been able to get things working with commands line the following:
# dump sources for a program
iree-compile ... \
  --iree-hal-dump-executable-sources-to=~/sources/
# <modify some of the sources>
# recompile with the new changes and substitute 2 executables
iree-compile ... \
    --iree-hal-substitute-executable=_main_dispatch_0=~/sources/modified_dispatch_0.mlir \
    --iree-hal-substitute-executable=_main_dispatch_1=~/sources/modified_dispatch_1.mlir
# same thing with search paths
iree-compile ... \
    --iree-hal-executable-object-search-path=~/sources/ \
    --iree-hal-substitute-executable=_main_dispatch_0=modified_dispatch_0.mlir
# same thing but matching all files with the right name:
mv ~/sources/modified_dispatch_0.mlir ~/sources/_main_dispatch_0.mlir
iree-compile ... \
    --iree-hal-substitute-executables-from=~/sources/

This works with ptx/spv as well:
# dump ptx binaries
iree-compile ... \
    --iree-hal-dump-executable-binaries-to=~/binaries/
# <modify dispatch ptx>
# replace @_main_dispatch_0 with the external ptx file
iree-compile ... \
    --iree-hal-substitute-executable=_main_dispatch_0=~/binaries/modified_dispatch_0.ptx

I also tested things like benchmark generation:
# dump all benchmarks with the substitution active
iree-compile ... \
    --iree-hal-substitute-executable=_main_dispatch_0=~/sources/modified_dispatch_0.mlir \
    --iree-hal-dump-executable-benchmarks-to=~/benchmarks/
# inspect benchmark for the dispatch and see the hello.world attr
# can use iree-compile to build the benchmark and then iree-benchmark-module

I'll get a test or two in (tricky due to external files in lit tests being wonky on bazel/etc) but it should be ready for testing now if you both want to try!

I have an actual use case for this, giving it a try now.

I wasn't able to get this to work with .spv files. I assumed that because the examples included .ptx, it should also work with .spv.
My workflow was something like this:

Compile the original program and dump binaries:
func.func @main() {
 %lhs = util.unfoldable_constant dense<1> : tensor<1024x1024xi8>
 %rhs = util.unfoldable_constant dense<1> : tensor<1024x1024xi8>
 %c0 = arith.constant 0 : i32
 %i0 = arith.constant 0 : index
 %init = tensor.empty() : tensor<1024x1024xi32>
 %CC = linalg.fill ins(%c0 : i32) outs(%init : tensor<1024x1024xi32>) -> tensor<1024x1024xi32>
 %D = linalg.matmul ins(%lhs, %rhs: tensor<1024x1024xi8>, tensor<1024x1024xi8>)
                   outs(%CC: tensor<1024x1024xi32>) -> tensor<1024x1024xi32>
 check.expect_eq_const(%D, dense<1024> : tensor<1024x1024xi32>) : tensor<1024x1024xi32>
 return
 }

iree-compile matmul_1k_i32_i32.mlir --iree-hal-target-backends=vulkan-spirv --iree-vulkan-target-triple=valhall-unknown-android31 --iree-hal-benchmark-dispatch-repeat-count=100 --iree-hal-dump-executable-binaries-to=binaries -o benchmark.vmf

Produce a new spir-v file based on the binary in binaries:
mkdir replace
cp binaries/module__main_dispatch_0_vulkan_spirv_fb.spv replace
# modify replace/module__main_dispatch_0_vulkan_spirv_fb.spv

Try using ./replace as the substitution dir:
iree-compile/matmul_1k_i8_i32.mlir --iree-hal-target-backends=vulkan-spirv --iree-vulkan-target-triple=valhall-unknown-android31 --iree-hal-benchmark-dispatch-repeat-count=100 --iree-hal-dump-executable-binaries-to=./dumps --iree-hal-substitute-executables-from=./replace -o benchmark.vmfb
WARNING: iree-hal-substitute-executables could not perform the requested substitution as the executable `_main_dispatch_0_vulkan_spirv_fb` was not found in the module

# Try to fix the name mismatch.
mv replace/module__main_dispatch_0{_vulkan_spirv_fb,}.spv

  # recompile
  NOTE: hal.executable `_main_dispatch_0` substituted with object file at `./replace/module__main_dispatch_0.spv`
  matmul_1k_i8_i32.mlir:8:8: error: failed to materialize conversion for result #0 of operation 'arith.constant' that remained live after conversion
    %D = linalg.matmul ins(%lhs, %rhs: tensor<1024x1024xi8>, tensor<1024x1024xi8>)
         ^
  matmul_1k_i8_i32.mlir:8:8: note: see current operation: %1 = "arith.constant"() {value = 1 : index} : () -> index
  matmul_1k_i8_i32.mlir:8:8: note: see existing live user here: %89:3 = "flow.dispatch.workgroup_count_from_dag_root"(%3, %3, %1) : (index, index, index) -> (index, index, index)
  matmul_1k_i8_i32.mlir:0:0: error: conversion to vm.module failed
  matmul_1k_i8_i32.mlir:0:0: note: see current operation:

Try using a substitution with explicit mapping:  --iree-hal-substitute-executable=_main_dispatch_0=./replace/module__main_dispatch_0.spv. This failed with the same error as 3.

@benvanik am I holding this wrong, or is .spv substitution not supported yet?

spv should work fine - not sure what the arith thing is - can you post the IR before that error?
(I'll try to repro here after breakfast)

I'll also see if there's an easier route for substituting single variants based on filename, but I'm not sure the filenames are something we want to make stable and it may always require a rename if trying to do the automatic directory-based substitution

spv should work fine - not sure what the arith thing is - can you post the IR before that error?

Sure, but the repro above is much more concise. (I assume you meant --mlir-print-ir-after-failure)
failure.log

oh I think I see the issue - if you substitute an spv file the automatic workgroup count calculation stuff can't then do its thing and doesn't know how to replace the workgroup_count_from_dag_root with what it should be - I'm not sure of a solution to that :/
that's unfortunate - may require some hand editing to make it possible

I'll also see if there's an easier route for substituting single variants based on filename, but I'm not sure the filenames are something we want to make stable and it may always require a rename if trying to do the automatic directory-based substitution

This doesn't seem like a big deal for me at all, it's very easy to drop the name suffix with a one-liner in bash

ok here's my thought: split the flag in two and support --iree-hal-substitute-executable-source= and --iree-hal-substitute-executable-object= - the former would do what this already does and the latter would run all of codegen just to produce the workgroup count and then do the swap afterward. If you want to change the workgroup size/count/etc you'd edit the mlir sources and pass one or both flags, and if you just wanted to edit the object without changing any host side stuff you'd just use the latter.

unfortunately the benchmark generation wouldn't work with this route in a single invocation as benchmarks are emitted prior to codegen, but on the compilation of the benchmark you could still swap out the object

PTAL @kuhar - I tested your command sequence (without the directory stuff, because that's harder) and it works now:
--iree-hal-substitute-executable-object=_main_dispatch_0=D:\\Dev\\iree-tmp\\exe\\bin\\module__main_dispatch_0_vulkan_spirv_fb.spv

executable

Thanks @benvanik. Now it does compile for me but fails to run on android:
iree/runtime/src/iree/hal/drivers/vulkan/native_executable.cc:157: UNAVAILABLE; VK_ERROR_INITIALIZATION_FAILED; while invoking native function hal.executable.create; while calling import;
[ 1]   native hal.executable.create:0 -
[ 0] bytecode module.__init:250 /usr/local/google/home/kubak/iree/i8-benchmarks/ubenchmark/matmul_1k_i8_i32.mlir:8:8

are you sure your spv is valid? can you turn on the validation layers?

Hmm, I'll double check, all I did was taking the dumped spir-v and cycling through spirv-cross and glslc...

Hmm, I'll double check, all I did was taking the dumped spir-v and cycling through spirv-cross and glslc...

I tried something simpler than going to glsl: I manually inserted 4 OpNop to the binary and re-assembled it. With the original .spv, the replacement works fine. When I supply the .spv with nops, it crashes. I added a debug print just before vkCreateShaderModule and can see that the size of the shader increased by 16 bytes like I expect, so it seems like it does use the substitute shader.

can you dump the data passed to vkCreateShaderModule? if it's the exact bytes you have in your .spv then I'm not sure it's related to this but maybe something else?

can you dump the data passed to vkCreateShaderModule? if it's the exact bytes you have in your .spv then I'm not sure it's related to this but maybe something else?

I confirmed the shader passed to vulkan is bit-identical to the substitute spv.

I also tried enabling validation layers (added --vulkan_validation_layers=true to iree-benchmark-module) and it didn't report anything

what's the diff between substituting and not after SubstituteExecutablesPass runs and does the swap? if there's nothing, then this sounds like spirv related stuff to dig down separately - if there are differences (maybe something changing in the pipeline layout, etc) then it's probably related to this.

Yeah, the IR dumps are identical modulo the contents of #hal.executable.object

Cool - something we can work through separately then?

Awesome!


Camera

A camera is one of the most requested feature for Ebiten. I don't have a concrete idea so far.

Instead of passing a screen *Image directly to the draw function maybe pass a struct that has all the same functionality DrawImage etc but it will behind the scenes multiply the DrawImageOptions with camera GeoM?
Something like this

type ScreenRenderer struct {
Screen *ebiten.Image
CameraMatrix *ebiten.GeoM
}
func (renderer *ScreenRenderer) DrawImage(image *ebiten.Image, options *ebiten.DrawImageOptions) {
matrix := renderer.CameraMatrix
options.GeoM.Add(*matrix)
renderer.Screen.DrawImage(image, options)
}

Yes, I think this represents the core idea. Thank you! Another necessary thing is clipping (https://github.com/hajimehoshi/ebiten/issues/996).

options.GeoM.Add(*matrix)

You meant Concat? And as the order matters, the GeoM would be like this:
options.GeoM = renderer.CameraMatrix.Concat(options.GeoM)

Yes, I think this represents the core idea. Thank you! Another necessary thing is clipping (#996).

options.GeoM.Add(*matrix)

You meant Concat? And as the order matters, the GeoM would be like this:
options.GeoM = renderer.CameraMatrix.Concat(options.GeoM)

EDIT: Oops, this should have been:
options.GeoM.Concat(renderer.CameraMatrix)

?

Yes sorry I meant Concat!


Scoped Epics

Case: Multiple elements on the page can depend on same state but they want to maintain their own instances of the state

https://github.com/avcs06/EpicManager/commit/b35d8f19a241e145f520e138ccb755d6090f3ae3


Normal closure status code returned as .protocolError type

In WebSocket.swift, lines 1009-1020 the normal closure with status code 1000 is potentially returned as an error with type .protocolError on disconnect - is this intentional/expected behavior?

This should be addressed in #653. We now send a 1000 normal with a "closed" event as opposed to an error.


Simple LensPop magnitude calculation demo

While figuring out #45 you can make a very short notebook that simply uses the ABFilterMagnitude function from lenspop.stellarpops to compute apparent magnitudes in the ugrizy filters given an AB absolute magnitude in the rest-frame r band, as computed by lenspop.EarlyTypeRelations. If you begin this notebook with import lenspops you'll then need to edit your local clone of LensPops until the functions you need can indeed by imported as a package. For a typical lens galaxy, you can assume vdisp=250.0 and zd=0.4. For a typical AGN source, you can assume 2.0, and you'll need to figure out how LensPops assigns quasar sources their absolute magnitudes. Nice if you can lay out the calculation so the K-corrections are separate from the distance moduli. Have fun!

@drphilmarshall I think I made a graph that calculates rest frame apparent magnitude in r, g, i, z bands.

Would there be any good way to test whether the magnitudes that I calculated are reasonable? Knowing that the HST's limiting magnitude is around 22, I think AGN's apparent magnitude should be at most around 22-23.
I have been pushing the changes to the fork, and the code that calculates the magnitudes is color.py, and the demo notebook is Graph.ipynb. color.py code needs more decompositions, comments, and stylistic improvements because the code is a bit hard to read right now. I will keep working on it!
Thank you so much, and have a nice weekend.

@drphilmarshall Assuming that I calculated the reasonable magnitude above, I modified OM10 code and calculated the magnitude. I pushed all the updates including python codes and notebooks to my fork. I did not make a pull request to your original directory yet.
The demo notebook is Graph.ipynb, the code used to produce demo notebook is color.py, and the final notebook showing the colored output is OM10/notebooks/Painting with lenspop.ipynb. The only problem I have is that the code takes too long time to compute magnitudes for around 15,000 lenses - it took 18 hours to finish running. Still, the magnitude seems reasonable - there are some outliers between 14-16, but most of them are near 20. Could you please check the final results? If everything looks okay, then I will make a pull request.
I will fly back to Korea next Friday, and I think I would have plenty of time to study and work on codes during the vacation. I think this was the only task given to me; would there be anything that I could study on/work on during the winter vacation?
Thank you!

That's great, Jenny! You should start the pull request right away, so that we can easily view and comment on your new code. I took a look at your painting notebook: very good to see that your code is robust enough to cope with all systems. It looks like you have not pushed Graph.ipynb, but that's OK - let's just work on one notebook, painting with lenspop. First thing will be to make a smaller sample of lenses to work with! (Also, note that when you choose the synthetic option, you don't need to provide any filenames.)
The first plot you made looked promising to me, but now let's make some more "diagnostic" plots, in the painting with lenspop notebook. The goal is to find the problems with our approach so that we can solve them.

Do the i-band magnitudes of the lens galaxy match the previous estimates, in the APMAG_I column? You could make a scatter plot of the differences vs 1) the lenspop i-band mag and 2) the redshift.

Are the colors of your lens galaxies reasonable, compared to the SDSS LRGs in data/SDSS_LRGs.txt? You could make a "corner plot" of a sample of OM10 lenses painted with lenspop (you don't need them all!), choosing the following columns: redshift, i-band magnitude, u-g, g-r, r-i, i-z color. The new OM10 tutorial notebook shows how to do this. Then, you can read in the SDSS LRGs and overlay them, in the same way that overlays are made in the tutorial. You'll have to read the plot_sample method to see how to use corner  but it's pretty straightforward.

Do the quasar colors reasonable? You could make a corner plot again, with the same axes as for the lens galaxies - but then overlay a sample of SDSS quasars. A better QSO sample is the one from our statistics course here at KIPAC, shown in this notebook. You should be able to give pandas.read_csv the HTTP URL of the appropriate csv file, to read it in.

To get the updates I made (including the tutorial notebook) you need to git pull from the base repo. First you make a new remote connection to it, and then pull like this:
git remote add base https://github.com/drphilmarshall/OM10.git
git pull base master

Watch out: if this causes conflicts you will need to read the message carefully, and edit the files that contain conflicts until everything works again. (Search for <<<<<< and >>>>>> in the files to make sure you have found them all.)

Closed via #49 !


 ERESOLVE unable to resolve dependency tree
npm ERR!
npm ERR! While resolving<EMAIL_ADDRESS>npm ERR! Found<EMAIL_ADDRESS>npm ERR! node_modules/react
npm ERR<EMAIL_ADDRESS>from the root project
npm ERR!
npm ERR! Could not resolve dependency:
npm ERR! peer<EMAIL_ADDRESS>|| ^16.0.0" from<EMAIL_ADDRESS>npm ERR! node_modules/react-textfit
npm ERR!   react-textfit@"*" from the root project
npm ERR!
npm ERR! Fix the upstream dependency conflict, or retry
npm ERR! this command with --force or --legacy-peer-deps
npm ERR! to accept an incorrect (and potentially broken) dependency resolution.
npm ERR!
npm ERR!
npm ERR! For a full report see:
npm ERR! C:\Users\pberl\AppData\Local\npm-cache_logs\2023-03-16T15_06_56_708Z-eresolve-report.txt
npm ERR!      C:\Users\pberl\AppData\Local\npm-cache_logs\2023-03-16T15_06_56_708Z-debug-0.log

After the error reply with this
 --force

I just tried it out on my react application and it worked

i was facing this same issue . After research i come to know that  npm install react-textfit --save -force works and required module  installed other wise you can try npm i react-textfit --save --legacy-peer-deps but my recommendation is  npm install react-textfit --save -force.

For me the problem is coming on deployment on vercel

The problem I encountered was not in the app rather in deployement, but the problem was resolved after I downgraded my react version


Add typespecs to the API

These should be propagated back to nerves_network_interface and nerves_wpa_supplicant where possible.

@fhunleth this was fixed with #34 yeah?

Yep - thanks!


Duplicate code snippet: storage

storage\azure-storage-blob\samples\blob_samples_authentication.py:create_blob_service_client_oauth

Hi @xiangyan99 could you please clarify what the issue is? I see that the same tags
# [START create_blob_service_client_oauth] and # [END create_blob_service_client_oauth] appear twice, but I believe the two samples provide two different use cases (the first one using ClientSecretCredential, the other using DefaultAzureCredential). Please correct me if I am misunderstanding 😄

Yes. The problem is the same tag. :)
Same tag confuses who references the snippet.
Please rename one of them to make sure they have different tags.


Feature: support to update the replication factor of each table

1. Motivation
Replication factor (RF), is the number of nodes (i.e. replica servers) where each partition of a table is replicated. In Apache Pegasus, we also call it max replica count or replica count.
Initially, the minimum RF was 3; Then, in order to reduce the cost for our users, in https://github.com/XiaoMi/rdsn/pull/932 we've supported RF=1 and RF=2, though it's not reliable enough for most distributed systems while RF is 1 or 2.
In other words, the minimum RF has become 1. Imagine a scenario that we had 2 replica servers in a cluster where a table can only be created with RF=1 or RF=2. As the volume of data continues to grow, the capacity of  this cluster is not enough; Also, it's not scalable and reliable for the 2 replica servers.
Now we would add another 2 nodes into this cluster and increase the RF of each table to 3 to improve the reliability and scalability. Therefore, besides that RF is specified while creating table, it should be changeable after the table is created. To make RF settable dynamiclly, appropriate commands should be added into the management tool such as Pegasus Shell to query or update the RF.
2. Limitation
Previously we've restricted the RF while creating table in https://github.com/XiaoMi/rdsn/pull/963. These restriction rules will also be applied as the prerequisites before the RF could be updated successfully.
3. Usage
As is described above, the RF could be queried or updated by Pegasus Shell. Here is the patterns of the commands.
It should be noted that to follow the naming convention in Pegasus, replica count is used in the command to represent the RF.
3.1 Query the RF
get_replica_count <table_name>

After this command is executed, the RF of the specified table will be printed if it's successful; Otherwise the error will be printed.
3.2 Update the RF
set_replica_count <table_name> <replica_count>

Since this command will update the RF of a table, in case of incorrect operations a prompt will be printed immediately after the command is executed to ask if the administrator chooses to continue to update the RF. Then, it will tell the command is finished successfully, or, similarly with get_replica_count,  the error will be printed.
4. Implementation
Here we only illustrate the process of set_replica_count, since get_replica_count is quite simple.
4.1 Basic process
The whole process contains the following steps:

Administrator inputs and confirms set_replica_count command by Pegasus Shell;
The set_replica_count request is sent to Primary Meta Server;
Meta Server checks whether the requested RF is valid, whether the table exists, etc.;
Meta Server submits to ZooKeeper to change the meta data;
Meta Server repairs the number of replicas of the specified table to reach the target RF.

As the saying goes, "A picture is worth a thousand words". The process described above can be illustrated as the following sequence diagram:

Obviously it's showed in the sequence diagram that:

Both partition-level and table-level meta data should be updated, with partition-levels first and table-level successively;
Each partition is set one by one, and all the meta data are updated asynchronously.

As for the 2 levels of meta data, see the following section for details.
4.2 Update the meta data
An example is given here to describe both partition-level and table-level meta data.
Suppose that there is a table named test2 with id 5.
First we can list all the tables in the ZooKeeper Shell:
[zk: localhost:2181(CONNECTED) 5] ls /pegasus_cluster/apps/5
[0, 1, 2, 3, 4, 5, 6, 7]

Then, table-level meta data of the target table (id 5) can be found as below, where max_replica_count is the RF to be updated:
[zk: localhost:2181(CONNECTED) 4] get /pegasus_cluster/apps/5
{"status":"app_status::AS_AVAILABLE","app_type":"pegasus","app_name":"test2","app_id":5,"partition_count":8,"envs":{},"is_stateful":1,"max_replica_count":2,"expire_second":0,"create_second":1611726087,"drop_second":0,"duplicating":0,"init_partition_count":-1}

Under table-level is the partition-level meta data showed as below where there is also a max_replica_count:
[zk: localhost:2181(CONNECTED) 3] get /pegasus_cluster/apps/5/0
{"pid":"5.0","ballot":11,"max_replica_count":2,"primary":"<IP_ADDRESS>:34801","secondaries":["<IP_ADDRESS>:34801"],"last_drops":[],"last_committed_decree":3799,"partition_flags":0}

4.3 Repair the number of replicas to reach RF
In Meta Server there is some thread in charge of detecting periodically (each lb_interval_ms), trying to repair the number of replicas of the table to reach the demanded RF.
Thus we can draw a conclusion that it will take about lb_interval_ms (typically 10s) from when the meta data is updated until the number of replicas is repaired.
5. Plan
This feature has been implemented and tested, then released as the internal version. In the internal release, https://github.com/XiaoMi/rdsn/pull/963 and this feature  are in the same PR. I split the monolithic PR into 2 independent ones:

The first PR whose purpose is to provide some rules used to check whether an RF is valid, has been committed and merged.
And another PR will be this feature.

However there are still some tasks that haven't be done. Previously all of our scenarios are to increase the RF. We haven't tested decreasing the RF.
Also, after the RF is decreased, there are 2 situations:

No node is removed, and then the cluster is balanced;
One or more nodes are removed, then the cluster is balanced.

For both situations, some operations should be tested together as a whole set of steps, such as scripts/pegasus_offline_node_list.sh.
Therefore after the unfinished jobs are done, this feature will be committed.

Can load balancer help to recover the cluster after RF changes?

Good enhancement~
I have some questions about updating table replica count:

There're also cluster-level config called max_replica_count, will it be confict if cluster max_replica_count is not equal to table max_replica_count? Will important features work such as user read/write, learn, load balance, rolling_update?
Table app_info file will store in each replica disk (_dir/.app_info), if max_replica_count is updated, this file should also be updated, how to update it?
In your description, cure (executed each 10s) will delete redunant replica or add lacking replica. Could you please add some manual test results?

Expecting your reply~

Can load balancer help to recover the cluster after RF changes?

It has no business with load balance when RF  changed

Each partition is set one by one, and all the meta data are updated asynchronously.

Does it has problem when each partition and meta data updated asynchronously?  Assuming that we are changing RF from 1 to 3. Because they are updated asynchronously, there will be a moment that partition 0 has 3 replicas and partition 1 has only one replica. Does it work well?

Good enhancement~
I have some questions about updating table replica count:

There're also cluster-level config called max_replica_count, will it be confict if cluster max_replica_count is not equal to table max_replica_count? Will important features work such as user read/write, learn, load balance, rolling_update?
Table app_info file will store in each replica disk (_dir/.app_info), if max_replica_count is updated, this file should also be updated, how to update it?
In your description, cure (executed each 10s) will delete redunant replica or add lacking replica. Could you please add some manual test results?

Expecting your reply~

Ok, let's discuss one by one:

Does cluster-level config mean max_allowed_replica_count in https://github.com/XiaoMi/rdsn/pull/963 ? Actually max_replica_count is the RF, and max_allowed_replica_count is a the upper bound to which an RF can be set. Or we should rename max_replica_count as replication_factor to avoid misunderstanding ?
Good question ! This implementation has't written into .app_info, this is a bug ! To guarantee the atomicity, I think first the request set_replica_count can be replicated to each replica; Received the request, each replica writes it into slog, and then writes the new max_replica_count into .app_info. Even if a replica server fails during the process, the request will not be lost.
As is described in this issue, increasing max_replica_count has been tested sufficiently, however decreasing max_replica_count has not been tested. I'll test decreasing later to see if redundant replica can be processed correctly.

Can load balancer help to recover the cluster after RF changes?

It has no business with load balance when RF changed

Yeah, as @levy5307 has said, I can explain this.
Actually set_replica_count just sets the meta data held by ZooKeeper. The number of replicas is repaired by cure, which is also mentioned by @hycdong.
For example, if we want to increase max_replica_count, say, from 2 to 3. A secondary should be added to reach RF 3. If  there's no dropped node, it will be added by emergency; Otherwise, a dropped node will be selected to deploy the replica. For details please see partition_guardian::on_missing_secondary.

Each partition is set one by one, and all the meta data are updated asynchronously.

Does it has problem when each partition and meta data updated asynchronously? Assuming that we are changing RF from 1 to 3. Because they are updated asynchronously, there will be a moment that partition 0 has 3 replicas and partition 1 has only one replica. Will it work well?

This is a problem of atomicity as has been discussed with @hycdong . Actually all the table-level and partition-level meta data, together with .app_info should be updated in a "transaction". Unfortunately, the whole process is not atomic except that we can introduce a mechanism to guarantee all the operations can be managed in a way of transaction.
However, in practice the update process of all meta data is very fast, usually finished within a second.

Good enhancement~
I have some questions about updating table replica count:

There're also cluster-level config called max_replica_count, will it be confict if cluster max_replica_count is not equal to table max_replica_count? Will important features work such as user read/write, learn, load balance, rolling_update?
Table app_info file will store in each replica disk (_dir/.app_info), if max_replica_count is updated, this file should also be updated, how to update it?
In your description, cure (executed each 10s) will delete redunant replica or add lacking replica. Could you please add some manual test results?

Expecting your reply~

Ok, let's discuss one by one:

Does cluster-level config mean max_allowed_replica_count in feat: restrict the replication factor while creating app XiaoMi/rdsn#963 ? Actually max_replica_count is the RF, and max_allowed_replica_count is a the upper bound to which an RF can be set. Or we should rename max_replica_count as replication_factor to avoid misunderstanding ?
Good question ! This implementation has't written into .app_info, this is a bug ! To guarantee the atomicity, I think first the request set_replica_count can be replicated to each replica; Received the request, each replica writes it into slog, and then writes the new max_replica_count into .app_info. Even if a replica server fails during the process, the request will not be lost.
As is described in this issue, increasing max_replica_count has been tested sufficiently, however decreasing max_replica_count has not been tested. I'll test decreasing later to see if redundant replica can be processed correctly.

Thanks for your reply~ Continue to discuss:

I mean meta option called max_replicas_in_group , you can reference it here:
https://github.com/XiaoMi/rdsn/blob/289eb4609ae781e3bcc6bbc6bfeae64dfd8fa785/src/meta/meta_options.cpp#L160-L161
I suggest that you can update max_replica_count through on_config_sync into .app_info
I think decreasing replica count may have some problems, expecting your test result~

Good enhancement~
I have some questions about updating table replica count:

There're also cluster-level config called max_replica_count, will it be confict if cluster max_replica_count is not equal to table max_replica_count? Will important features work such as user read/write, learn, load balance, rolling_update?
Table app_info file will store in each replica disk (_dir/.app_info), if max_replica_count is updated, this file should also be updated, how to update it?
In your description, cure (executed each 10s) will delete redunant replica or add lacking replica. Could you please add some manual test results?

Expecting your reply~

Ok, let's discuss one by one:

Does cluster-level config mean max_allowed_replica_count in feat: restrict the replication factor while creating app XiaoMi/rdsn#963 ? Actually max_replica_count is the RF, and max_allowed_replica_count is a the upper bound to which an RF can be set. Or we should rename max_replica_count as replication_factor to avoid misunderstanding ?
Good question ! This implementation has't written into .app_info, this is a bug ! To guarantee the atomicity, I think first the request set_replica_count can be replicated to each replica; Received the request, each replica writes it into slog, and then writes the new max_replica_count into .app_info. Even if a replica server fails during the process, the request will not be lost.
As is described in this issue, increasing max_replica_count has been tested sufficiently, however decreasing max_replica_count has not been tested. I'll test decreasing later to see if redundant replica can be processed correctly.

Thanks for your reply~ Continue to discuss:

I mean meta option called max_replicas_in_group , you can reference it here:
https://github.com/XiaoMi/rdsn/blob/289eb4609ae781e3bcc6bbc6bfeae64dfd8fa785/src/meta/meta_options.cpp#L160-L161
I suggest that you can update max_replica_count through on_config_sync into .app_info
I think decreasing replica count may have some problems, expecting your test result~

Thanks for suggestion !

I think we can eliminate max_replicas_in_group and turn to partition-level max_replica_count. I'll explain this as below:

Now max_replicas_in_group is only used to update config_context::MAX_REPLICA_COUNT_IN_GRROUP; And config_context::MAX_REPLICA_COUNT_IN_GRROUP is only used in:
void config_context::check_size()
{
    // when add learner, it is possible that replica_count > max_replica_count, so we
    // need to remove things from dropped only when it's not empty.
    while (replica_count(*config_owner) + dropped.size() > MAX_REPLICA_COUNT_IN_GRROUP &&
           !dropped.empty()) {
        dropped.erase(dropped.begin());
        prefered_dropped = (int)dropped.size() - 1;
    }
}

However, MAX_REPLICA_COUNT_IN_GRROUP can be replaced with partition-level max_replica_count as below:
void config_context::check_size()
{
    // when add learner, it is possible that replica_count > max_replica_count, so we
    // need to remove things from dropped only when it's not empty.
    while (replica_count(*config_owner) + dropped.size() > config_owner->max_replica_count &&
           !dropped.empty()) {
        dropped.erase(dropped.begin());
        prefered_dropped = (int)dropped.size() - 1;
    }
}

The partition_configuration pointed by config_owner is updated in server_state::on_update_configuration_on_remote_reply, which will be called by set_replica_count.

Good idea ! Each config_sync_interval_ms (typically 30 ms) the replica server requests query_configuration_by_node to the meta server; Received the response, if the ballot of the response is newer than the local, the replica server will update every table-level and partition-level max_replica_count to the one of the response. Is it right ?

Actually decreasing replica count has been forbidden till now by us. Decreasing is more error-prone than increasing. If we decide to support decreasing replica count, I'll test sufficiently.

Good enhancement~
I have some questions about updating table replica count:

There're also cluster-level config called max_replica_count, will it be confict if cluster max_replica_count is not equal to table max_replica_count? Will important features work such as user read/write, learn, load balance, rolling_update?
Table app_info file will store in each replica disk (_dir/.app_info), if max_replica_count is updated, this file should also be updated, how to update it?
In your description, cure (executed each 10s) will delete redunant replica or add lacking replica. Could you please add some manual test results?

Expecting your reply~

Ok, let's discuss one by one:

Does cluster-level config mean max_allowed_replica_count in feat: restrict the replication factor while creating app XiaoMi/rdsn#963 ? Actually max_replica_count is the RF, and max_allowed_replica_count is a the upper bound to which an RF can be set. Or we should rename max_replica_count as replication_factor to avoid misunderstanding ?
Good question ! This implementation has't written into .app_info, this is a bug ! To guarantee the atomicity, I think first the request set_replica_count can be replicated to each replica; Received the request, each replica writes it into slog, and then writes the new max_replica_count into .app_info. Even if a replica server fails during the process, the request will not be lost.
As is described in this issue, increasing max_replica_count has been tested sufficiently, however decreasing max_replica_count has not been tested. I'll test decreasing later to see if redundant replica can be processed correctly.

Thanks for your reply~ Continue to discuss:

I mean meta option called max_replicas_in_group , you can reference it here:
https://github.com/XiaoMi/rdsn/blob/289eb4609ae781e3bcc6bbc6bfeae64dfd8fa785/src/meta/meta_options.cpp#L160-L161
I suggest that you can update max_replica_count through on_config_sync into .app_info
I think decreasing replica count may have some problems, expecting your test result~

Thanks for suggestion !

I think we can eliminate max_replicas_in_group and turn to partition-level max_replica_count. I'll explain this as below:

Now max_replicas_in_group is only used to update config_context::MAX_REPLICA_COUNT_IN_GRROUP; And config_context::MAX_REPLICA_COUNT_IN_GRROUP is only used in:
void config_context::check_size()
{
    // when add learner, it is possible that replica_count > max_replica_count, so we
    // need to remove things from dropped only when it's not empty.
    while (replica_count(*config_owner) + dropped.size() > MAX_REPLICA_COUNT_IN_GRROUP &&
           !dropped.empty()) {
        dropped.erase(dropped.begin());
        prefered_dropped = (int)dropped.size() - 1;
    }
}

However, MAX_REPLICA_COUNT_IN_GRROUP can be replaced with partition-level max_replica_count as below:
void config_context::check_size()
{
    // when add learner, it is possible that replica_count > max_replica_count, so we
    // need to remove things from dropped only when it's not empty.
    while (replica_count(*config_owner) + dropped.size() > config_owner->max_replica_count &&
           !dropped.empty()) {
        dropped.erase(dropped.begin());
        prefered_dropped = (int)dropped.size() - 1;
    }
}

The partition_configuration pointed by config_owner is updated in server_state::on_update_configuration_on_remote_reply, which will be called by set_replica_count.

Good idea ! Each config_sync_interval_ms (typically 30 ms) the replica server requests query_configuration_by_node to the meta server; Received the response, if the ballot of the response is newer than the local, the replica server will update every table-level and partition-level max_replica_count to the one of the response. Is it right ?
Actually decreasing replica count has been forbidden till now by us. Decreasing is more error-prone than increasing. If we decide to support decreasing replica count, I'll test sufficiently.

Thanks for your reply~

If you have already checked that table level replica_count can replace cluster max_replicas_in_group, just go ahead~
Yes, I recommend update .app_info file through on_config_sync rpc
Okay. How about update this issue title from update the replication factor into increase the replication factor? As the decreasing is not supported.

Good enhancement~
I have some questions about updating table replica count:

There're also cluster-level config called max_replica_count, will it be confict if cluster max_replica_count is not equal to table max_replica_count? Will important features work such as user read/write, learn, load balance, rolling_update?
Table app_info file will store in each replica disk (_dir/.app_info), if max_replica_count is updated, this file should also be updated, how to update it?
In your description, cure (executed each 10s) will delete redunant replica or add lacking replica. Could you please add some manual test results?

Expecting your reply~

Ok, let's discuss one by one:

Does cluster-level config mean max_allowed_replica_count in feat: restrict the replication factor while creating app XiaoMi/rdsn#963 ? Actually max_replica_count is the RF, and max_allowed_replica_count is a the upper bound to which an RF can be set. Or we should rename max_replica_count as replication_factor to avoid misunderstanding ?
Good question ! This implementation has't written into .app_info, this is a bug ! To guarantee the atomicity, I think first the request set_replica_count can be replicated to each replica; Received the request, each replica writes it into slog, and then writes the new max_replica_count into .app_info. Even if a replica server fails during the process, the request will not be lost.
As is described in this issue, increasing max_replica_count has been tested sufficiently, however decreasing max_replica_count has not been tested. I'll test decreasing later to see if redundant replica can be processed correctly.

Thanks for your reply~ Continue to discuss:

I mean meta option called max_replicas_in_group , you can reference it here:
https://github.com/XiaoMi/rdsn/blob/289eb4609ae781e3bcc6bbc6bfeae64dfd8fa785/src/meta/meta_options.cpp#L160-L161
I suggest that you can update max_replica_count through on_config_sync into .app_info
I think decreasing replica count may have some problems, expecting your test result~

Thanks for suggestion !

I think we can eliminate max_replicas_in_group and turn to partition-level max_replica_count. I'll explain this as below:

Now max_replicas_in_group is only used to update config_context::MAX_REPLICA_COUNT_IN_GRROUP; And config_context::MAX_REPLICA_COUNT_IN_GRROUP is only used in:
void config_context::check_size()
{
    // when add learner, it is possible that replica_count > max_replica_count, so we
    // need to remove things from dropped only when it's not empty.
    while (replica_count(*config_owner) + dropped.size() > MAX_REPLICA_COUNT_IN_GRROUP &&
           !dropped.empty()) {
        dropped.erase(dropped.begin());
        prefered_dropped = (int)dropped.size() - 1;
    }
}

However, MAX_REPLICA_COUNT_IN_GRROUP can be replaced with partition-level max_replica_count as below:
void config_context::check_size()
{
    // when add learner, it is possible that replica_count > max_replica_count, so we
    // need to remove things from dropped only when it's not empty.
    while (replica_count(*config_owner) + dropped.size() > config_owner->max_replica_count &&
           !dropped.empty()) {
        dropped.erase(dropped.begin());
        prefered_dropped = (int)dropped.size() - 1;
    }
}

The partition_configuration pointed by config_owner is updated in server_state::on_update_configuration_on_remote_reply, which will be called by set_replica_count.

Good idea ! Each config_sync_interval_ms (typically 30 ms) the replica server requests query_configuration_by_node to the meta server; Received the response, if the ballot of the response is newer than the local, the replica server will update every table-level and partition-level max_replica_count to the one of the response. Is it right ?
Actually decreasing replica count has been forbidden till now by us. Decreasing is more error-prone than increasing. If we decide to support decreasing replica count, I'll test sufficiently.

Thanks for your reply~

If you have already checked that table level replica_count can replace cluster max_replicas_in_group, just go ahead~
Yes, I recommend update .app_info file through on_config_sync rpc
Okay. How about update this issue title from update the replication factor into increase the replication factor? As the decreasing is not supported.

Ok, we can first support increasing the replication factor. Decreasing can be supported later if needed.

Good enhancement~
I have some questions about updating table replica count:

There're also cluster-level config called max_replica_count, will it be confict if cluster max_replica_count is not equal to table max_replica_count? Will important features work such as user read/write, learn, load balance, rolling_update?
Table app_info file will store in each replica disk (_dir/.app_info), if max_replica_count is updated, this file should also be updated, how to update it?
In your description, cure (executed each 10s) will delete redunant replica or add lacking replica. Could you please add some manual test results?

Expecting your reply~

Ok, let's discuss one by one:

Does cluster-level config mean max_allowed_replica_count in feat: restrict the replication factor while creating app XiaoMi/rdsn#963 ? Actually max_replica_count is the RF, and max_allowed_replica_count is a the upper bound to which an RF can be set. Or we should rename max_replica_count as replication_factor to avoid misunderstanding ?
Good question ! This implementation has't written into .app_info, this is a bug ! To guarantee the atomicity, I think first the request set_replica_count can be replicated to each replica; Received the request, each replica writes it into slog, and then writes the new max_replica_count into .app_info. Even if a replica server fails during the process, the request will not be lost.
As is described in this issue, increasing max_replica_count has been tested sufficiently, however decreasing max_replica_count has not been tested. I'll test decreasing later to see if redundant replica can be processed correctly.

Thanks for your reply~ Continue to discuss:

I mean meta option called max_replicas_in_group , you can reference it here:
https://github.com/XiaoMi/rdsn/blob/289eb4609ae781e3bcc6bbc6bfeae64dfd8fa785/src/meta/meta_options.cpp#L160-L161
I suggest that you can update max_replica_count through on_config_sync into .app_info
I think decreasing replica count may have some problems, expecting your test result~

Thanks for suggestion !

I think we can eliminate max_replicas_in_group and turn to partition-level max_replica_count. I'll explain this as below:

Now max_replicas_in_group is only used to update config_context::MAX_REPLICA_COUNT_IN_GRROUP; And config_context::MAX_REPLICA_COUNT_IN_GRROUP is only used in:
void config_context::check_size()
{
    // when add learner, it is possible that replica_count > max_replica_count, so we
    // need to remove things from dropped only when it's not empty.
    while (replica_count(*config_owner) + dropped.size() > MAX_REPLICA_COUNT_IN_GRROUP &&
           !dropped.empty()) {
        dropped.erase(dropped.begin());
        prefered_dropped = (int)dropped.size() - 1;
    }
}

However, MAX_REPLICA_COUNT_IN_GRROUP can be replaced with partition-level max_replica_count as below:
void config_context::check_size()
{
    // when add learner, it is possible that replica_count > max_replica_count, so we
    // need to remove things from dropped only when it's not empty.
    while (replica_count(*config_owner) + dropped.size() > config_owner->max_replica_count &&
           !dropped.empty()) {
        dropped.erase(dropped.begin());
        prefered_dropped = (int)dropped.size() - 1;
    }
}

The partition_configuration pointed by config_owner is updated in server_state::on_update_configuration_on_remote_reply, which will be called by set_replica_count.

Good idea ! Each config_sync_interval_ms (typically 30 ms) the replica server requests query_configuration_by_node to the meta server; Received the response, if the ballot of the response is newer than the local, the replica server will update every table-level and partition-level max_replica_count to the one of the response. Is it right ?
Actually decreasing replica count has been forbidden till now by us. Decreasing is more error-prone than increasing. If we decide to support decreasing replica count, I'll test sufficiently.

Thanks for your reply~

If you have already checked that table level replica_count can replace cluster max_replicas_in_group, just go ahead~
Yes, I recommend update .app_info file through on_config_sync rpc
Okay. How about update this issue title from update the replication factor into increase the replication factor? As the decreasing is not supported.

Ok, we can first support increasing the replication factor. Decreasing can be supported later if needed.

I aggree with you about renaming max_replica_count to replication_factor : )

Besides, I agree with you about removing max_replicas_in_group, but we should keep config_context::MAX_REPLICA_COUNT_IN_GRROUP as partition-level max_replica_count + 1

Good enhancement~
I have some questions about updating table replica count:

There're also cluster-level config called max_replica_count, will it be confict if cluster max_replica_count is not equal to table max_replica_count? Will important features work such as user read/write, learn, load balance, rolling_update?
Table app_info file will store in each replica disk (_dir/.app_info), if max_replica_count is updated, this file should also be updated, how to update it?
In your description, cure (executed each 10s) will delete redunant replica or add lacking replica. Could you please add some manual test results?

Expecting your reply~

Ok, let's discuss one by one:

Does cluster-level config mean max_allowed_replica_count in feat: restrict the replication factor while creating app XiaoMi/rdsn#963 ? Actually max_replica_count is the RF, and max_allowed_replica_count is a the upper bound to which an RF can be set. Or we should rename max_replica_count as replication_factor to avoid misunderstanding ?
Good question ! This implementation has't written into .app_info, this is a bug ! To guarantee the atomicity, I think first the request set_replica_count can be replicated to each replica; Received the request, each replica writes it into slog, and then writes the new max_replica_count into .app_info. Even if a replica server fails during the process, the request will not be lost.
As is described in this issue, increasing max_replica_count has been tested sufficiently, however decreasing max_replica_count has not been tested. I'll test decreasing later to see if redundant replica can be processed correctly.

Thanks for your reply~ Continue to discuss:

I mean meta option called max_replicas_in_group , you can reference it here:
https://github.com/XiaoMi/rdsn/blob/289eb4609ae781e3bcc6bbc6bfeae64dfd8fa785/src/meta/meta_options.cpp#L160-L161
I suggest that you can update max_replica_count through on_config_sync into .app_info
I think decreasing replica count may have some problems, expecting your test result~

Thanks for suggestion !

I think we can eliminate max_replicas_in_group and turn to partition-level max_replica_count. I'll explain this as below:

Now max_replicas_in_group is only used to update config_context::MAX_REPLICA_COUNT_IN_GRROUP; And config_context::MAX_REPLICA_COUNT_IN_GRROUP is only used in:
void config_context::check_size()
{
    // when add learner, it is possible that replica_count > max_replica_count, so we
    // need to remove things from dropped only when it's not empty.
    while (replica_count(*config_owner) + dropped.size() > MAX_REPLICA_COUNT_IN_GRROUP &&
           !dropped.empty()) {
        dropped.erase(dropped.begin());
        prefered_dropped = (int)dropped.size() - 1;
    }
}

However, MAX_REPLICA_COUNT_IN_GRROUP can be replaced with partition-level max_replica_count as below:
void config_context::check_size()
{
    // when add learner, it is possible that replica_count > max_replica_count, so we
    // need to remove things from dropped only when it's not empty.
    while (replica_count(*config_owner) + dropped.size() > config_owner->max_replica_count &&
           !dropped.empty()) {
        dropped.erase(dropped.begin());
        prefered_dropped = (int)dropped.size() - 1;
    }
}

The partition_configuration pointed by config_owner is updated in server_state::on_update_configuration_on_remote_reply, which will be called by set_replica_count.

Good idea ! Each config_sync_interval_ms (typically 30 ms) the replica server requests query_configuration_by_node to the meta server; Received the response, if the ballot of the response is newer than the local, the replica server will update every table-level and partition-level max_replica_count to the one of the response. Is it right ?
Actually decreasing replica count has been forbidden till now by us. Decreasing is more error-prone than increasing. If we decide to support decreasing replica count, I'll test sufficiently.

Thanks for your reply~

If you have already checked that table level replica_count can replace cluster max_replicas_in_group, just go ahead~
Yes, I recommend update .app_info file through on_config_sync rpc
Okay. How about update this issue title from update the replication factor into increase the replication factor? As the decreasing is not supported.

Ok, we can first support increasing the replication factor. Decreasing can be supported later if needed.

Now our implementation has supported to decrease the replication factor.
It is noticeable that the redundant replica data will be garbage-collected only when meta level is lively, which means set_meta_level lively should be executed.


Links to hello world examples broken

In example's readme webpack/browserify hello world examples are described. Clicking on links results in 404, e.g.:
https://github.com/uber/deck.gl/blob/master/examples/hello-world-webpack2/README.md
Can't find these examples anywhere in the repo.

Links fixed. Moved to examples/react folder. If things don't work on master, try 4.1-release branch.


Normalization not working for valueschema and propertyschema

Hey there,
I just stumbled over the following issue: When using 'coerce' in valueschema or propertyschema, validator.document does not get updated accordingly. Consider the following (currently failing) tests:

    def test_coerce_in_valueschema(self):
        schema = {'thing': {'type': 'dict',
                            'valueschema': {'coerce': int}}}
        v = Validator(schema)
        self.assertEqual(v.validated({'thing': {'amount': '2'}})
                                     ['thing']['amount'], 2)  # noqa

    def test_coerce_in_propertyschema(self):
        schema = {'thing': {'type': 'dict',
                            'propertyschema': {'coerce': int}}}
        v = Validator(schema)
        self.assertEqual(v.validated({'thing': {'5': 'foo'}})
                                     ['thing'][5], 'foo')  # noqa

I don't know whether this is actually meaningful in case of propertyschema. For valueschema, normalization should be possible IMO.

that would work:
    schema = {'thing': {'type': 'dict',
                        'allow_unknown': {'coerce': int},
                        'schema': {}}}

but i see that it's not straightforward.
normalization of keys can be achieved with a 'rename_handler': int.
to streamline both, the need to give an empty schema would have to be removed.

Thank you! This solves my problem for now :).