Random() function giving an error with LUW DB2

Hi Team,
We are comparing the data between LUW DB2 and BQ and our validation are in YAML files. While executing the ROW validation, we gave use_random_rows = True and random_row_batch_size = '1000' and it fails to execute the random rows selection.
Points for your reference

We are using Cloud Run Job to execute the DVT
Our input validation files are in YAML format and stored in GCS bucket
Our connection json is also in GCS and we are overwriting PSO_DV_CONN_HOME environment variable and use it to execute cloud run job

Error:
08/22/2024 05:13:45 PM-ERROR: Error (ibm_db_dbi.ProgrammingError) ibm_db_dbi::ProgrammingError: Statement Execute Failed: [IBM][CLI Driver][DB2] SQL0440N No authorized routine named "RANDOM" of type "" having compatible arguments was found. SQLSTATE=42884 SQLCODE=-440
Input YAML file:
result_handler:
project_id: <project_id>
table_id: <results_table_id>
type: BigQuery
source: db2
target: bq
validations:

case_insensitive_match: false
comparison_fields:

cast: null
field_alias: ADDR_TYPE_ID_cell_level_validation
source_column: addr_type_id
target_column: ADDR_TYPE_ID
cast: null
field_alias: LANG_CD_cell_level_validation
source_column: lang_cd
target_column: LANG_CD
cast: null
field_alias: ADDR_TYPE_TEXT_cell_level_validation
source_column: addr_type_text
target_column: ADDR_TYPE_TEXT
filter_status: null
filters: []
format: table
labels: []
primary_keys:
cast: null
field_alias: ADDR_TYPE_ID_pk
source_column: addr_type_id
target_column: ADDR_TYPE_ID
cast: null
field_alias: LANG_CD_pk
source_column: lang_cd
target_column: LANG_CD
cast: null
field_alias: ADDR_TYPE_TEXT_pk
source_column: addr_type_text
target_column: ADDR_TYPE_TEXT
random_row_batch_size: '1000'
schema_name: DB2I0
table_name: TPY_ADDR_TYPE_DESC
target_schema_name: dp-dev-derived-ea49.product_ref_geographic
target_table_name: address_type_desc
threshold: 0
trim_string_pks: false
type: Row
use_random_rows: true

Hi @manojredward! What is the version of your DB2 instance? The error is indicating it could not found a RANDOM function to use without parameters being given.
I checked in our DVT test instance (LINUXX8664 <IP_ADDRESS>) and it works this way by using a default seed number, according to the documentation (https://www.ibm.com/docs/en/db2/11.5?topic=module-random-function).

But as we searched from version 11.1 and before, the function requires a seed number as parameter:

11.0: https://www.ibm.com/docs/en/db2/11.1?topic=SSEPGG_11.1.0/com.ibm.dwe.cubemdx.doc/mdx_random.htm
10.5: https://www.ibm.com/docs/en/db2/10.5?topic=functions-random-function-mdx

If that's your case we indeed need to make a code change, thanks for reaching out!

Thanks @helensilva14 for your response.
I will check the DB2 version and get back to you. Meanwhile, Could you please guide me how to pass the seed value in YAML file. Right now, we are passing something like mentioned below,
random_row_batch_size: '1000'
type: Row
use_random_rows: true

As of now, it is not possible to pass a seed number either using DVT CLI or YAML config file.
We'll work on an initial version of a code change to pass a seed direct in our code and ask your help to test it soon!

Hello @helensilva14
We are using the below DB2 version, could you please check this?
AIX v<IP_ADDRESS>
Mainframe V12R1M505

Hi @manojredward! Can you please try to execute the following SQL queries in your instance and let us know if they run without errors? Below you can see the output in our DB2 instance (LINUXX8664 <IP_ADDRESS>):

Without seed (using default value): db2 "select random() from sysibm.sysdummy1;"
With seed value: db2 "select random(2) from sysibm.sysdummy1;"

Thanks @helensilva14, I will check and let you know asap. We are using AIX v<IP_ADDRESS> Mainframe V12R1M505 DB2, hope this is fine. If we need to install any packages, please let us know.

Hi @helensilva14,
We tried to run the below queries in our AIX DB2 and got an error as
SQL Error [42884]: NO AUTHORIZED RANDOM NAMED  HAVING COMPATIBLE ARGUMENTS WAS FOUND. SQLCODE=-440, SQLSTATE=42884, DRIVER=4.33.31
Attached image for your reference

Hi @manojredward, thanks for sharing the error message and the images!
We did more research and might be the case that your instance/flavor only recognizes the function as RAND(), such as in this example: https://db2-sql-cookbook.org/#_getting_n_random_rows
Could you please try to execute the following query versions and let us know if they work?

select rand() from sysibm.sysdummy1;
select rand(2) from sysibm.sysdummy1;

Hi @helensilva14
We have tried with rand() command and it is working as expected in our DB2. Please find the screenshot below.

Great to hear it! We updated our code to always use RAND() since this method is always available in all DB2 versions, which is not the case for RANDOM().

Great, Thanks, Please let us know once the code is merged, we will recreate image and do our testing.

@helensilva14
By following the thread, we were actually cloned the code base from branch: 1239-random-function-error-luw-db2 and created an image and executed our validation and we are getting the below error.
Error (ibm_db_dbi.ProgrammingError) ibm_db_dbi::ProgrammingError: Binding Error: [IBM][CLI Driver][DB2] SQL0440N  No authorized routine named "SETSEED" of type "" having compatible arguments was found.  SQLSTATE=42884 SQLCODE=-440
We have added the below changes to our yaml file,
random_row_batch_size: '2'
use_random_rows: true
Could you please take a look on seed function and let us know the updates.

Hi @manojredward, thank you very much for your testing! I didn't face this error in our instance.
Anyways, I added a new line to our code to always set a default seed value before using the RAND() function.
Could you repeat the steps and test again the branch 1239-random-function-error-luw-db2?

Thanks @helensilva14 for making those changes, I will recreate the image with this branch and test.

Hi @helensilva14, We have testing the code from issue1239-random-function-error-luw-db2 branch with two yaml's and both are give me the same error.
YAML #1:
random_row_batch_size: '2'
type: Row
use_random_rows: true
YAML #2:
random_row_batch_size: null
type: Row
use_random_rows: true
Error:
09/03/2024 12:48:33 PM-ERROR: Error (ibm_db_dbi.ProgrammingError) ibm_db_dbi::ProgrammingError: Statement Execute Failed: [IBM][CLI Driver][DB2] SQL0583N  The use of routine or expression "RAND()" is invalid because it is not deterministic or has an external action.  SQLSTATE=42845 SQLCODE=-583

Hello @manojredward, thanks for reporting this new error!
I tried a new approach of setting the seed value for the RAND() function itself and not externally.
Could you please repeat the steps and test again the branch issue1239-random-function-error-luw-db2?

Hi @helensilva14, Please find the below error when we test with the updated code from issue1239-random-function-error-luw-db2.
Error:
09/05/2024 06:49:34 AM-ERROR: Error (ibm_db_dbi.ProgrammingError) ibm_db_dbi::ProgrammingError: Binding Error: [IBM][CLI Driver][DB2] SQL0418N  The statement was not processed because the statement contains an invalid use of one of the following: an untyped parameter marker, the DEFAULT keyword, or a null value.  SQLSTATE=42610 SQLCODE=-418
YAML:
random_row_batch_size: '2'
type: Row
use_random_rows: true

Hello @helensilva14 and @luispavaogoogle
Just want to check, do you have any updates for me?

Hello @helensilva14,  I am working with @manojredward on testing this. We tried testing with the latest pull from the branch issue1239-random-function-error-luw-db2 but it failed with similar error
Error (ibm_db_dbi.ProgrammingError) ibm_db_dbi::ProgrammingError: Binding Error: [IBM][CLI Driver][DB2] SQL0418N  The statement was not processed because the statement contains an invalid use of one of the following: an untyped parameter marker, the DEFAULT keyword, or a null value.  SQLSTATE=42610 SQLCODE=-418
If you could help to check on this?

Hi @Rushikesh-Hande! I was discussing with the other DVT members about your scenario here and how many attempts we already had, so we all agreed it would more efficient for both sides if we try to schedule a troubleshoot/debugging call. It will enable us to check things in real-time and get this issue solved as soon as possible, because our main goal is to unblock your usage!
Could any of you please send an email to<EMAIL_ADDRESS>with your availability slots for this week still and the next one? Please don't forget to provide timezone information. Thank you!

Hello @helensilva14, Please find the attached YAML file that we are using.
row_level_validation_test.txt

Thank you very much @manojredward! While talking to @luispavaogoogle, he asked if you ever tried to run the validations inverting the order, so instead of DB2 as source and BQ as target, you would set BQ as source and DB2 as target.
We are wondering if this might produce a different output, is it possible for you to test it?

Sure @helensilva14, We can do that testing and let you know the output.

Hi,
@helensilva14 - that is a great idea - at least the random function step will work in BigQuery.
If Helen's suggestion does not solve your problem, you can try to figure out how to get the query to find the random rows to execute in DB2. The statement the DVT uses is given below:
data-validation query -c db2 -q 'SELECT t0.addr_type_text, t0.lang_cd FROM (SELECT t1.addr_type_text AS addr_type_text, t1.lang_cd AS lang_cd FROM DB2I0."TPY_ADDR_TYPE_DESC" AS t1) AS t0 ORDER BY rand() ASC LIMIT 25'

This particular query will produce 25 rows. If it fails (as I expect), you will need to check with your DB2 experts - as this query works in our linux DB2. Maybe the user does not privileges to run RAND() or RAND() is not available.
Thanks.
Sundar Mudupalli

Hi @helensilva14
I have checked with swapping source and target and now i able to execute the row level validation. However i am unable to restrict the number of rows to test.
I have tested with a particular table and total number of rows in that table is 22 and execute DVT with random_row_batch_size = '2' and  random_row_batch_size = '1' and both are generated 66 rows in the target BQ table.
It looks like eventhough I have passed random_row_batch_size, it is executing all the records from the table.
Please find the attached YAML test file that I have used to test.
src_bq_target_db2.txt

Hi @sundar-mudupalli-work
I have tried to execute the below query and it failed in DBeaver.
SELECT t0.addr_type_text, t0.lang_cd FROM (SELECT t1.addr_type_text AS addr_type_text, t1.lang_cd AS lang_cd FROM DB2I0."TPY_ADDR_TYPE_DESC" AS t1) AS t0 ORDER BY rand() ASC LIMIT 25
Error:
SQL Error [42845]: THE USE OF FUNCTION OR EXPRESSION RAND() IS INVALID BECAUSE IT IS NOT DETERMINISTIC OR HAS AN EXTERNAL ACTION. SQLCODE=-583, SQLSTATE=42845, DRIVER=4.33.31

data-validation query -c db2 -q 'SELECT t0.addr_type_text, t0.lang_cd FROM (SELECT t1.addr_type_text AS addr_type_text, t1.lang_cd AS lang_cd FROM DB2I0."TPY_ADDR_TYPE_DESC" AS t1) AS t0 ORDER BY rand() ASC LIMIT 25'

Since I don't have access to db2 on z/os - can you execute this query on DBeaver (DB2) to see if it works:
data-validation query -c db2 -q 'SELECT t0.addr_type_text, t0.lang_cd FROM (SELECT t1.addr_type_text AS addr_type_text, t1.lang_cd AS lang_cd, RAND() as randnum FROM DB2I0."TPY_ADDR_TYPE_DESC" AS t1) AS t0 ORDER BY randnum ASC LIMIT 25'

Thank you.

Hi @helensilva14 I have checked with swapping source and target and now i able to execute the row level validation. However i am unable to restrict the number of rows to test.
I have tested with a particular table and total number of rows in that table is 22 and execute DVT with random_row_batch_size = '2' and random_row_batch_size = '1' and both are generated 66 rows in the target BQ table.
It looks like eventhough I have passed random_row_batch_size, it is executing all the records from the table.
Please find the attached YAML test file that I have used to test. src_bq_target_db2.txt

This is really odd - I am able to get the same thing to work in my environment. Can you try self row validation in BigQuery to see if the problem can be reproduced without DB2. For example (please check for typos)
data-validation validate row -sc bq -tc bq -tbls=dp-dev-derived-ea49.product_ref_insur_products.auto_veh_cnt -pk=LANG_CD,VEH_CNT_GRP,GRP_MIN_VALUE,GRP_MAX_VALUE,VEH_CNT_SHRTDESC,VEH_CNT_DESC -comp-fields=LANG_CD,VEH_CNT_GRP,GRP_MIN_VALUE,GRP_MAX_VALUE,VEH_CNT_SHRTDESC,VEH_CNT_DESC -rr -rbs 2

Thank you.

Hi @manojredward, please try to execute the following YAML:
result_handler:
  project_id: dp-dev-processing-4993
  table_id: cdw_dvt_validation.dvt_results_all_tables_sample
  type: BigQuery
source: bq
target: bq
validations:
- case_insensitive_match: false
  comparison_fields:
  - cast: null
    field_alias: LANG_CD_cell_level_validation
    source_column: LANG_CD
    target_column: LANG_CD
  - cast: null
    field_alias: VEH_CNT_GRP_cell_level_validation
    source_column: VEH_CNT_GRP
    target_column: VEH_CNT_GRP
  - cast: null
    field_alias: GRP_MIN_VALUE_cell_level_validation
    source_column: GRP_MIN_VALUE
    target_column: GRP_MIN_VALUE
  - cast: null
    field_alias: GRP_MAX_VALUE_cell_level_validation
    source_column: GRP_MAX_VALUE
    target_column: GRP_MAX_VALUE
  - cast: null
    field_alias: VEH_CNT_SHRTDESC_cell_level_validation
    source_column: VEH_CNT_SHRTDESC
    target_column: VEH_CNT_SHRTDESC
  - cast: null
    field_alias: VEH_CNT_DESC_cell_level_validation
    source_column: VEH_CNT_DESC
    target_column: VEH_CNT_DESC
  filter_status: null
  filters: []
  format: table
  labels:
  - !!python/tuple
    - validation_type
    - cell_level_validation
  - !!python/tuple
    - dag_name
    - dvt_auto_reference_42_tables
  - !!python/tuple
    - stm_file
    - stm_auto_reference_42_tables.json
  primary_keys:
  - cast: null
    field_alias: LANG_CD_pk
    source_column: LANG_CD
    target_column: LANG_CD
  - cast: null
    field_alias: VEH_CNT_GRP_pk
    source_column: VEH_CNT_GRP
    target_column: VEH_CNT_GRP
  - cast: null
    field_alias: GRP_MIN_VALUE_pk
    source_column: GRP_MIN_VALUE
    target_column: GRP_MIN_VALUE
  - cast: null
    field_alias: GRP_MAX_VALUE_pk
    source_column: GRP_MAX_VALUE
    target_column: GRP_MAX_VALUE
  - cast: null
    field_alias: VEH_CNT_SHRTDESC_pk
    source_column: VEH_CNT_SHRTDESC
    target_column: VEH_CNT_SHRTDESC
  - cast: null
    field_alias: VEH_CNT_DESC_pk
    source_column: VEH_CNT_DESC
    target_column: VEH_CNT_DESC
  random_row_batch_size: '2'
  schema_name: dp-dev-derived-ea49.product_ref_insur_products
  table_name: auto_veh_cnt
  target_schema_name: dp-dev-derived-ea49.product_ref_insur_products
  target_table_name: auto_veh_cnt
  threshold: 0
  trim_string_pks: false
  type: Row
  use_random_rows: true

It is based on this other testing YAML below that I created to validate BQ to BQ random row execution. I was able to get the exact number of random rows I provided.
result_handler: {}
source: bq
target: bq
validations:
- case_insensitive_match: false
  comparison_fields:
  - cast: null
    field_alias: station_id_CELL_LEVEL_VALIDATION
    source_column: station_id
    target_column: station_id
  filter_status: null
  filters: []
  format: table
  labels: []
  primary_keys:
  - cast: null
    field_alias: station_id_PK
    source_column: station_id
    target_column: station_id
  random_row_batch_size: '2'
  schema_name: bigquery-public-data.new_york_citibike
  table_name: citibike_stations
  target_schema_name: bigquery-public-data.new_york_citibike
  target_table_name: citibike_stations
  threshold: 0.0
  type: Row
  use_random_rows: true

Hi @helensilva14, Please find the details below,
BQ Table: auto_veh_cnt
DB2 Table: TPV_VEH_CNT
Total no. of rows in both table: 22
Code from this Branch:
https://github.com/GoogleCloudPlatform/professional-services-data-validator/tree/issue1239-random-function-error-luw-db2
Source: BQ
Target: BQ
src_bq_target_bq.yaml
Scenario #1:
random_row_batch_size = 5
132 new rows added
Scenario #2:
random_row_batch_size = 2
66 new rows added
Code from this Master Branch:
Source: BQ
Target: BQ
src_bq_target_bq.yaml
Scenario #1:
random_row_batch_size = 5
132 new rows added
Scenario #2:
random_row_batch_size = 2
66 new rows added
YAML File:
result_handler:
  project_id: dp-dev-processing-4993
  table_id: cdw_dvt_validation.dvt_results_row_v_issue
  type: BigQuery
source: bq
target: bq
validations:
- case_insensitive_match: false
  comparison_fields:
  - cast: null
    field_alias: LANG_CD_cell_level_validation
    source_column: LANG_CD
    target_column: LANG_CD
  - cast: null
    field_alias: VEH_CNT_GRP_cell_level_validation
    source_column: VEH_CNT_GRP
    target_column: VEH_CNT_GRP
  - cast: null
    field_alias: GRP_MIN_VALUE_cell_level_validation
    source_column: GRP_MIN_VALUE
    target_column: GRP_MIN_VALUE
  - cast: null
    field_alias: GRP_MAX_VALUE_cell_level_validation
    source_column: GRP_MAX_VALUE
    target_column: GRP_MAX_VALUE
  - cast: null
    field_alias: VEH_CNT_SHRTDESC_cell_level_validation
    source_column: VEH_CNT_SHRTDESC
    target_column: VEH_CNT_SHRTDESC
  - cast: null
    field_alias: VEH_CNT_DESC_cell_level_validation
    source_column: VEH_CNT_DESC
    target_column: VEH_CNT_DESC
  filter_status: null
  filters: []
  format: table
  labels:
  - !!python/tuple
    - validation_type
    - cell_level_validation
  - !!python/tuple
    - dag_name
    - dvt_auto_reference_42_tables
  - !!python/tuple
    - stm_file
    - stm_auto_reference_42_tables.json
  primary_keys:
  - cast: null
    field_alias: LANG_CD_pk
    source_column: LANG_CD
    target_column: LANG_CD
  - cast: null
    field_alias: VEH_CNT_GRP_pk
    source_column: VEH_CNT_GRP
    target_column: VEH_CNT_GRP
  - cast: null
    field_alias: GRP_MIN_VALUE_pk
    source_column: GRP_MIN_VALUE
    target_column: GRP_MIN_VALUE
  - cast: null
    field_alias: GRP_MAX_VALUE_pk
    source_column: GRP_MAX_VALUE
    target_column: GRP_MAX_VALUE
  - cast: null
    field_alias: VEH_CNT_SHRTDESC_pk
    source_column: VEH_CNT_SHRTDESC
    target_column: VEH_CNT_SHRTDESC
  - cast: null
    field_alias: VEH_CNT_DESC_pk
    source_column: VEH_CNT_DESC
    target_column: VEH_CNT_DESC
  random_row_batch_size: '5'
  schema_name: dp-dev-derived-ea49.product_ref_insur_products
  table_name: auto_veh_cnt
  target_schema_name: dp-dev-derived-ea49.product_ref_insur_products
  target_table_name: auto_veh_cnt
  threshold: 0
  trim_string_pks: false
  type: Row
  use_random_rows: true

Hi @manojredward, after chatting again with the team, we finally noticed the real root cause in here! 🚀
Even though you are informing 5 PKs, random rows is only being applied to the first PK (LANG_CD_pk), it's a known limitation we have, this is the open issue to be able to support multiple PKs, because it involves a lot more complexity to create the filters, please take a look here #774.
So it seems your key LANG_CD_pk is not unique enough so that's why you are always getting more values than request in random_row_batch_size. Being aware of it now, try to evaluate if this can be a problem for larger tables or if you could select other PK that could result in less rows.
Thanks again for your collaboration with us and apologies for not realizing it earlier.

Thanks @helensilva14  for checking on this. So the limitation we've encountered concerning the inability to specify a combination of columns as a primary key is specific to BigQuery, or if it applies universally across all source and target systems(even for DB2 as a source) ?

It applies across all of our supported engines

Thanks @helensilva14 for checking on this, whats a recommended way to do cell by cell row validation on sample records which is not having one columns as primary key ?

Hi @Rushikesh-Hande! You can still use this same approach of passing a PK and a random_row_batch_size, but first you need to make an analysis on which column would be the ideal PK in the sense of having more distinct values available.
Take a look of my examples:
db2 "SELECT content, COUNT(*) AS count FROM entries GROUP BY content;"
CONTENT                       COUNT
----------------------------------------
I got here!                    2
Me too!                        3
More data Coming               1
zippp!                         1

DVT is executing, so that's why we get more rows: db2 "SELECT content FROM entries ORDER BY RAND() LIMIT 2;"
CONTENT
----------------------
Me too!
Me too!

Hello @sundar-mudupalli-work
Thanks for checking this. Please find the error details after executing the queries that you specified above.
Query1:
SELECT t0.addr_type_text FROM
(SELECT t1.addr_type_text AS addr_type_text FROM DB2I0."TPY_ADDR_TYPE_DESC" AS t1) AS t0
ORDER BY rand() ASC LIMIT 25;

Error:
SQL Error [42845]: THE USE OF FUNCTION OR EXPRESSION RAND() IS INVALID
BECAUSE IT IS NOT DETERMINISTIC OR HAS AN EXTERNAL ACTION.

SQLCODE=-10, SQLSTATE=42603, DRIVER=4.33.31

Great to hear it! We updated our code to always use RAND() since this method is always available in all DB2 versions, which is not the case for RANDOM().

Closing issue as completed because the initial problem was solved as quoted here. Everything else after that were other issues specific only to their on-prem database and it was not possible to replicate any of them.


npm warnings

npm WARN deprecated<EMAIL_ADDRESS><EMAIL_ADDRESS>is no longer maintained. Upgrade to<EMAIL_ADDRESS>npm WARN engine<EMAIL_ADDRESS>wanted: {"node":">= 0.10.0 <= 0.11.0"} (current: {"node":"4.0.0","npm":"2.14.2"})

Moved to master issues repo: https://github.com/rangle/rangle-starter/issues/11


Issue 28

For formatting purposes, I added some line breaks to the "Working with the MIFC Repo" section of README.md

I pulled the branch and ran make all-all and no problems, this is safe to merge thanks @GregLyonJIFSAN!


preview thumbnails persist after deleting

It appears that certain auto-saved images are persisting even after attempting to delete them. Despite deleting the folder "stable-diffusion-webui\extensions\sd-webui-state-manager" and reinstalling the web-ui-manager, the images remain and cannot be removed.
Allow me to outline the steps to replicate this issue:
When selecting multiple preview images for deletion, such as selecting 3 and clicking delete, the selected files are removed.
However, the subsequent 3 files in the list occupy the vacant slots, still highlighted from the previous selection.
It seems that attempting to delete files highlighted in this manner results in an incomplete deletion process of files that occupy the space that was previously deleted. These files remain stuck on the preview bar and cannot be successfully removed.

To further clarify if you just highlight several images using opt/ctrl + left click and then just keep clicking the trash/delete icon, this will replicate the issue.

i have the same problem


[v1.2] Added new Contact Field (previously messed up the file locations)

Tidied up some code from other contributors

Changes updated with other pull request (Morphed Address field to Venue field)


chore(main): release 2.1.3

:robot: I have created a release beep boop
2.1.3 (2023-11-18)
Bug Fixes

remove test stream update delay (#52) (5d8b2d7)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/rickstaa/livepeer-exporter/releases/tag/v2.1.3 :sunflower:


Fix reg/login validation

Pull Request Checklist
Thank you for contributing to our project! Please ensure the following before submitting your pull request:
Description of Changes
Password validation is now performed during user registration on the User Signup page. A "show password" feature has been added for easier password verification. After registration, users are redirected directly to the home page without needing to log in again. Additionally, the User Signup and User Login pages are no longer accessible once the user is logged in.
Type of Change

[x] Bug fix
[x] New feature
[ ] Documentation update
[ ] Refactoring
[ ] Other (please describe):

Related Issue
Closes: #62
Testing
Demo:
https://github.com/user-attachments/assets/2426e4cb-c675-40a5-a04b-0e2d8c3657eb
Checklist

[x] I have tested my changes and ensured they work as expected.
[ ] I have added necessary documentation (if applicable).
[x] I have reviewed my code for style and linting issues.
[x] I have ensured that any dependent changes are merged before this PR.

Additional Notes

Good work @ArchanRD, keep contributing!


Documentation: remove legacy images

Description
This PR drops a set of legacy images from the documentation, as out of date and no longer referenced.

/doc-build

/test

/merge


Crash on startup

Describe the Bug
When I try to boot up Minecraft with this mod, I crash.
Reproduction Steps

Start up minecraft
???
Crash
...

Expected Result
Game to boot up.
Screenshots and Videos
No response
Crash Report or Log
https://pastebin.com/NQL8uh6s
Operating System
Windows 10
Mod Version
0.5.0i
Minecraft Version
1.19.2
Other Mods
No response
Additional Context
Also affects addons

NoClassDefFoundError: Could not initialize class com.simibubi.create.Create
Your computer may be haunted.
Try reinstalling Create? I really don't know, there's no info here. Try it alone too. If it works then, some mod is doing something really bad.

I will investigate this when I fix an unrelated error

maybe try the latest release

I'll test within the next few days. I have found the cause of the serious bug I was having.

Issue no longer appears to be present.


Campaign defaults

Addresses #1413
Allows user to choose a template response_set when setting up a campaign.

NB - I had some trouble with this earlier today but I was unable to reproduce the error. I'm going to put it down to my environment as when I rebuilt docker instances this all seemed to work fine. Still worth you all keeping an eye on it though!

Coverage decreased (-0.2%) to 81.022% when pulling 55c7a55afbd45903b5ea86f2431323f66d856b30 on digitalWestie:campaign-defaults into 957af451a2a4c657b553ed7806c3a40dd255461f on theodi:staging.

This works up to a point. I've just tried it with a template, running it against the parliament.uk CKAN instance at http://www.data.parliament.uk/
I set up a template that answers the questions to the publisher name and publisher url questions, adds a custom license (Open Parliament License with the URL http://www.parliament.uk/site-information/copyright/open-parliament-licence/), and answers yes whether it's an open license.
I then run the autocertification with the template and the certificates get created (yay!). However, they don't seem to have a level, but there are no questions missing, see:

When I got in to edit one of the certificates, it seems like all should be well, but when I scroll down to the 'Practical Information' section, it looks like the URLs have either been removed, or not added at all:

And:

If I add some URLs there and republish, I get a Bronze certificate as expected

Also, rerunning the certification seems to work too

Wasn't able to replicate the removed URLs issue. Confirmed that passable certificates weren't getting receiving the proper certificate level.

Coverage decreased (-0.2%) to 81.028% when pulling 2e39d13ff1c05874392f2116b82cf5af986ceef2 on digitalWestie:campaign-defaults into 957af451a2a4c657b553ed7806c3a40dd255461f on theodi:staging.

Nice - this works perfectly with the Parliament data. OK to merge?

I think so. I think we just have to be careful what answers we provide in the template given the behaviour around follow up questions like @JeniT suggested. Perhaps there's a way of making this a bit more clever but it works for now. :+1:

Kool and the gang. I'll merge now 😄


Added missing line in code examples for plugin

I noticed there was a missing line in the documentation when creating a plugin. I added the line so it will be easier for others to follow the example.

Thanks Johan!


Switch Bitfields to Little Indian - (See Spec PR #575)

Description
As a client implementor, I want bitfields to be little endian so that serialization of bitfields is compliant with the spec..
Acceptance Criteria

Bits in bitfield are stored in little endian byte order. This requires updating the
get_bitfield_bit function.
Changes are adequately tested.

Additional Information
https://github.com/ethereum/eth2.0-specs/pull/575

Resolved by #456


[frontend] the mobile login page with custom CSS doesn't look right

We should disable custom CSS for mobile devices

@leo220yuyaodog


Fix security issues wheel, urllib3, pygments and zipp. Bumping versions.

Bump versions:
wheel > 0.38.1
urllib3 > 1.26.19
Pygments > 2.15.0
zipp > 3.19.1

Dear @wgemm,
it would be great if you generate the *.txt files with  pip-compile --output-file=requirements.txt pyproject.toml

@wgemm Thank you for your work! I've updated the changelog. Now it's ready to merge.
Next time it would be great if you make a fork and also update the changelog as mention in contribution guide. See: https://github.com/IngenicsDigital/easy-release-automation/blob/main/CONTRIBUTING.md


Few Fixes

Gemfile is updated to ensure successful installation of phantomJS for running the tests.
While running the test cases, command_injection_spec is going to pending, I have fixed it which ensures successful run all the time.
A justification for this PR

To run tests, phantomJS is needed and there is a need for a gem installation. Hence I have added and sent a PR of the updated Gemfile to make developers life easier.

Coming to the change in command_injection_spec, this particular test is not succeeding all the time. This particular issue is fixed with the proposed changes

Hey @SriHarshaGajavalli 👋 - Thanks for submitting a PR. Can you provide some context as to the changes, their purpose, etc.?

Hello @cktricky - I have updated the description

@SriHarshaGajavalli - How does this PR impact the phantomJS installation instructions in the README file?

Why do you want the poltergeist gem twice in the Gemfile file?

@jasnow poltergeist gem is already existing but it is failing to get installing sometime. Hence I have added. But it is because of some issues on the local machine. However, I have removed it now.
Coming to the change command_injection_spec, everytime a test is running, it is creating a file and the test is going to pending status instead of failing (which is a successful run)

It isn't failing. Instead, it's successful execution leads to the creation of a file which in turn results in the test case status pending

It isn't failing. Instead, it's successful execution leads to the
creation of a file which in turn results in the test case status pending

With or without commenting out line 17?

With commenting, test case executes successfully (which means actually failed). If we uncomment the line, the test case is going to pending status.
I guess there could be bit of confusion with the concept itself

@SriHarshaGajavalli -  As you mention above:
Coming to the change in command_injection_spec, this particular test is not succeeding all the time. This particular issue is fixed with the proposed changes

Do you know why the command_injection_spec  is succeeding sometimes and not others?

@jasnow : Because of the above mentioned reason, it never succeeded for me

Could not reproduce the problem so rejecting the fix.

@SriHarshaGajavalli  - You are welcome to reopen this issue when you have more data to share.


WIP 🌱 Add scalability test

Add test that apply scaled number of bmh on dev-env using fakeIPA.
by default the test apply 100 BMH by batches each 20 BMHs applied in the same time
TODO:

Merge prow trigger: https://gerrit.nordix.org/c/infra/cicd/+/22355
Merge DevEnv running fakeIPA PR: https://github.com/metal3-io/metal3-dev-env/pull/1450

/cc @lentzi90

/hold cancel

This looks ok to me. Is the plan to do more in this test eventually, or why is it in CAPM3? As it is now it would make more sense in BMO (or dev-env) since it is just dealing with BMHs. /lgtm

This is the first step of applying BMH but will add on top of this cluster scalability i.e applying capm3 cluster templates on fake-env with certain available bmh
We might think of adding a scalability test that just test BMO or even just ironic somewhere

/test metal3-centos-e2e-integration-test-main metal3-ubuntu-e2e-integration-test-main

/hold
waiting for pipeline https://gerrit.nordix.org/c/infra/cicd/+/22355


Downloading Deviantart Watched Deviations

Ripme version: <IP_ADDRESS>
Java version: java version "1.8.0_201"
Java(TM) SE Runtime Environment (build 1.8.0_201-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)
Operating system: Microsoft Windows [Version 10.0.17763.253]
Exact URL you were trying to rip when the problem occurred: https://www.deviantart.com/notifications/#view=deviations
Please include any additional information about how to reproduce the problem:

Expected Behavior
Treat the watched list as a common gallery and downloading it.
Actual Behavior
Url not recognized.
This might be because even though i gave it my login credentials it does not login properly.
In fact, downloading from any gallery gets me banned from a few days every time i try and use RipMe
I've seen that you have implemented the OAuth 2.0 authorization system for Tumblr. Deviantart uses a very similar system detailed here: https://www.deviantart.com/developers/authentication
Thank you for all your work. I "dared" getting hospitalized for a couple of months and now deviantart expects me to download 7'000 deviations one by one. I would like to clear the watch list without losing the deviation and deleting those i'm not interested into by hand while scrolling a local gallery seems quicker than going through the process of opening up web pages.

Hmm, seeing the OAuth doesn't seem to be related to DA's own worthless API, this might be usable.
I've never gotten banned by DA for using RipMe, so what settings are you using ? Did you provide your own logins or use RipMe's ?

i provided my own logins it probably was because of that.


"go run" whole project directory

Is something like go run *.go possible instead of single file?

Does go run support that?
As for atom script, the defaults are tied to one file. You can open the 'script options' pane using the omnibar and tell it to run that as your command.

how can you do that?

Looking back at this, our default could be go run *.go in the current directory, since it has to run whichever file has main.

Agree with that


YubiHsm using PKCS11

Added YubiHSM integration using PKCS11 module
Removed YubiHsm-Shell integration

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.1 out of 2 committers have signed the CLA.:white_check_mark: usmansaleem:x: rain-onYou have signed the CLA already but the status is still pending? Let us recheck it.


Add the possibility to add import and respect the import sorting.

In #816, we need to add the sys module if missing and keep the imports sorted.

This would be good to tackle. It has now come up a few times now.
I think the idea would be to track the imports that need to be present for a given code modification, and then do a pass at the end to "add" those imports. It would somehow have to be connected to the autofix API too, to ensure that we only add the import if we end up applying the fix.
You can see how LibCST handles the problem here.

There are a few sub-problems:

How do we determine where the imports should go? It's probably not necessary to put the imports in the "right" spot, since we can rely on the next autofix iteration fixing up the order. But they should go in the first "block".
How do we only add imports we don't already have?
...

I am returning to this.

Getting close...
diff --git a/foo.py b/foo.py
index 0647ee33b..696a9e873 100644
--- a/foo.py
+++ b/foo.py
@@ -1,3 +1,4 @@
+import sys
 exit = 1

-quit(1)
+sys.exit(1)


Unaligned 64-bit access causes crash on Hexagon platforms

We've discovered the cause of a TF Lite Micro crash on some Qualcomm Hexagon platforms, and need to figure out a good solution. Here are the details:
Quote from tensorflow/lite/micro/micro_allocator.cc:327 :

      // Note that the zero_point field in the FlatBuffers schema is a 64-bit

      // integer, but the zero_point field in the TfLiteQuantizationParams struct

      // is a 32-bit integer.

      result->params.zero_point =

          static_cast<int32_t>(src_quantization->zero_point()->Get(0));

As stated in the comment, the zero_point is a 64bit value inside flatbuffer’s schema.
But when hexagon DSP tries to read a 64bit value, the address needs to be 64bit aligned. This is a hardware instruction level limitation of our DSP.
So, when the execution fell into IndirectHelper::Read(), hexagon-sim crashed.
Quote from tensorflow/lite/micro/tools/make/downloads/flatbuffers/include/flatbuffers/flatbuffers.h:104 :

>>  static return_type Read(const uint8_t *p, uoffset_t i) {

      return EndianScalar((reinterpret_cast<const T *>(p))[i]);

    }

This pointer “p” is not guaranteed to be 8 byte aligned. So when I am not lucky, the linker will give it a bad location. Hence, random crash.

After evaluating options, I've opted to add alignas(8) to our serialized flatbuffers.  The entire flatbuffer is internally aligned properly (for example, 64-bit datatypes are aligned to 8 byte boundaries).
I have submitted a change to update our in-tree example and benchmark models to align the flatbuffer array to 8 bytes.  I am working to update alignment in models which are downloaded at compile time.

@petewarden
Please update as per above comment


Counter forward stops at 60 seconds

In the demo if you start the counter so it goes forward (Count up checkbox) it stops at 60 seconds.

The "problem" is here https://github.com/FlowingCode/SimpleTimerAddon/blob/9a71b928de47aa6d6127fa04a5f47872698a6920/src/main/resources/META-INF/frontend/simple-timer/simple-timer.js#L156. The calculations to know when to stop the counter are using the startTime. This property has a default value of 60. So, if you don't define a startTime (in this case it will be considered as a end-time) then it will stop at 60 seconds.
Should the default value for the startTime property be set to some other default when countUp is set?

The default value and the decrease logic are inherit from the original component that the add-on is based on. I would also expect that the count up never ends if no end time is defined. So how can we make the API less poor? Should we remove the default value in case on countUp or just replace the decrease logic or remove the default value completely from the component?

At least the demo should make sense, when enabling the count up checkbox the textfield changes to "End time", we could populate it with 60.

The default value and the decrease logic are inherit from the original component that the add-on is based on. I would also expect that the count up never ends if no end time is defined. So how can we make the API less poor? Should we remove the default value in case on countUp or just replace the decrease logic or remove the default value completely from the component?

Any fix would be a breaking change, so let's figure out what makes most sense in this case.
IMHO either there should be no default value (some initial value is needed for count down, but there's nothing special in "60" and users will likely need to set an initial value that is different from 60), and/or there should be an endTime property that applies to count up, so that count up does not end at the so-called "startTime"


bold text looks weird

Strong/bold text has currently font-weight: bolder which probably should just get redefined to medium.
Two options:

define every strong/bold to font-medium
define only definition list titles to be medium, and add the semibold font back into the mix

After investigation, it was all strong text that was weird. The definition list titles are ok. I want to tweak the spacing between the definition list title and text a bit.


can i use it in IE8, chrome,and firefox ?

The point of this library (versus other ones mentioned in the readme) is to be lightweight. We only support modern browsers. I was okay with adding IE support in #9 since it didn't add new lines of code or hacks.
I'll test in IE11 just to confirm it works, add that to the readme, and then close this issue.

Great, confirmed it works in IE11. ✨
Let's not support anything lower, the other libraries already work for that!


add pyang

Hello @fredhsu @sulrich @noredistribution @burnyd
I am  about to add some pyang examples. Just examples, no code.
do we want them here? https://github.com/aristanetworks/openmgmt/tree/main/docs/examples
reason I am asking is because of this https://github.com/aristanetworks/openmgmt/blob/main/docs/examples/index.md
Many thanks!

will have the same question about pyangbing soon


fix: bridge fixes

What ❔

some l2 bridge contract fixes
reworking l2 foundry tests:

previous WETH and ERC20 tests are split, with the shared deployment utils put in _SharedL2ContractL2DeployerUtils and SharedL2ContractDeployer.sol
tests are moved to abstract contracts, so they can also be run in l1 tests, for faster foundry.
the DeployL1 is split so we can compile for L2, and other small changes, e.g. deployViaCreate2 is modified.
adding l2 integration tests for GW

Why ❔

Checklist

[ ] PR title corresponds to the body of PR (we generate changelog entries from PRs).
[ ] Tests for the changes have been added / updated.
[ ] Documentation comments have been added / updated.


Build v2 links.json for all schemas

If a schema doesn’t have a customised links.json, it should still use
the base, so we should generate it anyway.

Seems right to me, but I suggested it. Anyone else have an opinion?

Do we know when this bug was introduced? Have we always had it?

The bug was likely introduced when the source json files were carved up to accommodate v2 publisher examples. I refactored the links into separate files as v2 publishing api endpoints are split between content and links. I evidently didn't catch this case so thank you for catching this and fixing :+1:


[ruyi-reimu] Board image ubuntu-server-riscv64-sifive-unmatched need update

Description

In upstream repo https://cdimage.ubuntu.com/releases/, the latest version is 24.04

Need update https://cdimage.ubuntu.com/releases/24.04/release/ubuntu-24.04-preinstalled-server-riscv64+unmatched.img.xz


Infinite Loop in ElementalArea getOwnerPage

When I save a page I get an infinite loop when owner ClassName is Null. Removed write from getOwnerPager to fix.

https://github.com/dnadesign/silverstripe-elemental/pull/319

Thanks for the contribution =)

Hi @Lolly101, we are having trouble with this change in the master branch CI builds so I've reverted the change and made a new one at #348 - can you let me know if this still fixes your problem? Unfortunately I'm unable to replicate your issue to try it out myself


DOF Sync broken?

According to a user, the config is not updated.
Re-check the error handling there.
Setup up 32-bit too and retest.
https://github.com/vpinball/b2s-backglass/wiki/64bit-VPX-Pincab-Setup-Guide

Fixed with 2.14.0


Make prepare script work on current Node LTS

Also added unlinkSync to make the prepare.js file delete itself after run.
Closes morewings/cra-template-npm-library#147

@sky0matic thank you for help!


Add contentEncoding and contentMediaType for Kubeconfig

This is related to https://github.com/linode/linode-cli/issues/247
The kubeconfig returned from GET /lke/clusters/{clusterId}/kubeconfig
is a base64-encoded YAML config.  The OpenAPI standard now supports this
with the standard JSON Schema fields.
This change notes the encoding of the kubeconfig field, which the CLI
may use to decode the base64 data and display the actual YAML for
convenience (see linked CLI PR below).

This will not pass validation without https://github.com/Dorthu/openapi3/pull/48, as these fields were previously not included in the parser/validator.


ASTField query, not its subqueries

Hi, thanks for this great repo.
I suggest we get the $ast itself up to the ResolveInfo, not its subfields ($astFields), since we could want to retrieve its location. Example : in a Resolve callback, of a specific field, we could check if user has sufficient rights to read the field, and if not nullify the value and add an error to the ExecutionContext with Location information inside the message (line, column).
I did a fork to get this working, but I'd love to hear about you on this.
Thanks

Feel free to make a PR if you already have it coded up.


Find and Fix Broken Links

I found two broken links showing error, Page Not Found. And update README.md and change broken links with working links.
For issue #3 .

I did it.

@Thusal06 Thanks for the help 👍


Benchmark for finding resources

What does the pull request do?

Adds a benchmark case for finding resources. Right now it contains only simple cases but it can be expanded later.
What is the current behavior?

dotTrace showing 50% time spent casting

Benchmark results on my machine:

Method
Mean
Error
StdDev
Allocated

FindPreResource
1,716.78 us
86.903 us
256.235 us
1 B

FindPostResource
42.62 us
1.943 us
5.668 us
-

FindNotExistingResource
1,580.46 us
57.811 us
170.456 us
1 B

You can test this PR using the following package version. 0.10.999-cibuild0019854-beta. (feed url: https://nuget.avaloniaui.net/repository/avalonia-all/index.json) [PRBUILDID]

You can test this PR using the following package version. 0.10.999-cibuild0019917-beta. (feed url: https://nuget.avaloniaui.net/repository/avalonia-all/index.json) [PRBUILDID]

You can test this PR using the following package version. 0.10.999-cibuild0019942-beta. (feed url: https://nuget.avaloniaui.net/repository/avalonia-all/index.json) [PRBUILDID]


Error code -32002 when not logged into Metamask

When logged out of metamask, and upon first load of page this error appear.s

This happens when autoConnect: true and readOnlyUrls is set (required for autoconnect).
Is there a way to catch this error? Thanks.

Does the issue still persist in the latest version of useDapp?

Does not seem to be an error on recent version of useDapp.
Feel free to reopen if there are still problems with a recent version.


"Markdownify" YAML Metadata Blocks When Building HTML

First off, I love Quartz, this is a great project! I have only one small bottleneck.
Is your feature request related to a problem? Please describe.
When I build my site, I often reference titles of works in the title of my articles, because I am a neurotic librarian, I must have them italicized in Chicago style. Currently, when I build the site with default settings
For example:
---
title: My *italicized* title
---

Does not render as My italicized title but rather as My italicized title
Describe the solution you'd like
When I use
---
title: My *italicized* title
---

It renders on the site as My italicized title
Describe alternatives you've considered
On my fork of quartz, I used the DangerouslySetInnerHTML to make a hacky fix, but I don't like being forced to use Dangerous settings even when there is no user-generated content on the site. I simply do not have the knowledge of typescript to meaningfully contribute any other insight.
Additional context
N/A

The option DangerouslySetInnerHTML is called that because it executes any arbitrary HTML. That include <script> tags inside notes.
If you curated the content of your notes, there should be no issue.


Add faster versions of equals, is_eql and is_equal

Hi there,
I'm just getting my feet wet with Rust/Rutie, so its entirely possible that this patch is nonsensical.  Also, I'm not sure if I have put things in the right place, etc. Happy to make any changes you consider necessary!
First, because Ruby < 2.5 has a different implementation of rb_eql, I've added a bunch of cfg directives to build.rs, so that functions can be defined conditional on the Ruby version.  This means that the faster is_eql function is only used on Ruby 2.5+. Presumably I should document these somewhere, but I haven't done this.  The remaining changes are:

Object::equals is now a wrapper around the rb_equal C function;
Object::is_eql is now a wrapper around the rb_eql C function on Ruby 2.5+;
Object::is_equal simply performs an == comparison on the underlying values, which is the same behaviour as the rb_obj_equal C function.

OK, so in Ruby 2.4, rb_eql returns an int instead of a VALUE ...

This is a very good point. Hopefully it will get into master soon :pray:

I'm hesitant to allow Ruby versioning for different code paths.  Once you travel down this road it grows into an unmaintainable beast.  I have some code commented out in Rutie waiting for "End of Life" of older Ruby versions before I can use Ruby's frozen feature in certain places.
Once Ruby 2.7 comes out we can stop officially supporting 2.4 as we only support the 3 most recent versions of Ruby.  Then you won't need this Ruby version check.  So you just need to wait until Christmas for the new implementation.

Sounds reasonable!  I'll rework this PR some time in the lead-up to xmas

I've dropped the is_eql implementation from this PR, so it now works without the versioning.  I'll try to remember to come back & open a follow-up PR once 2.7 gets released.

Feel free to re-add is_eql.  I'll take the time to review once that's back in.

K, I've reverted back to the original commit from August.


NLP: JAX Machine Translation

Keras/TF -> Jax/Flax.nnx convert and expand of
https://keras.io/examples/nlp/neural_machine_translation_with_transformer/
English-to-spanish encoder/decoder transformer training and inference.

Alright - adapted your comments @vfdev-5 - will get the dataloader changes in as a secondary commit


[RFC] Concurrency, syntax and GC

This is a kind of open ended issue, touching on the subject of concurrency implementation and GC's, just felt it could be good food for thought to not get locked in a path prematurely.
At writing moment, there is just single thread concurrency in Crystal. Of course we all want multi thread. And of course we want things to be neat to use (also now).
The inspiration at the moment seems to primarily be Go for these areas.
As the prospect of a future Crystal specific precise GC (eventually we'll all want that - right?) might affect how concurrency is handled (memory passing wise), I'll just drop some thoughts.

There is some criticism on Go's way of using channels - but they might mainly be because of lack of generics. In any event - before locking things down in syntax perhaps some other angles of attack should (also) be considered.
Nim has a GC for each thread - a different model where no "stop the world" ever occurs, this requires "nothing shared" policy.
Go's latest GC effort uses multiple mark/scan threads to handle one GC - there is "stop the world" - but they're reducing the latency at expense of larger memory consumption ("the GC knob" setting).

That's all folks.
Issues somewhat related: #1454, #688.

Felix's GC is also rather interesting, since it interoperates surprisingly well with other languages. I believe it is multithreaded and uses a stop-the-world mechanism.

@kirbyfan64 - any link? I'm curious about these things :)

@ozra http://felix-lang.org/share/src/packages/gc.fdoc

Felix uses a garbage collector to provide secure automatic memory management.

For reasons of C and C++ compatibility, the allocator used is just malloc. Similarly, to ensure C and C++ interoperability, Felix GC uses a naive mark and sweep algorithm instead of a faster and more modern copying collector. C/C++ objects can't be moved easily and read or write barriers cannot be easily implemented.

The collector depends heavily on the Judy1 and JudyL data structures. These are ultra-high performance stores which provide fast, scalable, O(1) linear scanning both up and down, as well as random access, deletion, and insertion. Judy's key disadvantage is that it only works with machine word size keys and (for JudyL) machine word size data.

The Felix collector is a hybrid. Heap allocated objects are associated with a shape object which provides a precise map of the location of all pointers in the object, accelerating the scan for reachable objects and reducing the number of unreachable objects that are not collected. On the other hand the machine stack of each thread is scanned conservatively, since Felix has no precise information on stack allocated data.

The GC can handle interior pointers. These are pointers to an address past the starting point of an object. However Felix does not consider a pointer "one past the end" to be interior. Care must be taken not to increment pointers into arrays past the last element. This is a deliberate design choice: a past the end pointer might be the head of a consecutively allocated object. Without this constraint an allocator might be forced to introduce padding.

The GC can also handle C pointers mixed in with Felix pointers, in other words, where an object has a designated slot for a pointer, either a Felix or C pointer may be used. It can do this because it keeps track of all objects it allocated.

The GC also assumes all objects are arrays of fixed maximum extent. It tracks the maximum number of elements the array might contain as well as the actual number used. JudyL arrays map addresses to the base shape, the array bound, and the actual usage, with missing keys designating 1 element in the last two cases.

The GC also supports finalisation. By default the finaliser of a C++ object is a function calling the object's type's destructor. This means a complex data structure such as an STL map can be used in Felix, provided the value type is does not contain GC managed pointers. The finaliser calls the destructor to release storage, relieving the GC of the task of tracking the individual objects comprising the data structure.

Thus, the Felix GC is more efficient that one might expect because it does not need to track every allocation, nor scan every object. In addition, the programmer is free to use C style manual allocation and deallocation for many data types. Never-the-less the GC is required for some union types and some function and procedure closures. It is also useful for many function data structures including persistent purely functional lists.

We should note that the Felix GC does have one significant drawback: although it is thread safe, and performance suffers a bit from serialisation of allocations, collections require a voluntary world stop by all threads. There is no portable way to stop threads at arbitrary times, so if one thread requests a collection, all the threads must wait until the last one yields to the stop request, and then the collection is performed.

This thread is gonna be interesting :+1:

Well Felix GC didn't inspire too much ;-)
What can be leveraged in Crystal is of course:

Types are known, so type maps are "already available" (specific scanning of just pointers)
All "hidden" pointers ("references" that is) are known to point to beginnings of objects, so there too cycles can be shaved, not having to worry about  if it's "internal pointing".
When using pointers for hard core performance work, a specific BoundPointer type could be used (so GC can skip checking "internal pointing" and just look at the bound-obj-start-ptr in it (when working with slices and iterating in a method or so). These could also be bounds asserted in dev mode as a bonus. (this requires a pragma to tell crystal that GC should ignore a specific pointer member)
A specific "raw allocator" ("malloc") could benefit a specific GC, rather than relying on clib malloc.
And then the real reason for the issue: depending on if thread-specific GC-heaps or global is used, and how thread synchronization is done when time to stop the world (if the global approach is chosen) - well - this can possibly be tied together tighter too when designing it holistically.

If picking an outsider GC the advantages are harder to gain (generating intermediate type maps, or just brute force scanning, like I presume boehm does now)
But well, above scanning nitpicking (assuming a non reference counting GC) aside, the important thing really comes down to the concurrency issues.

I have been toying around with cross platform concurrency on Ruby for awhile now with https://github.com/cotag/libuv which primarily wraps all IO in promises and supports multiple event loops.
I think there are some really powerful things we could do with IO / threading on crystal.

Dedicated IO threads

Spin up a maximum of one per-core
Lazy load them with basic thresholds
Shared nothing

Have seperate threads for processing

Internally managed thread pool (can grow and shrink as required)
The initial / main thread is the first of these
Optionally allow users to execute code on their own dedicated threads

Promises and Futures for coordination

So threads are transparent to the users
Code is automatically run in parallel
Heavy use of fibers

Effective this would allow you to write code like this:

stream = IO.new
stream.write 'hello'   # => Promise
# Writes are performed on an IO thread. The write itself is scheduled on that thread
# We can use Fibers to prevent further execution until the write has been buffered (not completed) so there is no concurrent access of the variable.
# Reference counting may provide a way to avoid a fiber yield here if this is the last reference to the variable or if the data is an object literal or constant etc
stream.write 'world'  # => Promise
resp = stream.recv  # => Promise
output = resp.value
# ---- Call to value is a future. Fiber yields here until response is received
# Once response is received we continue processing on the same thread
puts output

Further to this, if you would like to use multiple threads for processing

stream = IO.new #=> Promise
# stream itself is a promise, so easy to detect stream closure.
# write will yield the fiber until the stream is ready. We can provide on_ready callbacks too
stream.write('hello').then(:concurrently) do
    # Anything in a callback block could optionally be executed on a different thread
    # (Idle threads watching a work queue)
    # This call to write would only occur after the first write had completed
    stream.write 'world'  # => Promise
end
resp = stream.recv  # => Promise
output = resp.value
# The existing thread
puts output

Much like Go Lang or .NET async, it would be easy to provide concurrent execution options:

def some_func(arg1)
    'return value'
end

# co keyword, for concurrently, to execute function on another thread
result = co some_func(arg1) # => Promise
puts 'no stopping me'
puts result.value

This approach takes the best of Node.JS and GO Lang and combines them with IO that never blocks.
I'd be happy to start implementing this, I assume there is nothing preventing multiple threads executing crystal code? (I can disable the GC for now if it is an issue)
What are peoples thoughts on the syntax?

+1 for libuv, but I'm not sure I am a fan of extra syntax. Then again co could maybe be a macro, similar to the current concurrency macros.

Are you guys completely aware of the current concurrency and IO model? Because Crystal already works with non blocking IO and schedules fibers. Working with callbacks and promises for IO is not the approach I like for this language.
What's so good with libuv? In fact I did the first implementation with it but then I moved to libevent because it fits better with the IO model based on coroutines instead of callbacks.

Note: you do well in skipping the tl;dr "my limited experience" section
My limited experience for perspective
If someone feels like enlightening me
The experience I have with concurrency is node.js since 2010, and C++ over processes.
In 2010 node.js there weren't many libs, so I just worked with what came up in my head. A pattern that I came to use almost everywhere was what I called the Reaper. You simple wrap the callbacks for tasks you want to "collect" with a call, and it kept reference count on them and fired "ready" when all where done.
Then there were lots of crashes at a point. Turned out that teams worked at the same time in sort of "team work sessions" with specific things like uploading media (photos, mp3's and movies). I used a custom built mplayer and other tools to convert these to standardized bitrates and sizes. When the teams did this at the same time, the VPS ran out of memory, duh. So I thought of a funnel you pour lemonade into bottles with. I made a "Funnel" (which I've learned later exists as "Throttle" in established pattern terminology). Those two patterns did basically all the async-work in the entire node-app. Sequential async was just callback-stacks (with LiveScript back-calls it looks very synchronous). So that's my back-turned-on-the-world experience there.
The other side was a heavy computation system in C++. There's one data-samples collector, it writes the data packed via fix-pointing, then delta valuing, and finally variable length encoding and writing into fixed-element-size "chapters", to provide random access entry points at reasonable intervals. That is one producer. Then all the analytics artillery mmaped this data (64-bit arch was required for the system), read and decoded in to circular buffes on a "circular heap", synchronized for all data sources within each process. So there was one process for each avail core on the machine running at the same time. And of course also more machines, the data was replicated from the encoded buf. Synchronization per machine, was done simply through a "available-elements-counter" which was updated per "chapter" (instruction)atomically and fenced as new data had been written. (There where also read-ahead threads that simply stayed ahead of computations needs to make the kernel page in data earlier to avoid waiting for disk-snailing - hard to tell if there really was a benefit to this. Hard to benchmark it all.)
So all I know of concurrency is unfortunately my own constructs, which very likely overlaps with existing ones. (Reinventing the wheel seems to be my speciality).
Now
Most concurrency in most apps is just IO. And naturally taking one item of the grocery list, hopping on the bus, asking the guy at the counter for it, he getting it, returning home, then taking the next item, is a ridiculous waste of time. So of course Fibers are a super solution to be able to give the OS as many of hundreds of file-requests as possible so that it can order them well to the guy at the counter (storage) and that dude can swing his arms around the shelves for all items in the optimal order (hard-drive arm). Yes, I know I sound rudimentary stupid here, but, that's the point: I am :)
So, all I'm aiming at here is: those of us with rudimentary concurrency / parallelization skills would of course want a clean, uniform, well agreeing model that forms the foundation of all stdlib concurrency in Crystal (duh, again).
The Fibers, Threads, Scheduling and GC team
It seems to me libuv performance wise is superior to libevent2. The downsides are that not all of pthread API is available, the upside is that the reason for this is that it works optimally under Linux, Windows, Mac, etc. - a range of systems. It's actively maintained and developed, and it's in C.
What I'd like to see in Crystal is that all parts are orchestrated together so well that - I think optimally - a GC for each thread (the never-stop-the-world Nim model) could be integrated. It seems to me share-nothing would facilitate this well.
For solutions requiring parallel sharing (huge data) - highly specialized synchronization for the domain / algo could be used, for instance with above mentioned mmap'ing.
You guys (and gals of course) who know your stuff about this...
I'd love to learn a bit more.  Of the top off my head:

How transparent can it be made? Will the multiple event loops be handled without any active consideration without clashes and locks?
Can one use blocking IO, and it just automatically schedules the next Fiber without special notation, continuing when the data is there (remember, I've worked with callbacks only, in node.js)
Can the condition variable aspects of libuv be used to facilitate cross-thread-calling in a transparent manner?
Will GC pressure be an issue for promises? (no matter what GC model is chosen in the future). And won't promises require "wordier code" that could be avoided for a more sync-style looking code?
 B-)

What I'm proposing is to expose promises as an option. They are very powerful coordination constructs.
They wouldn't have to be used and for the most part serial looking code with fibers can be used - this code is very easy to facilitate using promises.
Libuv make sense as it covers so many more OS primitives in a cross platform way. It has a lot of community involvement and is very consistent. TCP, UDP, Filesystem, Timers, Pipes, Locking, etc
Pushing all IO to dedicated threads prevents worker code from starving the event loop. Even if your code is non-blocking it will still take time to execute and this becomes a point of contention.
There is no feasible way to make cross thread calling transparent. Promises would make this easier. Promise pipelining effectively allows map-reduce style processing very simple and often without locks.
Locks will be needed unless very deliberately avoided by the developer.

I see now there were comments while I was writing my novel like monster of a comment.
@waj - since libuv and Boost::ASIO is all I know of these (via node and making C++ extensions for it), and the pure C++ apps. What made it difficult to get it to work with Crystal? Seeing that libevent2 seems to be "not that maintained" and the fewer platforms it supports mainly.

Also, is there a particular reason for the std concurrency constructs not supporting multi threading yet? So I get the grits...

@stakach - what would be the differences to the currently Crystal idiomatic concurrency constructs? Performance wise, and "boilerplate-wise"?

What I'm proposing is we rip out all the legacy socket code and replace it with equivalent high performance code. Effectively all IO (File, Network, Pipes etc) go through Libuv running on dedicated IO threads.
Legacy classes can be wrapped to support non-blocking IO with Fibers. https://github.com/celluloid/celluloid does this to make regular ruby code non-blocking. I don't particularly like the actor model however that aspect of Celluloid is awesome and would make sense in a new language.
This would mean libraries don't need to deal with futures, promises or callbacks and it won't slow down highly parallel apps.
Anyone who wants raw access to the async callbacks via promises or wants to deal with multiple threads and locking etc can without worrying about how the libraries are performing IO.
the exception here would be C libraries however implementors of crystal libraries could use threads and futures / promises to wrap these
There would be a minor IO performance trade off for smaller applications, where it is unlikely to matter. However for high IO applications it would make a world of difference, for both code confidence and performance.

I don't think the current IO implementation could be considered "legacy". Current IO already takes care of concurrent operations in a very transparent and comfortable way to me. Thousands of concurrent socket sessions for example, can be currently handled with high performance on a single thread. No paralellism is occurring, granted, and that's why we want to add support for multithread scheduler in the near future.
Also, I don't think you can easily beat the current performance for concurrent IO sessions if you try to schedule the IO on separate threads. I'm worried that doing context switching on each IO operation is unnecessary and poorly comparable with the current implementation.
All that being said, I could probably be wrong, and you could demonstrate that to me if you wish to play with libuv and show some real numbers.

Crystal already has concurrency. All IO is already running on an event loop. We're not using libuv for different reasons. See @waj comment for instance, and IMHO it lacks multithread support and wraps far too much things to be any flexible (ie. it's tied to node.js as a single threaded event loop, and... nobody wants that for Crystal).
Libevent is also actively maintained and supports all operating systems, but it doesn't wrap everything, and has multithread support. IMHO it's a better fit for Crystal evolution to have something smaller and flexible.
What Crystal lacks, for the time being, is parallelism (an event loop that runs and resumes routinea across a pool of threads), and it's a complex thing to put together, as outlined here.
But again, Crystal already has concurrency (spawn) and synchronisation/communication (Channel) of concurrent code. One can already create an actor or promises library on top of it, thought we would prefer to have actors (and I guess supervisors) than promises into the stdlib.

What's so good with libuv?
—@waj

It's a nice library with lots of general asynchronous task and AIO primitives. I wasn't aware that Crystal already uses libPCL and libevent2; I thought there was no concurrency or threading support at all, which is why I chipped in about adding support for libuv.
I think the real benefit of libuv would be if Crystal switched all IO and all to libuv. Using libuv just for multithreading would be adding unecessary bloat.
What's wrong with the current state of concurrency? I personally think using libPCL for Fibers is great, and as @ysbaddaden pointed out  there's the spawn macro/Channel to make it convenient for use.

Just to clarify: fibers are an internal implementation detail and spawn is the public way to create coroutines (ie. dont use fibers directly).

Most of the foundation to get transparent parallelism working have already been completed by @waj.   The IO system is (almost) ready for parallelism with minimal modification The scheduler needs a bit of work and there are thread safety issues scattered in a few places but there's not much left to do,
I have a crystal branch where multithreading (mostly) works.  All work has stopped as @waj and @asterite have asked me to stop work on anything thread/parallelism related.  I'm not sure why other than they want to do it themselves but also said they don't have a plan and are unwilling to work on a plan publicly, remotely, or in person.
A basic outline was sent to @asterite and @waj.  You can see it here: https://github.com/technorama/crystal/wiki/Concurrency-Parallelism
There was also a proposal for adopting the Actor model. https://github.com/technorama/crystal/wiki/Actors
Please excuse the rough nature of the outlines.  They assume you know the internals of the scheduler and IO system.  Minimal effort was put in as I was already asked to stop working on threading.
Both proposals (and others) have been ignored.
There could have been a working multithreaded crystal over a month ago if I was not asked to stop.  I am still unclear on the reasons.
At this point I have no idea how to get parallelism code in to crystal as all attempts to submit code, have plans reviewed or work towards a solution have mostly been ignored.

@ysbaddaden Is this going to change, though? I don't personally see a reason for using Fibers directly myself, since spawn seems sufficient in most cases, but it is odd.

@omninonsense no, just use spawn.

Nice to see the different perspectives. Concurrency is a wide subject, even more so with parallelism thrown in to the batter.
There are merits to all.
I've been to many Erlang/Haskell combo seminars hosted by Ulf Wiger, and I've used the "mailbox part" of that and supervisorish structuring, over processes. I like that. Within one app, the mailbox-method for "everything" would be quite heavy, I've been tinkering with ideas about it. I do believe the mailbox matched receiving is a better model than channels for many things, but channels are more low level and can, with more boiler, be used for more efficient flows. That's a pro for them.
So Actor model is very interesting, Fiber-based - not a thread per actor - imho. Sounds way to heavy.
As @waj points out, using more threads is not necessarily a win. A system is a complex eco-system of processes. So lets say your machine has got four cores. You've got four apps running on your machine most of the time. Now all of them decide it's more performant to use a thread for each core to utilize resources to the max. Now you've got four apps, running one thread on each core. Where's the gain? You've made a net-loss because of synchronization! Of course it's a contrived example.
@stakach - regarding starving the event loop, work always has to be done, and iff work has to be done before other work, there's no difference in throughput anyway, just an additional cost of synchronization. I'm not arguing against multi-thread usage - obviously. I'm just trying to highlight all the aspects of it as my understanding grows from all input here and else where.
For IO work, just getting as much orders to the kernel / net as possible is the biggest gain, the rest of formalia around loading / writing the data is miniscule. So I believe just Fibers works suitably well for that, since the "waiting work" is moved out to the kernel.
@waj - my main concern on libevent2 is - as I understand it - it's platform support is much smaller. Myself I've used Linux only since my Amiga got outdated, but I might very well have to code windows code for some client, and then I'd love to use Crystal. What's the view here?
@technorama @omninonsense @stakach - I assume Crystal team wants to get this "really" right also, and that is the reason why they've recommended the spawn/channels constructs only for now - to keep possibilities of change, and don't accept PR's at the drop of a hat. Once details are public, we're locked in (or in fiftyeleven different incompatible libs zone). And that's also the reason I opened this issue. Because it's encouraging to see some discussions and the reasoning behind it.
@technorama - thanks for the link to your proposals in the wiki!
@ysbaddaden - libuv is used as a single threaded loop in node.js, I assume it would murder V8 otherwise. That doesn't mean you can't run a loop per thread in crystal. I think I'm missing something here, because it seems very multi-thread-ready to me?
I like many of the ideas I hear here. So I hope something fruitful, simple and powerful, that all plays along comes out of it in the end.

Running 1 event loop per thread would tie a routine to a thread, with no means to resume it on another thread if a routine starts being less IO bound and more CPU bound.
Take Bcrypt for instance. It's CPU bound and it will block the event loop when computing a digest (it should have a cost of 1s or more to be secure), thus blocking all routines on the same thread whenever someone wants to login on your webapp. With a multithreaded event loop, they would be resumed on another thread —as long as they don't get all filled up with people login, that is :-)

@ysbaddaden that is a good example of why I think a dedicated IO thread would make sense. Removes the possibility of inadvertently blocking.
I am excited to see how threads are implemented @waj and also interested to see what impact offloaded IO has in terms of context switch overheads. I'll let you know once I have an example app

@ysbaddaden - thanks for clarifying!
@stakach - looking forward to benches :-)

@ozra The first part of my proposal doesn't change the public API.  Further phases add thread groups/pools and a simple API to create them or change the number of running threads (probably 3 new methods).  I don't see any reason to turn the first few phases of the proposal down.  It wouldn't harm future API changes as the API wouldn't change from it's current state.

@technorama - that is a good way forward in any event I'd think. The only costs are a few locks here and there I presume?

Just a question out of a short example, based on the simple example I mentioned: say I want to load 800 files (given here they will all fit in memory without problem), and giving the kernel and disk best opportunity to make that the fastest, I'd do something like this in node.js for simplicity (in LS - untested code):
fs = require "fs"
my-files = ["foo.file", "bar.file"] # 798 more ...
my-files-contents = new Array(800)
files-rr = Reaper()  # create a reaper instance

for let f, i in my-files  # loop the files, make sure i is a "copy" for the lambda-closures
    fs.open f, files-rr.sow (err, content) ->  # wrap the callback in a cb from files reaper
        my-files-contents[i] = if err then null else content

<- files-rr.reap  # "all-files-done.then"
# Now work with the files contents as wanted synchronously from here
say "File #{files.0} contains:\n#{my-files-contents.0}"

Obviously one would likely work on the files asynchronously too, if more than one thread was avail, but lets take this example. And obviously this node 0.4 based methodology works with callbacks with its drawbacks (stack inversion in traces etc.)
How would this be expressed in the different asynchronous models mentioned?

files = ["foo.file", "bar.file", ...]
promises = []

files.each do |file|
    File.open(file).then do |file|
        promises << file.read(file.length)
    end
end

Q.all(promises).then(proc { |file_data|
    puts "File #{files[0]} contains:\n#{file_data[0]}"
}).catch(proc { |error|
    puts "error occurred reading files"
})

of course you could optionally use a fiber on the last promise there to avoid a callback:

files = ["foo.file", "bar.file", ...]
promises = []

files.each do |file|
    File.open(file).then do |file|
        promises << file.read(file.length)
    end
end

begin
    # Similar implementation here: https://github.com/cotag/libuv/blob/master/lib/libuv/coroutines.rb
    file_data = wait Q.all(promises)
    puts "File #{files[0]} contains:\n#{file_data[0]}"
rescue => error
    puts "error occurred reading files"
end

@ozra Locks will work.  Lock free lists would be better (requires cmpxchg).  I asked about adding LLVM cmpxchg support to crystal but never received an answer.  If someone wants to attempt this it would help (if accepted).

With existing Crystal (verified):
record Message, filename, contents
channel = Channel::Buffered(Message).new

filenames.each do |filename|
  spawn do
    contents = File.read(filename)
    channel.send(Message.new(filename, contents))
  end
end

filenames.size.times do
  message = channel.receive
  puts "#{ message.filename } contains #{ message.contents }"
end

@ysbaddaden How do you coordinate when the processing is complete? Will the results be ordered?
i.e. If I want to do 3 things, how does one know if all 3 succeeded or failed and then process the results of those tasks?

pmap? if one failed you get exception (https://github.com/manastech/crystal/pull/1611)
filenames.pmap { |filename| File.read(filename) }

@stakach You could make Message's contents be a String | Exception (verified):
record Message, filename, contents
channel = Channel::Buffered(Message).new

filenames = %w(foo.cr goo.go goo.lala)

filenames.each do |filename|
  spawn do
    begin
      contents = File.read(filename)
      channel.send(Message.new(filename, contents))
    rescue ex
      channel.send(Message.new(filename, ex))
    end
  end
end

filenames.size.times do
  message = channel.receive
  filename = message.filename
  contents = message.contents
  case contents
  when String
    puts "#{filename} contains #{contents.inspect}"
  when Exception
    puts "#{filename} raised #{contents.inspect}"
  end
end

Or have the channel be of Message | Exception (but there you loose the filename), or maybe have another record Failure, filename, exception and have the channel be of Message | Failure.
The idea is that with spawn + channels you can do whatever you want. If you later need patterns on top of those primitives you can do them just fine because we can abstract over types, generics, etc, and we support union types.

we can abstract over types, generics, etc, and we support union types.
@asterite is it zero cost abstraction?

@sdogruyol Depends on how you do it and how smart the compiler/LLVM is. If you use a wrapper struct than it's zero overhead. If you hide something behind a method and LLVM can inline it, it's zero overhead. If you wrap something with a class there's heap allocation overhead, unless the compiler some day does escape analysis and determines it can allocate that on the stack. And so on...

Nice discussion here, I like it.
@ysbaddaden - regarding the moving of coroutines to other threads - I'm not at all convinced that's a good thing (TM). There's a complexity increase in reasoning regarding possible thread global references, and it could be argued that the program has a design error to begin with. I/O and longer "side work" would be better of put on their own threads to begin with then. I have no problem with application "main logic" following a synchronousish flow and block slightly then and when, the parallel "outbreaks" should work on other threads. Sticking to a mainly synchronous flow is much easier to reason about, and a full scale parallel to the bone monster should be rather rare, and then other patterns akin to the Erlang side of the universe might be better incorporated (these low level things aside). Thoughts?
@waj - I'm sorry if I'm repeating myself and starting to sound like a nagging child, but what are the thoughts on platform breadth and event-lib chosen? I still feel libuv has an upper hand concerning that?
@technorama - I agree atomic "instructions" in Crystal is definitely a must, and since LVVM has them down for C++11 stdlib compatibility, we should have them in Crystal - (there is a dependency on libatomic currently, I haven't looked where/how it's used - dep-dep?), giving tighter integration for LLVM optimizations.
The fact that a certain atomic operation might not always be possible to generate on a certain target architecture should simply be "be aware of" information imo - I would prefer it to not compile instead of transparently replacing the ops with expensive mutexes etc. to emulate it. After all, one would use the atomic operations for performance, so when not available on a target, one would likely choose another implementation for that specifically, or drop support for it (trying a 64 bit atomic op on a 32 bit arch, etc.).
The go-style channels seems to me a slightly primitive form of working - granted they enable more complex abstractions on top - but the "go way" smells a bit of "if the hammer is the only tool, every problem looks like a nail" (I read somewhere, and found the phrase fitting). Intuitively though, it feels like they're a more performant way (memory overhead vs synchronization), but that's by just vibing it - no bench done here ;-)
The LS example I gave would have worked with multi thread parallel work (if the engine had it), since each index is orderly set specifically - the unique location is given before async work. Of course It's a very specific and simple example though. Whether an error occurred could also be determined synchronously without syncing, in the data afterwards - (assume correct working -> fast, spend time if shit went wrong). Of course if errors are expected and/or are show stoppers it would be better to set a flag to avoid unnecessary work being performed, and that would be very quick with an atomically set/read cond. var.
As a parentheses the above Reaper can take index/key as part of the collecting and take care of the whole ordered result array business when wanted, I avoided that to make the example clearer.
It seems @kostya 's approach with pmap is very clean here.
I would really love to see several different common problems written in the different styles - it would be highly enlightening, also thoughts on performance and memory usage! (preferably benchmarks) In the end, of course straight forward code staving away from error prone flows is prio one.
If anyone has a good sample and others would chip in with a different approach it would really make my day!
Practical examples says more than a thousand flame wars :-)

Please note that I state my personal opinion here. Not waj or asterite or anybody from the core team.
@ozra How to solve my Bcrypt stops the world for 1 second problem? Granted, GC already stops the world, but for a tiny fraction of that time.
I can spawn a single coroutine that would compute Bcrypt digests and send it back through channels. But with a single event loop on a single thread? It would still stop the world, just preventing to have many coroutines that would do the same in a row.
Maybe Thread.new could create a new event loop. Sounds cool and easy. Even libuv can do it. I could spawn a thread instead of a coroutine and have it compute digests. No more stops the world! But how to send back the digest? Through channels? They'd have to work across different event loops... which sounds hard, because unnatural. Not even mentioning that we'd have 2 concurrency models, one is parallelism and the other one is concurrency only, but concurrent in their specific parallel world (I'm getting confused...). That's a lot to keep in mind when designing software. I guess I should just have a Bcrypt server and to communicate through an UNIX socket.
Now, maybe the event loop could run across many threads. That sounds complicated to put in place (and looks impossible with libuv), especially with efficient locking. I could then spawn my Bcrypt coroutine and communicate through channels, exactly like with the event loop running on a single thread: there is no difference for the developer, and a single concurrency and parallelism model to reason about. Except that computing a Bcrypt digest won't stop the world now (as long as there are at least 2 threads, of course), just slow it down.

@ysbaddaden - of course. Yes, for a computation taking one sec, I think I'd definitely would go with messaging, whether it's to a worker in another thread, or - I'd prefer - in it's own process. Using nanomsg, ZMQ or plain socket. My comment was on the idea that it runs in another thread, and is messaged. libuv multi-loops allows communicating through async message events "sent" into each other's loops, well: one way of doing it.
As soon as the cost of messaging is much lower than the work, I like the concept of external processes. Makes it possible to upgrade code of parts of the application, without stopping the main-logic, etc. But surely tastes vary. :) It might be because of the time I had with node that I liked delegating jobs out to daemons, seeing that node is just one thread.

Revisiting a bit.
Whether or not direct Thread control becomes an "acceptable use" or even "open part" of the languages/libs, a measure of control over fiber-thread-affinity (not thread-core-affinity, that's a different level entirely) is important.
One thing is making sure two pieces of algorithms / code are ensured to run in different threads, because they're very nature is making sure they work in parallel to maximize throughput. So some construct for spawning code and saying "in different-thread from some-fiber".
But then, for the second "level" of control, it might also be good to be able to set aside a core within a program for specific threads -> fibers, where one instead wants to maintain as high responsivity on action as possible.
These things should be possible to integrate in a fashionable manner without making it overly low-level smelling.

The GC and Runtime issue will be tackled by @waj and me (well, probably thought more by @waj and then we'll both implement it :-P), and we'll definitely see how other languages do this and see what fits best for Crystal. I can imagine implementing a GC of our own, because right now it's that or modify boehm, but in Crystal we can have a precise GC.
I'm closing this, a place like Google Groups is better for this kind of discussions.


Enabling and disabling - Components on Control Center

We need to allow System user to enable/disable components/features such as Contacts on Control Center.
Success story:
System user is on Control Center, Catalog view.
System user wants to enable a component on their Airy instance. e.g: Contacts, Inbox
System user clicks on "Enable".
The frontend should send such config. to CLI endpoint.
E.g.: "Contacts" is enabled on Airy core.
E.g.: Contacts appear on the left nav bar on control center to allow config.
E.g.: Contacts is Enabled on Inbox by "Contacts" view.
Control Center - DISABILING options:
Disable a "Service" from Status view.
Disable Connector from Connectors view.
Disable Component after installing and config from left nav. bar.
Control Center - ENABLING options:
1- Enable Components/connector from "Catalog" view,

The enabling/disabling of components in the backend, addresses scaling up/down the deployments of a component, when the configuration is already present in the ConfigMap for that component.
In that sense, I think that it will be a good approach to annotate or label the configMap with a label core.airy.co/enabled: true or false. So the endpoint would only add annotation or label to the configMap and then when we receive an event about a changed configMaps in the airy-controller - we can:

on configMap add - don't scale up if this is set to false
on configMap delete - no change to the current code
on configMap update - check if this value was changed and then scale up or down depending on if it is true of false.

I will do the front-end part in a separate PR


TLS client Authentication?

Hi,
I would like to secure my connection with TLS.
How I can setup client authentication?
Thank you.
Best regards,
Roman

You need to setup TLS as shown here: https://docs.dgraph.io/deploy#tls-configuration
On the client side, you need to add credentials to the DgraphClientStub constructor sa shown below:
const dgraph = require("dgraph-js");
const grpc = require("grpc");

const clientStub = new dgraph.DgraphClientStub(
  // addr: optional, default: "localhost:9080"
  "localhost:9080",
  // credentials: pass your root certificates here
  grpc.credentials.createSsl(root_certs),
);
const dgraphClient = new dgraph.DgraphClient(clientStub);

Hi @gpahal, thanks for hints ;).
There is some issue with TLS setup between dgraph and js but I found working solution.

@selmeci what is you working solution? I'm try many different thing without success... I tried common way and with nginx grpc ssl proxy. It look fro me like some kind of problem with "dgraph-js" library.

@mkucharz Unfortunately I did not solve it directly and I have to use nginx grpc ssl proxy for authentication.

@selmeci maybe this will help if you still need it: https://github.com/dgraph-io/dgraph-js/issues/50


compiling my plugin fails because of different versioned kotlin runtime wrapped in dokka embeddable

Hi,
I'm using version 0.9.9 of the plugin-tools in my plugin project. Compiling (with mvn clean install) fails because of a different versioned kotlin runtime wrapped in the dokka embeddable. This is the error message:
[INFO] --- maven-plugin-plugin:3.6.0:helpmojo (help-goal) @ gorky-maven-plugin ---
[INFO] Using 'UTF-8' encoding to read mojo source files.
[ERROR] warning: some JAR files in the classpath have the Kotlin Runtime library bundled into them. This may cause difficult to debug problems if there's a di
fferent version of the Kotlin Runtime library in the classpath. Consider removing these libraries from the classpath
[ERROR] C:\Users\...\.m2\repository\com\github\gantsign\maven\plugin-tools\dokka-embeddable\0.9.9\dokka-embeddable-0.9.9.jar: warning: library has Kotlin
 runtime bundled into it
[ERROR] warning: some JAR files in the classpath have the Kotlin Runtime library bundled into them. This may cause difficult to debug problems if there's a di
fferent version of the Kotlin Runtime library in the classpath. Consider removing these libraries from the classpath
[ERROR] C:\Users\...\.m2\repository\com\github\gantsign\maven\plugin-tools\dokka-embeddable\0.9.9\dokka-embeddable-0.9.9.jar: warning: library has Kotlin
 runtime bundled into it
[ERROR] warning: some JAR files in the classpath have the Kotlin Runtime library bundled into them. This may cause difficult to debug problems if there's a di
fferent version of the Kotlin Runtime library in the classpath. Consider removing these libraries from the classpath
[ERROR] C:\Users\...\.m2\repository\com\github\gantsign\maven\plugin-tools\dokka-embeddable\0.9.9\dokka-embeddable-0.9.9.jar: warning: library has Kotlin
 runtime bundled into it
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------

Any idea, how to solve this?

Is there some reason you're using version 0.9.9 (it's almost two years old - 0.9.24 is the latest version)?
Can you run mvn -X plugin:helpmojo and add a comment with the output.

the reason is that i get rather confused at times with version listings and pick the latest version as if it be sorted numerically.
Since the prev post above, I removed the plugin and cannot reproduce the issue above any more, sorry.

Hi,
I have a similar issue related to the bundled dokka dependencies.
I am using the version 0.9.26 (latest as of writing this comment) and my project uses the latest kotlin version (1.6.10). After some digging I found that the plugin is using dokka version 0.10.1 (released over a year ago). The latest version of dokka is 1.6.10, which supports kotlin 1.6.10 (which I think would solve my problem).
https://github.com/Kotlin/dokka/releases/tag/v1.6.10
Are there any plans on upgrading the dokka version sometime soon?


Small fixes

add missing color for H2 liquidation to config.default.yaml
adjust for myopic optimisation selection of assets to new default lifetime and build year
remove trailiing whitespace in index of electricity load of pypsa-eur network

Sorry, made a big mess squashing this time since it's only a few lines and doesn't contribute to stats.


Bazel CI Failing for Tensorflow with Bazel@HEAD

Tensorflow is impacted by an incompatible Bazel change: https://github.com/bazelbuild/continuous-integration/issues/1404
Tensorflow will need to migrate usages of @bazel_tools//platforms to @platforms, as described in the above bug.
Example breakage: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/2590#0182aec8-7365-4995-81b0-a0494cafc3ea

gentle ping. TensorFlow remains broken in the downstream pipeline, and if we can't resolve that soon, we'd need to disable TensorFlow on Bazel CI.

Thank you very much for your reminder.
I receive the following errors when I make the change @bazel_tools//platforms:linux" to @platforms//:linux":
ERROR: /[root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/external/sigbuild-r2.10_config_platform/BUILD:17](https://cs.corp.google.com/piper///depot/google3/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/external/sigbuild-r2.10_config_platform/BUILD?l=17&cl=469746102):9: no such target '@platforms//:linux': target 'linux' not declared in package '' defined by /root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/external/platforms/BUILD and referenced by '@sigbuild-r2.10_config_platform//:platform'
ERROR: /[tf/tensorflow/tensorflow/compiler/mlir/tools/kernel_gen/tests/tf_to_kernel/BUILD:5](https://cs.corp.google.com/piper///depot/google3/tf/tensorflow/tensorflow/compiler/mlir/tools/kernel_gen/tests/tf_to_kernel/BUILD?l=5&cl=469746102):15: While resolving toolchains for target //tensorflow/compiler/mlir/tools/kernel_gen/tests/tf_to_kernel:add_v2.mlir.test: Target @sigbuild-r2.10_config_platform//:platform was referenced as a platform, but does not provide PlatformInfo
ERROR: /[tf/tensorflow/tensorflow/compiler/tests/BUILD:1530](https://cs.corp.google.com/piper///depot/google3/tf/tensorflow/tensorflow/compiler/tests/BUILD?l=1530&cl=469746102):15: While resolving toolchains for target //tensorflow/compiler/tests:case_test_gpu: Target @sigbuild-r2.10_config_platform//:platform was referenced as a platform, but does not provide PlatformInfo
ERROR: /[tf/tensorflow/tensorflow/compiler/tests/BUILD:1729](https://cs.corp.google.com/piper///depot/google3/tf/tensorflow/tensorflow/compiler/tests/BUILD?l=1729&cl=469746102):13: While resolving toolchains for target //tensorflow/compiler/tests:async_comp_test_gpu: Target @sigbuild-r2.10_config_platform//:platform was referenced as a platform, but does not provide PlatformInfo
...

Any tips?

You need to use @platforms//os:linux (note the "os"). Similarly for architectures (for example @platforms//cpu:x86_64)

Thank you very much! Changes submitted!

https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/2606#0182f214-c75f-48b4-aefb-b1cdfc945293
TF build remains broken, likely due to an outdated rules_go.

@Wyverald , Thank you very much for letting us know! Could this line be causing the issue? If yes, do you have a tip for updating it?

My suspicion is that you need to upgrade to the latest version of rules_go, due to this line in the error logs:
ERROR: /var/lib/buildkite-agent/builds/bk-docker-9gkq/bazel-downstream-projects/tensorflow/tensorflow/tools/build_info/BUILD:9:10: While resolving toolchains for target //tensorflow/tools/build_info:gen_build_info: invalid registered toolchain '@go_sdk//:go_darwin_arm':

Thank you @Wyverald! I searched TF code for rules_go and io_bazel_rules_go but couldn't find anything. Nothing relevant in the Docker toolchain, too.
I submitted this change to update bazel_toolchains. Could you check if this solves the problem?

Hi @c-parsons ,
COuld you please confirm whether the commit mentioned in comment-1234951858 solves the issue. Please confirm and let us know if this still an issue. Thanks

CC @fweikert
I don't have cycles to look into this for the next ~3 weeks, sorry.

Hi @c-parsons ,
I hope this might have resolved already. Is this still an issue? Please feel free to close if resolved already.
Thanks!

@Wyverald @fweikert


[backport] Update CODE_OF_CONDUCT typo

Backport of #2609

[automatic post] Jenkins, test this please.

Jenkins CI test (for commit ac7ed219a11ebd29954db3091f80c781042d2887, target branch v6) succeeded!

Testing errors unrelated. Merging.

Jenkins CI test (for commit ac7ed219a11ebd29954db3091f80c781042d2887, target branch v6) succeeded!


Provide author data

For authors having a Wikidata ID the following data if available should be collected from Wikidata and provided by the DraCor API:

year and place of birth
year and place of death
image
full name

Has been resolved with above PR.