Build Report is not working

In your documentation when i tried to build the report, simply i need to type
report.buildReport
then you are output the Json in html file which is incorrect, however i couldn't get the structure from that built report, why you output the json two times one for build and the second one for json data, since the build strucutre takes json then it should be
reporter.writeReport(reporter.buildReport(JSON.parse(result)), reportOutputPath, (err, res) => {
  if (err) console.err('Error writting report: ', err)
  else console.log('Report written to: ', reportOutputPath)
})

not reporter.writeReport(res) // json again
or you can assign it to a constant variable and use this variable for outputting the html file

Would you like to send a PR to address this problem?

Yes


add:Jyothishree2k5

Description
I have included my json file in the public/data directory.
I have also added my json file in the src/profilelist directory
Changes Proposed
This is the code snippet that i have added.
{
"name": "Jyothishree Rajkumar",
"location": "TamilNadu",
"bio": "I am a pre-final year IT student with knowledege in MERN stack and i do dsa in python. I am currently trying to do open source as it interests me the most.",
"avatar": "https://github.com/Jyothishree2k5.png",
"portfolio": "https://github.com/Jyothishree2k5",
"skills": ["HTML", "CSS", "JAVASCRIPT", "MONGODB", "NODEJS", "EXPRESSJS", "SQL"],
"social": {
"GitHub": "https://github.com/Jyothishree2k5",
"Twitter": "https://twitter.com/Jyothishree2k58",
"LinkedIn": "https://www.linkedin.com/in/jyothishree-rajkumar-055022243"
}
}
Checklist

[ . ] I have read and followed the Contribution Guidelines.

[. ] All new and existing tests passed.

[. ] I have updated the documentation to reflect the changes I've made.

[. ] My code follows the code style of this project.

[ .] The title of my pull request is a short description of the requested changes.

@Jyothishree2k5  Thank you so much for your contribution! Your efforts are greatly appreciated and will go a long way in improving our project. Please feel free to share it with others and help us grow even further!


[SUPPORT] Small Table Upsert sometimes take a lot of time

Hi Guys,
Sometimes my Hudi upserts take so long, this table job used to run in less than 2 minutes and I have this behavior for all tables, I think it is a delete operation to remove old commits, but why take so long? only 20mb files

hudi_options = {
'hoodie.table.name': table_name,
'hoodie.datasource.write.recordkey.field': 'id',
'hoodie.datasource.write.table.name': table_name,
'hoodie.datasource.write.operation': 'upsert',
'hoodie.combine.before.upsert': 'true',
'hoodie.datasource.write.precombine.field': 'LineCreatedTimestamp',
'hoodie.parquet.small.file.limit': 200000000,
'hoodie.parquet.max.file.size': 256000000,
'hoodie.parquet.block.size': 256000000,
'hoodie.cleaner.commits.retained': 10,
'hoodie.datasource.hive_sync.enable': 'true',
'hoodie.datasource.hive_sync.table': table_name,
'hoodie.datasource.write.keygenerator.class': 'org.apache.hudi.keygen.NonpartitionedKeyGenerator',
'hoodie.datasource.hive_sync.database': 'datalake_raw',
'hoodie.datasource.hive_sync.jdbcurl': 'jdbc:hive2://ip-10-0-94-214.us-west-2.compute.internal:10000',
'hoodie.copyonwrite.record.size.estimate': 512,
'hoodie.insert.shuffle.parallelism': 10,
'hoodie.upsert.shuffle.parallelism': 10
}

@rubenssoto  how are you writing? could you paste the spark submit command ?

Hi @bhasudha ,
Yes, this is a simple job, reads parquet generated by aws DMS and writes Hudi.
My spark submit:
spark-submit --deploy-mode cluster --conf spark.dynamicAllocation.cachedExecutorIdleTimeout=60s --conf spark.dynamicAllocation.executorIdleTimeout=60s --conf spark.dynamicAllocation.maxExecutors=1 --conf spark.executor.memoryOverhead=2048 --conf spark.executor.cores=3 --conf spark.executor.memory=10g --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.sql.hive.convertMetastoreParquet=false --packages org.apache.hudi:hudi-spark-bundle_2.11:0.5.3,org.apache.spark:spark-avro_2.11:2.4.4 --py-files python_modules main.py poc_available_history

Hudi.zip
I think it is the commit files.

This is my write function:
def write_hudi_dataset(spark_data_frame, write_folder_path, hudi_options, write_mode):
spark_data_frame \
                  .write \
                  .options(**hudi_options) \
                  .mode(write_mode) \
                  .format('hudi')\
                  .save(write_folder_path)

I observed slow upsert:
val options = HudiOptionFactory.create(tableName = tableName,
tableType = HudiTableType.COW,
recordKeyField = "ulid",
partitonPathField = "",
preCombineField = "sid",
insertDropDups = false,
keyGeneratorType = HudiKeyGeneratorType.Nonpartitioned,
partitionValueExtractorType = HudiPartitionValueExtractorType.NonPartitioned
)
val helper = HudiHelper(options, log)
/**
 * insert perf
 */
val listInfo = spark.createDataFrame(Seq(
  (Some(1), Some("11"), Some("111")),
  (Some(2), Some("11"), Some("444"))
)).toDF("zid", "sid", "ulid")
var t1 = ZonedDateTime.now
helper.insert(listInfo, path, saveMode = SaveMode.Overwrite)
var t2 = ZonedDateTime.now
// Duration.between(t1, t2).getSeconds is about 5 seconds here

/**
 * upsert perf
 */
val listInfo2 = spark.createDataFrame(Seq(
  (Some(1), Some("22"), Some("111"))
)).toDF("zid", "sid", "ulid")

t1 = ZonedDateTime.now
helper.upsert(
  df = listInfo2,
  path = path
)
t2 = ZonedDateTime.now
// Duration.between(t1, t2).getSeconds is about 131 seconds here

This discounts the benefit of using upsert to partial update a big dataset. Thanks for any help.

@jiangok2006 : Can you enable INFO log and paste the logs here. Also, can you attach spark UI screen shots at job, stage and task levels.

@rubenssoto : can you please respond to Balaji's comment when you can.
@jiangok2006 : can you please respond to Balaji's comment when you can.

@rubenssoto : can you please respond to Balaji's comment when you can.
@jiangok2006 : can you please respond to Balaji's comment when you can.

My problem is an old problem, so you could close this issue.

My problem is an old problem, so you could close this issue.

@jiangok2006 : since the original post was from rubenssoto, would you mind creating a new support ticket w/ you issue. We can close this out.

@jiangok2006 : since the original post was from rubenssoto, would you mind creating a new support ticket w/ you issue. We can close this out.

Closing this ticket due to inactivity. @jiangok2006 please re-open if your slow issue continues to persist.


Update full description for Fastlane's support

It's a Fastlane with again Gemfile liked it, was application for the aboutic FAQ.

Merged in #41


Patch for Windows

Submitting this pull request to make #468 easier to review.
Updates #406, fixes #50
@Solanar et al. created a cluster.py file that allows django_q to work with Windows 10; this patch updates that file for the most recent version [1.3.2] of django_q.
I'm not a GitHub wizard (yet), so I can't speak for the test_*.py files in this pull request; however, I've been able to run django_q with the following (note that I'm not setting sync to true in settings.py):
Windows 10 Home x64

version 1909
OS build 18363.959
AMD Ryzen 7 3700U
8 GB RAM

Django 3.0.6
settings.py:
...
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
    }
}

Q_CLUSTER = {
    'recycle': 500,
    'orm': 'default',
}

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.db.DatabaseCache',
        'LOCATION': 'q_cluster_cache',
    }
}
...

Django_Q 1.3.2

Codecov Report

Merging #469 into master will decrease coverage by 1.06%.
The diff coverage is 48.10%.

@@            Coverage Diff             @@
##           master     #469      +/-   ##
==========================================
- Coverage   91.03%   89.97%   -1.07%
==========================================
  Files          46       46
  Lines        3022     3081      +59
==========================================
+ Hits         2751     2772      +21
- Misses        271      309      +38

Impacted Files
Coverage Δ

django_q/cluster.py
85.00% <43.05%> (-6.71%)
:arrow_down:

django_q/tests/test_brokers.py
72.01% <100.00%> (+0.31%)
:arrow_up:

django_q/tests/test_monitor.py
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c838a1e...88e63f1. Read the comment docs.

works for me with pipenv install https://github.com/claypooj21/django-q/archive/master.zip. thank you so much for this work, maybe one day we'll be lucky to have it merged
my suggestion would be to add a command to run the worker without any clustering/forking/multiprocessing at all. it'd be quite useful for local development and debugging (much more useful than sync=True, including for cases such as schedules), without any additional complexity of this fork-based clustering. in some cases, it might even be more convenient for production, allowing to dedicate the watch over processes to a dedicated process manager


OPA- Unable to create constraint

What kind of request is this (question/bug/enhancement/feature request):
Bug
Steps to reproduce (least amount of steps as possible):
From dashboard -> install OPA
Create a constraint from form:

Create
Result:
User is unable to create constraint:

This behavior is also seen when all parameters and fields are empty
Environment information

Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): master-head version db8a864
Installation option (single install/HA): HA

Refreshing the page will get past this behavior. This error is only seen after navigating to the constraint create page after a fresh install of OPA.

Because this is occurring in getters and resolved with a refresh I believe this is a syncing issue so I'm reassigning.

The bug fix is verified with Rancher server rancher\rancher:master-head version ec04f7878
Steps:
From dashboard -> install OPA
Without refreshing the page, navigate to OPA Gatekeeper
Create a constraint from form
Result:
Constraint is created and enforced


Rewriting processor module in Kotlin

Doing in kotlin branch.

After releasing 1.2.0. Restart this issue. I'm gonna recreate kotlin branch again.

I have started working on the Kotlin implementation and enhancements in my fork (not pushed yet, though)! I will inform you once it's available.

Wow amazing! You should push it!

Available now in the kotlin-new branch. Most of the test suite needs to be ported, but other than that the foundation for future development in Kotlin is created!

I created wip pull request so close this issue.


Missing ethers subpackages in package.json

Background
Typechain currently depends on the umbrella ethers package in package.json for its code
https://github.com/ethereum-ts/TypeChain/blob/fc9182b33beceb0037607a00d8508c0d1419215c/packages/target-ethers-v5/package.json#L37
but outputs subpackage import statements in the codegen.
https://github.com/ethereum-ts/TypeChain/blob/fc9182b33beceb0037607a00d8508c0d1419215c/packages/target-ethers-v5/src/codegen/index.ts#L34-L36
Usecase
In order to satisfy the imports in the codegen'ed files, we must install these subpackages ourselves or our build will fail.
Ideas
In general, importing packages that aren't listed in package.json can lead to weird bugs. Since the codegen output requires these subpackages, they should be included as explicit peerDependencies of typechain.

I think it's a valid point 👍 I will try to squeeze it in https://github.com/ethereum-ts/TypeChain/pull/363


Add parsing of type alias statements

Extends #95
Part of #82
Adds parsing of new type soft keyword for defining type aliases.
Supports type alias statements as defined in PEP 695 e.g.
# A non-generic type alias
type IntOrStr = int | str

# A generic type alias
type ListOrSet[T] = list[T] | set[T]

# A type alias that includes a forward reference
type AnimalOrVegetable = Animal | "Vegetable"

# A generic self-referential type alias
type RecursiveList[T] = T | list[RecursiveList[T]]

All type parameter kinds are supported as in #95.
Builds on soft keyword abstractions introduced in https://github.com/RustPython/RustPython/pull/4519

Nice job figuring this out. A couple questions to confirm my understanding of the soft-keyword heuristics.


Build(deps-dev): Bump @typescript-eslint/parser from 5.9.0 to 5.9.1 (#2326)

Pull request opened by the merge tool on behalf of #2326

Merged in 5a5237674e7e651b0ce5bbb6bb789b9529fc1364


(Gradient) Colors Not Applying for LQI (Link Quality Indicator - Zigbee2MQTT) Values (0–255)

Describe what is wrong
I am using the Battery State Card to display Zigbee Link Quality Indicator (LQI) values with dynamic gradient colors and icons. While the icons work as expected, the gradient colors do not behave as intended. The card fails to reflect the gradient for the defined steps (red, yellow, green) even though my LQI values range from 0 to 255.
Here are the steps I have tried so far:

Set default_state_formatting: false to prevent default formatting from interfering.
Used state_map to scale the LQI range (0-255) to percentages (0-100).
Configured colors with and without gradient: true, as well as with different numbers of steps.
Defined steps using both color names (e.g., red) and HEX values (e.g., #FF0000).
Configured steps with and without value fields.
Changed the order of steps to ensure proper threshold interpretation.
Tested with and without value_override.
Attempted different min and max settings for the colors section.

None of these approaches resolved the issue. The colors remain static and do not dynamically reflect the defined thresholds or the gradient behavior.
YAML configuration
type: custom:battery-state-card
secondary_info: "{last_changed}"
icon: >-
  mdi:signal-cellular-{state| abs()| lessthan(64,outline)| between(64,128,1)|
  between(128,192,2)| greaterthan(191,3) }
default_state_formatting: false
unit: LQI
round: 0
colors:
  min: 0
  max: 255
  steps:
    - color: red
    - color: yellow
    - color: green
  gradient: true
state_map:
  - from: 0
    to: 0
  - from: 128
    to: 50
  - from: 255
    to: 100
filter:
  include:
    - name: entity_id
      value: sensor.*_linkquality
  exclude:
    - name: entity_id
      value: binary_sensor.*
    - name: entity_id
      value: sensor.2312dra50g*
sort:
  - by: state
    desc: false
collapse: 8
bulk_rename:
  - from: " Linkquality"
  - from: " Signal"
  - from: " strength"
  - from: " Rssi"
  - from: " numeric"

Screenshot

Version
Card Version: 3.2.1 (Installed via HACS)
Core 2024.12.2
Supervisor 2024.11.4
Operating System 14.0
Frontend 20241127.7
Debug Output from Example Sensor
{
  "entity_id": "sensor.bad_fenster_linkquality",
  "state": "120",
  "attributes": {
    "state_class": "measurement",
    "unit_of_measurement": "lqi",
    "icon": "mdi:signal",
    "friendly_name": "Bad Fenster Linkquality"
  },
  "context": {
    "id": "01JETMQ3AB9KN5830F1NYPNSV4",
    "parent_id": null,
    "user_id": null
  },
  "last_changed": "2024-12-11T10:37:27.755Z",
  "last_updated": "2024-12-11T10:37:27.755Z",
  "display": {
    "entity_id": "sensor.bad_fenster_linkquality",
    "device_id": "68309f1c029fc37dcb8f97cebfc9fb2a",
    "labels": [],
    "platform": "mqtt",
    "entity_category": "diagnostic",
    "name": "Linkquality"
  },
  "device": {
    "area_id": "bad",
    "configuration_url": null,
    "config_entries": [
      "01JDD9Q5J5XY8QTDKA8HSTYDXX"
    ],
    "connections": [],
    "created_at":<PHONE_NUMBER>.788037,
    "disabled_by": null,
    "entry_type": null,
    "hw_version": null,
    "id": "68309f1c029fc37dcb8f97cebfc9fb2a",
    "identifiers": [
      [
        "mqtt",
        "zigbee2mqtt_0xa4c138fff7f46111"
      ]
    ],
    "labels": [],
    "manufacturer": "Tuya",
    "model": "Contact sensor (WL-19DWZ)",
    "model_id": null,
    "modified_at":<PHONE_NUMBER>.72391,
    "name_by_user": null,
    "name": "Bad Fenster",
    "primary_config_entry": "01JDD9Q5J5XY8QTDKA8HSTYDXX",
    "serial_number": null,
    "sw_version": "0122052017",
    "via_device_id": "4885be2b1f71f0dbc262e14df82076c4"
  },
  "area": {
    "aliases": [],
    "area_id": "bad",
    "floor_id": null,
    "icon": "mdi:shower",
    "labels": [],
    "name": "Bad",
    "picture": null,
    "created_at":<PHONE_NUMBER>.411868,
    "modified_at":<PHONE_NUMBER>.959435
  }
}

I would greatly appreciate any guidance on how to resolve this issue. If there are additional settings or adjustments I may have overlooked, please let me know!

Onverted my Feature Request into this Question and Closed the old Feature Request.


[ITensorGaussianMPS] Fix keyword argument forwarding issue

Fix keyword argument forwarding issue exposed by the keyword argument refactor in #1226.

[test ITensorGaussianMPS] [test ITensors mps]

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (66f5d39) 85.28% compared to head (5a6fdcc) 54.43%.

:exclamation: Current head 5a6fdcc differs from pull request most recent head 3d2db08. Consider uploading reports for the commit 3d2db08 to get more accurate results

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@             Coverage Diff             @@
##             main    #1228       +/-   ##
===========================================
- Coverage   85.28%   54.43%   -30.85%
===========================================
  Files          89       88        -1
  Lines        8445     8392       -53
===========================================
- Hits         7202     4568     -2634
- Misses       1243     3824     +2581

see 41 files with indirect coverage changes

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Hide keyboard on spinner opening

Description
MA-2458
Topics spinner while creating a new discussion post causes the spinner
popup to appear over the already displayed keyboard. Which causes
irregular behaviour upon orientation change as well.
This PR solves this issue by always dismissing keyboard whenever
spinner is tapped.
Reviewers
If you've been tagged for review, please check your corresponding box once you've given the :+1:.

 Last update 1e1bd47...67b74f9

LGTM :+1

LGTM :+1

LGTM :+1

Hmm, this has a secondary animation effect, but I suppose this is the best way to solve the issue.


Create Design Document for Elearning

Create a document, with Champa, that provides design directions for the creation of training.
The document will include:

Sample screens

Marketing approved colors

Marketing approved fonts

Target Mid-February completion.

Meeting with Champa week of 2/17.

Draft received from Champa. Meeting with her tomorrow (Feb 20) to discuss.

Provided feedback to Champa and expect to have a final draft in the next few work days.

Completed Design Guide can be found on this page: https://cockroachlabs.atlassian.net/wiki/x/i4DkDw


Add numblocks= argument to normalize_chunks

Precursor to https://github.com/dask/dask/issues/9412. This seemed more useful than inlining all the necessary logic into from_delayed.
cc @ian-r-rose @jrbourbeau @jsignell

[x] Tests added / passed
[x] Passes pre-commit run --all-files

I agree with your trepidation. I tried to avoid doing this, but my implementation without it ended up copying so much from normalize_chunks that I grudgingly made this change. You can take a look at https://github.com/dask/dask/pull/9417 and see if you can think of a way to avoid it, but supporting the many possibilities of numeric chunk, tuple chunksize, explicit tuple of tuples, mixed tuple of ints and tuple of tuples, all of that with the possibility of lists or ndarrays, was not something I wanted to write test coverage for :)
I was hoping that this might at least be useful to something somewhere else in the future.


updated location and hash of ubuntu server image

Title mostly self-explanatory.
Thank you for making these templates available.

Just realized that @NBardelot pulled up the issue and fixed it.


two document fixes.

Closing as already fixed/irrelevant. The last documentation block is in the docs at http://docs.deis.io/en/latest/installing_deis/install-platform/#install-the-deis-platform. Thanks for the PR, though :)


use apache toree

we are still using the spark-kernel package, but should update to use toree.
@parente should we switch the base image from minimal to one of the spark-including stacks, rather than duplicate that installation here?

I was thinking about that. We could start from all-spark-notebook. It'll require some rejiggering of the dockerfile here. Or we can just keep the spark links here + the pip install for toree.
Probably want to try starting from all-spark-notebook and see how much that impacts the dockerfile here.

Starting from all-spark sounds like a good idea.
I only just now realized all-spark might be a transformers reference.

I only just now realized all-spark might be a transformers reference.

I snuck the name in there back in July and waited many long months for someone to catch it.  A trophy emoji isn't magical enough for you? Fine.


multiple GPUs accessing the same part of the dataset

We find that multiple GPUs are accessing the same part of the dataset while testing the MLPerf Benchmark v1.0.

Our environment:

mlperf storage v1.0-branch (https://github.com/mlcommons/storage/tree/v1.0-branch )
dlio_benchmark @99d3b0 (https://github.com/argonne-lcf/dlio_benchmark/tree/99d3fb0863983b0681ddca612978e05824bb6833, mlperf storage v1.0-branch reference)

Experimental settings:
workload: unet3d
num_files_train: 500
read_threads: 1
test command:
./benchmark.sh run --hosts x.x.x.x --workload unet3d --accelerator-type h100 --num-accelerators 10 --results-dir /result --param dataset.num_files_train=500 --param dataset.data_folder=/data

Result:
We added the output of access information in reader_handle.py , printing the file name and pid. Below are the results for two processes (GPUs).
Process 2070861:
pid and filename:  2070861 /home/mlperf_data/unet3d_500/train/img_329_of_500.npz 1
pid and filename:  2070861 /home/mlperf_data/unet3d_500/train/img_475_of_500.npz 1
pid and filename:  2070861 /home/mlperf_data/unet3d_500/train/img_382_of_500.npz 1
pid and filename:  2070861 /home/mlperf_data/unet3d_500/train/img_157_of_500.npz 1
pid and filename:  2070861 /home/mlperf_data/unet3d_500/train/img_367_of_500.npz 2
pid and filename:  2070861 /home/mlperf_data/unet3d_500/train/img_418_of_500.npz 2
pid and filename:  2070861 /home/mlperf_data/unet3d_500/train/img_232_of_500.npz 2
pid and filename:  2070861 /home/mlperf_data/unet3d_500/train/img_354_of_500.npz 2
pid and filename:  2070861 /home/mlperf_data/unet3d_500/train/img_074_of_500.npz 3
pid and filename:  2070861 /home/mlperf_data/unet3d_500/train/img_338_of_500.npz 3
pid and filename:  2070861 /home/mlperf_data/unet3d_500/train/img_468_of_500.npz 3
pid and filename:  2070861 /home/mlperf_data/unet3d_500/train/img_458_of_500.npz 3
Process 2070579:
pid and filename: 2070579 /home/mlperf_data/unet3d_500/train/img_329_of_500.npz 1
pid and filename:  2070579 /home/mlperf_data/unet3d_500/train/img_475_of_500.npz 1
pid and filename:  2070579 /home/mlperf_data/unet3d_500/train/img_382_of_500.npz 1
pid and filename:  2070579 /home/mlperf_data/unet3d_500/train/img_157_of_500.npz 1
pid and filename:  2070579 /home/mlperf_data/unet3d_500/train/img_367_of_500.npz 2
pid and filename:  2070579 /home/mlperf_data/unet3d_500/train/img_418_of_500.npz 2
pid and filename:  2070579 /home/mlperf_data/unet3d_500/train/img_232_of_500.npz 2
pid and filename:  2070579 /home/mlperf_data/unet3d_500/train/img_354_of_500.npz 2
pid and filename:  2070579 /home/mlperf_data/unet3d_500/train/img_074_of_500.npz 3
pid and filename:  2070579 /home/mlperf_data/unet3d_500/train/img_338_of_500.npz 3
pid and filename:  2070579 /home/mlperf_data/unet3d_500/train/img_468_of_500.npz 3
pid and filename:  2070579 /home/mlperf_data/unet3d_500/train/img_458_of_500.npz 3

The above results show that different GPUs have accessed the same part of the dataset, which should ideally be non-overlapping for different datasets. Should we consider using my_rank and comm_size to split the dataset when generating dataloader in dlio_benchmark/dlio_benchmark/data_loader/torch_data_loader.py?

@yxkyong Thank you. I confirmed the observation. We will fix that.
@hariharan-devarajan Could you take a look? This has to do with the build sample index

@zhenghh04 Thank you. Will the dilo benchmark branch in mlperf storage v1.0 continue to use 99d3fb0 or another branch?

@yxkyong could you check whether this PR#166 fixes the issue?

@yxkyong could you check whether this PR #166 fixes the issue?

Thank you for addressing the issue! I just checked the latest update, my problem has been resolved.  We've encountered an issue with the global index map. Could you take a look issue: https://github.com/argonne-lcf/dlio_benchmark/issues/167?

Closing this issue now. The other issue is addressed with another PR as mentioned.


Netflix with Ads messes up the timing of subtitles.

I don't have many details, since I haven't watched much on Netflix, but I know this is an issue that I ran into a few times in the past week or so. Asbplayer is not aware there is an ad playing so it assumes the video is still chugging along. and displays subtitles over the ad, and then once the ad is over it does not correct itself. I'm not 100% certain if refreshing the page works, but I know for certain that re-loading the subtitles through asbplayer does not fix the desync.
I may be willing to work on fixing this or providing more detail since I have ad-supported Netflix through a mobile plan and japanese Netflix is a goldmine it seems...

This issue exists on other platforms where "extra video" such as ads (Amazon Prime) or logo animations (Disney Plus) are played prior to the actual video. Because extra video is essentially appended to the beginning of the stream, currentTime property of the video stops matching up with the actual video content. So far I have not been able to find a way to work around this but I haven't spent a lot of time looking into. it.


Invalid example json fix

Description
Example under radios and radiobuttons had invalid json. I fixed it.
@json-schema-form/angular-schema-form-lead
Checklist

[x]  I have read and understand the CONTRIBUTIONS.md file
[x]  I have searched for and linked related issues
[ ]  I have created test cases to ensure quick resolution of the PR is easier
[ ]  I am NOT targeting main branch
[x]  I did NOT include the dist folder in my PR

The remaining 2 elements on the checklist is unnecessary, because It's in an md file.

@stsatlantis thanks for that, love doc update PRs, appreciate your time :)
I have applied the change to development branch, so I will close this PR.


icu  port on Windows creates in  its buildtree files with names that cannot be deleted from command line with Powershell or Git Bash

Describe the bug
icu port on Windows creates in its buildtree files with names that cannot be deleted from command line with Powershell or Git Bash
Environment

OS: Windows
Compiler: VS2019

To Reproduce
On Git Bash:
./vcpkg.exe install icu:x64-windows
rm -rf buildtrees

This will fail with the following message:
2020-03-09T17:54:41.5736150Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-dbg/common': Directory not empty
2020-03-09T17:54:42.6218293Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-dbg/data': Directory not empty
2020-03-09T17:54:42.6242660Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-dbg/extra/uconv': Directory not empty
2020-03-09T17:54:42.7189320Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-dbg/i18n': Directory not empty
2020-03-09T17:54:42.7218943Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-dbg/io': Directory not empty
2020-03-09T17:54:42.7395213Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-dbg/stubdata': Directory not empty
2020-03-09T17:54:42.7540731Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-dbg/tools/ctestfw': Directory not empty
2020-03-09T17:54:42.7559696Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-dbg/tools/genbrk': Directory not empty
2020-03-09T17:54:42.7560696Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-dbg/tools/genccode': Directory not empty
2020-03-09T17:54:42.7568641Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-dbg/tools/gencfu': Directory not empty
2020-03-09T17:54:42.7584840Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-dbg/tools/gencmn': Directory not empty
2020-03-09T17:54:42.7598746Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-dbg/tools/gencnval': Directory not empty
2020-03-09T17:54:42.7609501Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-dbg/tools/gendict': Directory not empty
2020-03-09T17:54:42.7624626Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-dbg/tools/gennorm2': Directory not empty
2020-03-09T17:54:42.7658514Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-dbg/tools/genrb': Directory not empty
2020-03-09T17:54:42.7667980Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-dbg/tools/gensprep': Directory not empty
2020-03-09T17:54:42.7692643Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-dbg/tools/icuinfo': Directory not empty
2020-03-09T17:54:42.7700635Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-dbg/tools/icupkg': Directory not empty
2020-03-09T17:54:42.7719057Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-dbg/tools/makeconv': Directory not empty
2020-03-09T17:54:42.7731100Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-dbg/tools/pkgdata': Directory not empty
2020-03-09T17:54:42.7791331Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-dbg/tools/toolutil': Directory not empty
2020-03-09T17:54:42.8333276Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-rel/common': Directory not empty
2020-03-09T17:54:43.9256518Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-rel/data': Directory not empty
2020-03-09T17:54:43.9280921Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-rel/extra/uconv': Directory not empty
2020-03-09T17:54:44.0025014Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-rel/i18n': Directory not empty
2020-03-09T17:54:44.0054658Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-rel/io': Directory not empty
2020-03-09T17:54:44.0166092Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-rel/stubdata': Directory not empty
2020-03-09T17:54:44.0282365Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-rel/tools/ctestfw': Directory not empty
2020-03-09T17:54:44.0294920Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-rel/tools/genbrk': Directory not empty
2020-03-09T17:54:44.0304162Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-rel/tools/genccode': Directory not empty
2020-03-09T17:54:44.0311362Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-rel/tools/gencfu': Directory not empty
2020-03-09T17:54:44.0318633Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-rel/tools/gencmn': Directory not empty
2020-03-09T17:54:44.0327805Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-rel/tools/gencnval': Directory not empty
2020-03-09T17:54:44.0337758Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-rel/tools/gendict': Directory not empty
2020-03-09T17:54:44.0352058Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-rel/tools/gennorm2': Directory not empty
2020-03-09T17:54:44.0388455Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-rel/tools/genrb': Directory not empty
2020-03-09T17:54:44.0399722Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-rel/tools/gensprep': Directory not empty
2020-03-09T17:54:44.0422974Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-rel/tools/icuinfo': Directory not empty
2020-03-09T17:54:44.0431967Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-rel/tools/icupkg': Directory not empty
2020-03-09T17:54:44.0447936Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-rel/tools/makeconv': Directory not empty
2020-03-09T17:54:44.0461990Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-rel/tools/pkgdata': Directory not empty
2020-03-09T17:54:44.0517702Z rm: cannot remove 'C:/idjl/vcpkg/buildtrees/icu/x64-windows-rel/tools/toolutil': Directory not empty

If you check, it seems that those directories contain a  file named C:.
Deleting the folder is also impossible in powershell (see https://github.com/iit-danieli-joint-lab/idjl-software-dependencies-vcpkg/runs/495928460) while it works fine with cmd prompt (https://github.com/iit-danieli-joint-lab/idjl-software-dependencies-vcpkg/pull/5/checks?check_run_id=496262390).
Expected behavior
I expect the delete to work correctly.
Failure logs
See https://github.com/iit-danieli-joint-lab/idjl-software-dependencies-vcpkg/runs/495838460 .
Additional context
To be honest, I do not know if the problem is in vcpkg's icu  port, in icu buildsystem or in  Git Bash, but I think it any case it would be good to have an issue to track this if someone faces this problem. Feel free to close if you think it is not directly related to vcpkg.

@traversaro Thanks for posting issue. I reproduced this issue on my vmware. I found:

The powershell remove the directory failed due to file icu-config is a read-only file.

And git bash remove the directory failed due to the directory name has a special symbol.

I try to use powershell and git bash to remove other ports source directory, and I remove successfully. I think the files generated by the building port is based on the source file.

To clarify, i reported the issue because "deleting buildtrees" is document as a way to save space: https://github.com/microsoft/vcpkg/blob/2020.01/docs/about/faq.md#how-can-i-remove-temporary-files .

This is not a VCPKG bug, so I am closing this issue for now.

This is not a VCPKG bug, so I am closing this issue for now.

Hi @LilyWangL , while this is not a vcpkg bug, it affects vcpkg users that follow the vcpkg documentation in https://github.com/microsoft/vcpkg/blob/2020.01/docs/about/faq.md#how-can-i-remove-temporary-files . Perhaps it could make sense to document in the FAQs that using Git Bash or Powershell is not supported to delete the buildtrees?

Perhaps it could make sense to document in the FAQs that using Git Bash or Powershell is not supported to delete the buildtrees?

The powershell remove the directory failed due to file icu-config is a read-only file.

Upstream issue. The portfile could ensure that the offending files are not left in the buildtrees directory with the read-only attribute flag set but only if this issue cannot be solved by reporting it upstream.

And git bash remove the directory failed due to the directory name has a special symbol.

This would appear to be something that needs to be patched out of the source code either upstream or by the port if upstream can't/won't for whatever reason.
It is also a git bash problem which should be reported to the developers.
I don't really understand the calls for support of git bash on Windows when powershell and the command prompt are available. Command prompt has worked in pretty much the same way since DOS through all versions of Windows and basic command scripting should have been taught at schools for a considerable number of years by now.
If the Windows alternative to bash was a python shell however..

I don't really understand the calls for support of git bash on Windows when powershell and the command prompt are available. Command prompt has worked in pretty much the same way since DOS through all versions of Windows and basic command scripting should have been taught at schools for a considerable number of years by now.

Hi @heydojo , I am not sure what you refer to as a "calls for support". Git Bash support is explicitly supported by vcpkg (as the boostrap.sh script works fine), and  I think that opening an issue on the vcpkg issue tracker on a specific problem occurring on vcpkg is helpful for vcpkg users, for which the equivalent upstream report may not be as easily found or understandable. If any of the vcpkg maintainer instead feel that these kind of issue are off topic, feel free to close them, thanks!


FixEventSource

There were a few problems with EventSource, I updated the way it's implemented by switching from a C4EventSource object to extending NSObject. A couple of reasons for this:

It's easier to maintain between objects that inherit from C4View and those that don't... Now objects like C4CanvasController respond to events
The current implementation has a bug in it, and needed fixing anyways (i.e. objects were always listening to self and so weren't responding to events from other objects)
The post method was internal and wasn't accessible.

The following now works properly:
let s = C4Circle(frame: C4Rect(0,0,100,100))
canvas.add(s)

on(event: "ping") {
    s.opacity = random01()
}

canvas.addTapGestureRecognizer { (location, state) -> () in
    self.post("ping")
}


[Bug]: Error fetching GCP Access Token - java.net.UnknownHostException: Host not allowed

Is there an existing issue for this?

[X] I have searched the existing issues

Description
Context
We are evaluating Appsmith for building a custom solution for a customer. This will be a self-hosted installation with Appsmith deployed on GCP. It will use Postgres Database for storing the customers' objects which will be connected as datasource. We have a requirement to store, fetch, update some documents (Images, PDFs etc) in GCS.
Problem
To enable interaction with GCP Cloud Storage, I tried to create a simple curl-import API to fetch the access token from GCP's metadata server. What I have observed is that it throws a "java.net.UnknownHostException: Host not allowed" error. I have also tried this with a simple REST API database connector with no luck.
I ran a simple test with a curl command as shown below from the GCE VM as well as also the AppSmith container running on the same GCE VM; it seems to work fine. This only appears to be an issue from Appsmith application.

curl -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token

Doing a code search on the repository, I found this PR #34250 which appears to disallow access to GCP's Metadata Server.
Could you please advise if there is a way to workaround this perhaps with an environment variable. Please note that is critical for us as this blocking the deployment of Appsmith. Could you please let us know if S3 data source connector is supported with GCS and has been tested.
Alternatively, we'd be forced to probably create a proxy REST API which seems to be an overkill and will increase the development time for us.
Environment
We are using the Docker container of appsmith/appsmith-ee:latest   with image ID df6aca35b499. Additional details from Appsmith container logs are below for your reference
{
  "commitSha": "cc81d5df34d5109b317cf6703dc365d0a54aa20d",
  "commitUrl": "https://github.com/appsmithorg/appsmith-ee/commit/cc81d5df34d5109b317cf6703dc365d0a54aa20d",
  "githubRef": "refs/tags/v1.50",
  "githubRunUrl": "https://github.com/appsmithorg/appsmith-ee/actions/runs/12078597533/attempts/1",
  "version": "v1.50",
  "imageBuiltAt": "2024-11-29T03:51:08+00:00",
  "isCI": true
}

Please let me know if you need any further information
Steps To Reproduce

Deploy Appsmith on GCP using Docker-compose environment
Create a simple curl-import API to fetch the access token from GCP's metadata server with the following URL

curl -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token

Same can be tried with a REST API as well

Public Sample App
No response
Environment
Release
Severity
High (Blocker to building or releasing)
Issue video log
No response
Version
Self Hosted - v1.50

@ravir2m2, thanks for reaching out. This is by design. Requests to metadata.google.internal are blocked prevent SSRF attacks. The S3 datasource works with GCP and would be the recommended option to talk to GCS.

@sharat87 - Thanks for your prompt response. Was there a GCP related SSRF issue/bug that required this fix? Pardon my ignorance but this authentication to the API would still happen from Appsmith (on the server side) rather than from the UI client running in the browser, is that right?
Can you please recommend any other options to access Google Cloud APIs (other than GCS which can be worked around with S3 datasource) from Appsmith in a containerized environment, such as Kubernetes or Docker Compose based environment on GCE VMs on GCP?
I can think of the following, can you please share your thoughts?

Workload identity on Kubernetes - Most secure and recommend option on GCP, however requires access to metadata server and so may have the same issue as mentioned above
Less secure with GCP's application default credentials using environment variable GOOGLE_APPLICATION_CREDENTIALS and mounting service account key

Hey, sorry for the delay, just seeing this.

Could you clarify if a specific SSRF vulnerability related to GCP necessitated this design decision?

An SSRF was identified and fixed for AWS' metadata service, and as part of that effort we identified this for GCP as well, and added it to the block list.

I am exploring alternative methods for accessing Google Cloud APIs from Appsmith

Which APIs do you have in mind? If it's a native integration, like Firestore, I'd recommend using that integration directly. If you want to call generic GCP's HTTP APIs directly from Appsmith, I'd recommend using an API key method to do this. Like AWS' Access key and secret access key.
I'm not very familiar with GCP, but from what I gather, they are methods to give the Kubernetes Node, permissions to hit the GCP APIs, right? That means any process on the node can now access those APIs. Correct?
If that is right, it is important to note that, that means any user on your Appsmith can now access those APIs. GCP can't differentiate an API call from one Appsmith application from another. This authentication method cannot give workspace-specific permissions. That's why an API key way of authentication (un-intuitively) ends up being more secure for Appsmith.


Multiple date pickers on the page creating performance issue

@Hi team, @DanielYKPan
I am using this date picker in my form which has many input and datepicker fields (20+)and can grow even more based on the user input. As pickers count increase, web page is loaded slowly and performance is very low. Is this because of new datepicker object instance created for each picker?
Could you please help to resolve this issue?
Thanks,
Ceepee

@DanielYKPan  Hi,
Any updates ? Its kinda blocker for me? It would be nice if you give me work around.
If the date pickers are more in count in a page then, page is loaded very very slow . What should be done?


Question about how to incorporate upsample layer for stax

Hi Every,
This issue is related #1889 and #862:
I am following @mattjj suggestion in #862 and it works perfectly (please see below). But when I try to make it a stax layer, I have difficulties. I guess there is something in stax that I don't understand and this is why it is not working.
The layer works when it is standalone but I serialize it with a convolution layer, it complains about the shape
This is the code:
from __future__ import print_function, division, absolute_import
import jax.numpy as np
from jax import grad, jit, vmap
from jax import random

import matplotlib.pylab as plt

from jax.experimental import stax
from jax.experimental.stax import AvgPool, BatchNorm, Conv

def _interpolate_bilinear(im, rows, cols):

  # based on http://stackoverflow.com/a/12729229
  col_lo = np.floor(cols).astype(int)
  col_hi = col_lo + 1
  row_lo = np.floor(rows).astype(int)
  row_hi = row_lo + 1

  nrows, ncols = im.shape[-3:-1]
  def cclip(cols): return np.clip(cols, 0, ncols - 1)
  def rclip(rows): return np.clip(rows, 0, nrows - 1)
  Ia = im[..., rclip(row_lo), cclip(col_lo), :]
  Ib = im[..., rclip(row_hi), cclip(col_lo), :]
  Ic = im[..., rclip(row_lo), cclip(col_hi), :]
  Id = im[..., rclip(row_hi), cclip(col_hi), :]

  wa = np.expand_dims((col_hi - cols) * (row_hi - rows), -1)
  wb = np.expand_dims((col_hi - cols) * (rows - row_lo), -1)
  wc = np.expand_dims((cols - col_lo) * (row_hi - rows), -1)
  wd = np.expand_dims((cols - col_lo) * (rows - row_lo), -1)

  return wa*Ia + wb*Ib + wc*Ic + wd*Id

def _upsample(img,resize_rate):
   nrows, ncols = img.shape[-3:-1]
   delta = 0.5/resize_rate

   rows = np.linspace(delta,nrows-delta, np.int32(resize_rate*nrows))
   cols = np.linspace(delta,ncols-delta, np.int32(resize_rate*ncols))
   ROWS, COLS = np.meshgrid(rows,cols,indexing='ij')

   img_resize_vec = _interpolate_bilinear(img, ROWS.flatten(), COLS.flatten())
   img_resize =  img_resize_vec.reshape(random_image.shape[:-3] +
                                        (len(rows),len(cols)) +
                                        random_image.shape[-1:])

   return img_resize

def upsample(resize_rate):
  """Layer construction for upsampling"""
  def init_fun(rng, input_shape):
    nrows, ncols = input_shape[-3:-1]
    print(input_shape[:-3] + ( np.int32(resize_rate*nrows), np.int32(resize_rate*ncols)) + input_shape[-1:])
    output_shape = input_shape[:-3] + ( np.int32(resize_rate*nrows), np.int32(resize_rate*ncols)) + input_shape[-1:]
    return output_shape, ()
  def apply_fun(params, inputs, **kwargs):
    return _upsample(inputs,resize_rate)
  return init_fun, apply_fun

## first make data ready to test
!pip install -q --upgrade tfds-nightly tf-nightly

import tensorflow_datasets as tfds

data_dir = '/tmp/tfds'

# Fetch full datasets for evaluation
# tfds.load returns tf.Tensors (or tf.data.Datasets if batch_size != -1)
# You can convert them to NumPy arrays (or iterables of NumPy arrays) with tfds.dataset_as_numpy
mnist_data, info = tfds.load(name="mnist", batch_size=-1, data_dir=data_dir, with_info=True)
mnist_data = tfds.as_numpy(mnist_data)
train_data, test_data = mnist_data['train'], mnist_data['test']
num_labels = info.features['label'].num_classes
h, w, c = info.features['image'].shape
num_pixels = h * w * c

random_image = train_data['image'][:20,...]

This test passes
########## THE FOLLOWING TEST WORKS
init_testFcn2, apply_testFcn2 = stax.serial(  upsample(2) )
key = random.PRNGKey(1)
_, params = init_testFcn2(key, input_shape=random_image.shape)

plt.figure()
plt.imshow(random_image[0,:,:,0])
y =  apply_testFcn2(params, np.float32( random_image) )
print("y.shape : ", y.shape)

plt.figure()
plt.imshow(y[0,:,:,0])

But this fails
#Conv(3,(3,3),(2,2))
#f_init, f_apply = stax.serial(Conv(3,(1,1),(2,2)), upsample(2) , AvgPool((5,5)))
#f_init, f_apply = stax.serial(upsample(2) , Conv(3,(3,3),(2,2)) )
f_init, f_apply =  stax.serial(Conv(3,(3,3),(2,2)) , upsample(2) )
#f_init, f_apply  = stax.serial(GeneralConv(('NHWC','OIHW','NHWC'),3,(3,3),(2,2)),
#                               upsample(2) )

key = random.PRNGKey(1)
_, params = f_init(key, input_shape=random_image.shape)

plt.figure()
plt.imshow(random_image[0,:,:,0])
y =  f_apply(params, np.float32( random_image) )
print("y.shape : ", y.shape)

plt.figure()
plt.imshow(y[0,:,:,0])

this is the output:
(10, 26, 26, 3)
/usr/local/lib/python3.6/dist-packages/jax/lax/lax.py:4578: UserWarning: Explicitly requested dtype <class 'int'> requested in astype is not available, and will be truncated to dtype int32. To enable more dtypes, set the jax_enable_x64 configuration option or the JAX_ENABLE_X64 shell environment variable. See https://github.com/google/jax#current-gotchas for more.
  warnings.warn(msg.format(dtype, fun_name , truncated_dtype))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-99-18d3ea7106e2> in <module>()
      9 plt.figure()
     10 plt.imshow(random_image[0,:,:,0])
---> 11 y =  f_apply(params, np.float32( random_image) )
     12 print("y.shape : ", y.shape)
     13

6 frames
<__array_function__ internals> in reshape(*args, **kwargs)

/usr/local/lib/python3.6/dist-packages/numpy/core/fromnumeric.py in _wrapfunc(obj, method, *args, **kwds)
     59
     60     try:
---> 61         return bound(*args, **kwds)
     62     except TypeError:
     63         # A TypeError occurs if the object does have such a method in its

ValueError: cannot reshape array of size 20280 into shape (10,26,26,1)

I am confused! As you see in the print statement, the size should be correct. I don't know why it is complaining about reshaping. This is definitely my bug. I am trying to understand how stax works.
Thanks,

That was my bug due to workspace contamination in the Jupyter notebook!


Fix use-after-free by mrb_gc_unregistor()

Calling mrb_gc_unregistor() from mrb_data_type::dfree caused a use-after-free deep inside mrb_close().
The impetus to investigate was https://github.com/mruby/mruby/pull/6342#pullrequestreview-2292747530.
Currently, when mrb_close() is called, all objects are destroyed first.
The process is done heap page by heap page, and when all objects belonging to a heap page are destroyed, the heap page is released.
If the next heap page contains RData objects, the mrb_gc_unregistor() function may be called from the mrb_data_type::dfree function.
At this time, the mrb_gc_unregistor() function gets an array object from a Ruby global variable.
If the array object belongs to a freed heap page, use-after-free is established by referencing this array object.
About the fixes.
First of all, there is the fact that the mrb_gv_get() function returns nil if mrb->globals is NULL.
Therefore, before destroying all objects, free mrb->globals and set mrb->globals to NULL at the same time.
Now the mrb_gv_get() function will return nil to the calling mrb_gc_unregistor() function and mrb_gc_unregistor() will do nothing more.
ref. https://github.com/mruby/mruby/issues/4618

Reproduction code: uaf-gc-unregister.c
#if 0
#!/bin/sh

set -e

$(build/host/bin/mruby-config --cc --cflags) "$0" $(build/host/bin/mruby-config --ldflags --libs)

valgrind -- ./a.out
#lldb19 -- ./a.out

exit 0
#endif

#include <mruby.h>
#include <mruby/array.h>
#include <mruby/data.h>

#ifndef MRB_HEAP_PAGE_SIZE
# define MRB_HEAP_PAGE_SIZE 1024
#endif

struct mydata
{
  mrb_value ary;
};

static void
mydata_free(mrb_state *mrb, void *opaque)
{
  if (opaque) {
    struct mydata *p = (struct mydata *)opaque;
    mrb_gc_unregister(mrb, p->ary);
    mrb_free(mrb, p);
  }
}

static const mrb_data_type mydata_type = { "mydata", mydata_free };

int
main(int argc, char *argv[])
{
  mrb_state *mrb = mrb_open();

  struct RData *mydata_obj = mrb_data_object_alloc(mrb, mrb->object_class, NULL, NULL);
  mydata_obj->data = mrb_calloc(mrb, 1, sizeof(struct mydata));
  mydata_obj->type = &mydata_type;

  struct mydata *p = (struct mydata *)mydata_obj->data;
  p->ary = mrb_ary_new(mrb);
  int ai = mrb_gc_arena_save(mrb);

  // RData オブジェクトと "_gc_root_" グローバル変数オブジェクトを別のヒープページに確保する
  for (int i = 0; i < MRB_HEAP_PAGE_SIZE * 2; i++) {
    mrb_ary_push(mrb, p->ary, mrb_ary_new(mrb));
    mrb_gc_arena_restore(mrb, ai);
  }

  mrb_gc_register(mrb, p->ary);

  mrb_close(mrb);

  return 0;
}

Run through valgrind based on current master
% sh ./uaf-gc-unregister.c
==24341== Memcheck, a memory error detector
==24341== Copyright (C) 2002-2024, and GNU GPL'd, by Julian Seward et al.
==24341== Using Valgrind-3.23.0 and LibVEX; rerun with -h for copyright info
==24341== Command: ./a.out
==24341==
==24341== Invalid read of size 1
==24341==    at 0x26F3F7: mrb_gc_unregister (gc.c:452)
==24341==    by 0x25F25C: mydata_free (uaf-gc-unregister.c:32)
==24341==    by 0x26FD96: obj_free (gc.c:850)
==24341==    by 0x26F1B8: free_heap (gc.c:359)
==24341==    by 0x26F1B8: mrb_gc_destroy (gc.c:368)
==24341==    by 0x27D892: mrb_close (state.c:188)
==24341==    by 0x25F234: main (uaf-gc-unregister.c:60)
==24341==  Address 0x54df090 is 10,800 bytes inside a block of size 49,184 free'd
==24341==    at 0x484F2EC: free (vg_replace_malloc.c:993)
==24341==    by 0x27DD48: mrb_default_allocf (allocf.c:22)
==24341==    by 0x26F16E: mrb_free (gc.c:259)
==24341==    by 0x26F16E: free_heap (gc.c:361)
==24341==    by 0x26F16E: mrb_gc_destroy (gc.c:368)
==24341==    by 0x27D892: mrb_close (state.c:188)
==24341==    by 0x25F234: main (uaf-gc-unregister.c:60)
==24341==  Block was alloc'd at
==24341==    at 0x4852321: realloc (vg_replace_malloc.c:1806)
==24341==    by 0x26EFDD: mrb_realloc_simple (gc.c:197)
==24341==    by 0x26EFDD: mrb_realloc (gc.c:211)
==24341==    by 0x26EFDD: mrb_malloc (gc.c:227)
==24341==    by 0x26EFDD: mrb_calloc (gc.c:245)
==24341==    by 0x26EFDD: add_heap (gc.c:299)
==24341==    by 0x26EDA5: mrb_obj_alloc (gc.c:506)
==24341==    by 0x25F331: ary_new_capa (array.c:48)
==24341==    by 0x25F331: mrb_ary_new_capa (array.c:65)
==24341==    by 0x25F331: mrb_ary_new (array.c:72)
==24341==    by 0x25F207: main (uaf-gc-unregister.c:54)
==24341==
==24341==
==24341== HEAP SUMMARY:
==24341==     in use at exit: 0 bytes in 0 blocks
==24341==   total heap usage: 729 allocs, 729 frees, 261,916 bytes allocated
==24341==
==24341== All heap blocks were freed -- no leaks are possible
==24341==
==24341== For lists of detected and suppressed errors, rerun with: -s
==24341== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)


Automate Crossplane generation process

The idea is to automate as much as possible the provider's generation process:

Detect a new release from the Terraform provider repository with GitHub API and GitHub Actions & update the Terraform provider version in the Makefile of the Crossplane provider
Check for new resources added in the new release through diffs
If new resources are found, add them to the Crossplane provider with the help of code generation & boilerplate code

Greetings,
Are you guys implementing this automation soon to keep the Crossplane provider on par with Terraform's?
Thank you!


ui-builder: EACCES: permission denied, open '/.yarnrc'

I'm trying to build a docker image for ui-builder but I'm getting an error:
#0 0.424 warning Skipping preferred cache folder "/.cache/yarn" because it is not writable.
#0 0.424 warning Selected the next writable cache folder in the list, will be "/tmp/.yarn-cache-1001".
#0 0.431 error An unexpected error occurred: "EACCES: permission denied, open '/.yarnrc'".

@promo360 in #23 I simplify the usage model to standalone Node.js start as described in @kaspersky/ui-builder Readme file.
Please, check new approach.


[qt-advanced-docking-system] updated qt-advanced-docking-system to 3.2.1

Updates the library to the newest release which fixes several bugs and adds features.

I'm not sure why only the Linux version is saying that there is no CMakeLists.txt present while the Windows one can find it fine and I made no big changes to the project.

Hi @stijnherfst
Can you install this port on Linux platform successfully on your local?

It seems like this might be an issue on your end with qt5-base indeed not compiling properly.

We have been working on the regreassions on Linux about libhdfs3 ,osg-qt and try to fix them now.
But I'm not sure if this PR is realted to these regressions.
Once fixed, I will rerun this PR.
Let's wait for the new CI test results.

/azp run

The regressions on Linux like this:
CMake Error at /ci/myagent/_work/1/s/installed/x64-linux/share/cmake/Qt5Gui/Qt5GuiConfig.cmake:15 (message):
  The imported target "Qt5::Gui" references the file

     "/ci/myagent/_work/3/s/installed/x64-linux/lib/libEGL.so"

  but this file does not exist.  Possible reasons include:

  * The file was deleted, renamed, or moved to another location.

  * An install or uninstall procedure did not complete successfully.

  * The installation package was faulty and contained

     "/ci/myagent/_work/1/s/installed/x64-linux/share/cmake/Qt5Gui/Qt5GuiConfigExtras.cmake"

  but not all the files it references.

Since this error is related to qt5-base.
Due to conflicts with angle and qt5-base, ms-angel:x64-linux is set to fail now.
So currently, we can add qt-advanced-docking-system:x64-linux=fail to ci.baseline.txt.

Just to note, building under Linux works for me now so I think the update is ok.
Homepage line is now included too.

This issue should automatically be resolved at some point in the future and thus I think merging it should be fine.

So is this going to be merged? The problem with compilation on Linux seems to be due to another package that this depends on having issues and is not related to this package then. The previous version of this package should then also fail to build because one of its dependencies fails and thus this PR doesn't regress anything.

The changes look good to me. Right now, none of the Qt packages are getting downloaded on our CI for some reason. We're looking into it and trying to get it resolved ASAP. After that, we can merge this. I will tag this 'reviewed' so we don't forget about it.

@stijnherfst Can you resolve the conflict? we just fixed the Qt problem in our CI

@stijnherfst
Thanks for your update.
LGTM now. Once the regression about qt5-tools on Linux is fixed, I will tag this as 'Reviewed'.

That's an issue on your end, right?

/azp run

/azp run

LGTM, thanks for the PR!


Update & Improve CI

Hi, this CI updates the CI workflow file.
This PR:

Updates checkout v2 -> v3
Updates setup-go v2 -> v4
Configures setup-go to use the go version consistent with project setup via go.mod file
Delegates build caching to setup-go
Changes dependency installation to go mod download (official documentation, difference with the current go get)
Replaces the deprecated set-output
Uploads coverage file as an artifact, as currently the generated file is discarded

Btw, what's the use for Print Go version and environment? go env is displayed via setup-go step, printing environmental variables is not safe as it may contain secrets?

Thanks for this!
We print that stuff out mostly to aid in debugging. When playing with the CI config, it usually saves time to have immediate access to the envs as a reference.
go env didn't used to be printed I think. And I don't think there's any relevant secrets in the system env vars in this case (secrets typically get fed through CI variables which aren't in env).
Most of the things that look pointless in here are largely because we copy-pasted from https://github.com/caddyserver/caddy/blob/master/.github/workflows/ci.yml which has some additional steps.
Like short_sha isn't actually used in here, it was used in the caddy repo to attach the commit sha to the binary artifacts. This project is a lib, so there's no binary to publish.
I think we should probably be testing on latest Go + minimum from go.mod, so I'm not sure using go-version-file: 'go.mod' is ideal since that approach isn't compatible with a matrix.
I'm not sure publishing the coverage report on its own is that useful, we're not using any tooling to analyze it currently. I think Matt's opinion is that code coverage is usually a misleading metric, so it's not of great importance to us.

Oh, I just saw your PR in caddy's CI. Lucky we both found this out at (almost the same time).
Cool, I guess I will remove the short_sha thing, revert the go-version-file, and disable coverage reporting.

I don't think there's a minimum compatible go version declared in docs. I will just leave go version matrix in CI as is.
@francislavoie should I add  1.19, 1.20 as is in Caddy?

The minimum is what's in go.mod basically. I think we can test on 1.18 + 1.20 for now.

The minimum is what's in go.mod basically. I think we can test on 1.18 + 1.20 for now.

Updated

LGTM! I don't actually have merge permissions in this repo 😅

LGTM! I don't actually have merge permissions in this repo 😅

ops haha no worries


debug: filter exceptions from DAP telemetry on web

This change will filter DAP keys beginning with ! when the remote
authority is "other", which SteVen recommended as a way to tell if we're
on web/codespaces.
Isidor mentioned previously that there was filtering for keys beginning
with an underscore, but this was only applied to some data in error
responses, not general telemetry events. An alternative approach to this
is filtering exceptions based on the environment on the extension side,
but I figured it would be better to have VS Code do that so that we
don't end up with N many possibly deviant sets of environment detection
logic.

Looks good, great job 👏
Just left one minor comment, if you can tackle it great, if not no problem.


Fix the bug of command history

fix #13
Make the behavior of the command history more consistent with minecraft.

No empty or space-only messages will be recorded.
Consecutive messages with the same content are recorded as one.
"UserInputBufferCopy" will be updated even if the user has no input.
Cursor moves to the end after switching.
Reset "BufferBackreadPos" after pressing Enter.

Thank you for the contribution.
The command history is something that I wasn't really satisfied with. I haven't tested it rigorously enough, and I am glad to see that it has been fixed.


Feat/projection toggle

the button should be more like a toggle switch, similar to what max proposed in his draft:

And put it into the top header next to the info button

when someone hovers over the button the curser is currently as if you would want to write text, change it to pointer:
cursor: pointer;

in the prediction view the buttons are disabled. I would enable  them and if someone changes the view from globe to map or vice versa in the prediction view I would perform the same action as if they would click on the back button.

fyi when I say "prediction view" I mean this:

With "globe view" I mean this:

please change the base branch of the pr to "dev" instead of "main"


Problem hitting endpoints

When I try and hit any of the endpoints (including the one in test_posting.py, http://<IP_ADDRESS>:5000/get_user_scores), I get the attached error in the browser, and the message FileNotFoundError: [Errno 2] No such file or directory: './climate_mind_ontology'.  This is the latest code from master, but running the code after cloning from scratch from 'climatemind-backend' fails because of the requirements.txt issue I described previously.

@biotom  now that docker is implemented in the main branch, does this error still occur?


[feature suggestion] add placeholder for highest_on_track_<track>

I think it would be useful if placeholders for Prefix and Suffix Stacking were also available through PAPI etc.
My use case is this: VentureChat supports JSON hover / click events, and I would like to set up prefixes from individual tracks to have different hover elements. My current meta-formatting from LuckPerms looks like this:

`meta-formatting` from `config.yml`
meta-formatting:
  prefix:
    format:
      - "highest_on_track_staff"
      - "highest_on_track_vip"
      - "highest_on_track_patron"
      - "highest_on_track_discord-booster"
      - "highest_own"
    duplicates: none
    start-spacer: ""
    middle-spacer: ""
    end-spacer: "&r &7|&r "
  suffix:
    format:
      - "highest_on_track_staff"
      - "highest_on_track_vip"
      - "highest_on_track_patron"
      - "highest_on_track_discord-booster"
      - "highest_own"
    duplicates: none
    start-spacer: ""
    middle-spacer: " "
    end-spacer: ""

And my VentureChat config like this:

`config.yml` from VentureChat
jsonformatting:
  Default:
    json_attributes:
      luckperms_prefix:
        hover_text:
        - '&cI am default!'
        click_action: 'run_command'
        click_text: '/help'

# ...

channels:
  Global:
    # ...
    format: '{luckperms_prefix}{player_displayname}{luckperms_suffix}:'

I would like the ability to break out highest_on_track_<track> so I could do the meta formatting in the VentureChat config itself and add a section in jsonformatting to have different hover text for every track prefix. Then I could, for example, have something like:
jsonformatting:
  Default:
    json_attributes:
      luckperms_highest_on_track_vip:
        hover_text:
        - '&cVIP member! &eClick to learn how to buy this rank.'
        click_action: 'run_command'
        click_text: '/donate'

channels:
  Global:
    # ...
    format: '{luckperms_highest_on_track_vip}{player_displayname}{luckperms_suffix}:'

%luckperms_prefix_element_highest_on_track_staff%

exists and nobody told me???


Add installation instructions for MacPorts

I am the maintainer of the openjdk*-temurin ports in MacPorts. I recently contributed installation instructions, but I see the website has been replaced with a new version. The instructions can still be found on the old site here: https://old.adoptium.net/installation.html#macports
I looked at the code for the new site, but it seems there is currently a one-to-one mapping from operating system to package manager, which doesn't allow for easily adding a second package manager for an operating system. I'm afraid I don't understand React well enough to refactor the code to allow for instructions for two package managers for a single operating system, but it would be nice if the MacPorts instructions could be restored on the new website.

We use Asciidoctor to generate the documentation part, which generates static HTML.
As we ported the documentation from the old website we, or better I, found no easy way to replace names (like openjdk17-temurin) based on a selection.
You could:

add your instruction to the MacOS installation page. Similar to the yum and apt sections at the linux installation page
add your instruction as new macport.adoc page and link the page in the installation idex.html

If you have any problems, feel free to reach out here or via slack :-)


Datalogger csv simulator

Adds in datalogger
fix https://github.com/microsoft/pxt-microbit/issues/4167
close https://github.com/microsoft/pxt-microbit/issues/4492
uploadtrg with test link: https://makecode.microbit.org/app/57c0fa23ac50cdca72b24b113df6d4cf4111e670-be79ef3912#pub:_8Fy8aFbriLjJ
couple small things:

a few css issues I'm working on but just put up the pr, e.g. probably gonna flexbox the serial header bar for centering, make it prettier
the micro:bit side isn't sending time info as that cut muddled in with the serial mirroring, but will work on that over on micro:bit side; putting up for now as that's a separate issue.
(I was going to do a bit bigger change with refactoring the serial view but ended up not having time with other things coming up, so this is more in line with that existing code; next time >:( )
toggle change was pretty last minute, it should only show up if you're using both datalogger & serial in the same project. let me know if you see anything surprising there.
I made it so the csv table view only autoscroll if you can already see the bottom -- that is, if you scroll up it won't auto scroll anymore so that you can read old data. Maybe worth playing around with a little give there

@jaustin fyi

I've taken a look at the preview link, it's looking great
@jwunderl do you plan to change the language from "show console" to "show log" or something
My main outstanding concern having seen this is that I think we really need a way to hint to users how and where to find the real log file from their device. Because this still looks somewhat like serial view, I'm worried it leads people to expect the log file to be visible in MakeCode.
I think a single line of text on the simulated log or a new button underneath could help this.

@Jaqster I recall you had some thoughts about how/where to help users find the log file too?


.NET MAUI Blazor Hybrid tutorial updates

Fixes #25932
Addresses #24956
Internal Review Topic

Moves the Windows activity (section) before the Android activity (section).
I think it best to start the emulator from the Android Device Manager. After the phone is fully loaded, have the dev start the app in VS. I had a little trouble when we first put this topic up, where we had the start button in VS launch the emulator. If we start the emulator first and get it fully cook'in, VS has no trouble deploying and running the app.
This assumes that we'll merge this now to cover the Windows scenario and then hit the topic with a follow-up PR for VS4Mac later.

I'll merge now and go over the content again in the morning.
... and make API doc cross-links for all of the new API.
... and I should check the Security overview content to confirm it's right, especially for MAUI.


Serialize is defined multiple times

The version at crates.io has a bug (Serialize is defined multiple times).
Seems like it has already fixed, but don't pulled into crates.io. Please do it

Crate version: 0.4
OS: Linux

published a new release what should include that fix https://github.com/softprops/shiplift/blob/master/CHANGELOG.md#050


Illegal URI When Trying To Download Stubs From Maven Repo

I am moving an app to Finchley RC1 and when I run the contracts I am getting the following error
Caused by: java.lang.IllegalArgumentException: Illegal uri [https://oss.sonatype.org/content/repositories/snapshots/]
	at org.springframework.cloud.contract.stubrunner.ClasspathStubProvider$1.relativePathPicker(ClasspathStubProvider.java:139)
	at org.springframework.cloud.contract.stubrunner.ClasspathStubProvider$1.downloadAndUnpackStubJar(ClasspathStubProvider.java:89)
	at org.springframework.cloud.contract.stubrunner.CompositeStubDownloader.downloadAndUnpackStubJar(CompositeStubDownloaderBuilder.java:77)
	at org.springframework.cloud.contract.stubrunner.StubRunnerFactory.createStubsFromServiceConfiguration(StubRunnerFactory.java:60)
	at org.springframework.cloud.contract.stubrunner.BatchStubRunnerFactory.buildBatchStubRunner(BatchStubRunnerFactory.java:59)
	at org.springframework.cloud.contract.stubrunner.spring.StubRunnerConfiguration.batchStubRunner(StubRunnerConfiguration.java:78)
	at org.springframework.cloud.contract.stubrunner.spring.StubRunnerConfiguration$$EnhancerBySpringCGLIB$$ad2f449b.CGLIB$batchStubRunner$0(<generated>)
	at org.springframework.cloud.contract.stubrunner.spring.StubRunnerConfiguration$$EnhancerBySpringCGLIB$$ad2f449b$$FastClassBySpringCGLIB$$14d4050e.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
	at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:361)
	at org.springframework.cloud.contract.stubrunner.spring.StubRunnerConfiguration$$EnhancerBySpringCGLIB$$ad2f449b.batchStubRunner(<generated>)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
	... 74 more

Here is my configuration
stubrunner:
  idsToServiceIds:
    ocr-participants: participants
  ids: 'com.ryanjbaxter.spring.cloud:ocr-participants:+:stubs'
  repositoryRoot: https://oss.sonatype.org/content/repositories/snapshots/

Stubs are located here https://oss.sonatype.org/content/repositories/snapshots/com/ryanjbaxter/spring/cloud/ocr-participants/
It seems weird to me that ClasspathStubProvider is running since the stubs are not on the classpath but in a Maven repo.

You haven't provided the stubsMode which by default points to class path.
One doc problem, this should be stubs-mode not ``stubsMode`
http://cloud.spring.io/spring-cloud-static/Finchley.RC1/single/spring-cloud.html#_spring_cloud_cli
Another question related to this....
So when I specify REMOTE and the stubs happen to be installed locally as well the tests fail
Caused by: java.lang.IllegalStateException: The artifact was found in the local repository but you have explicitly stated that it should be downloaded from a remote one
Why not just ignore the ones locally and download them?  Why throw the exception?

As discussed offline, here (https://cloud.spring.io/spring-cloud-contract/single/spring-cloud-contract.html#_ci_server_setup) we explain how to workaround this issue and the reasons for that issue.
After doing another digging into the problem, obvioulsy we're not the only ones with this issue (http://maven.40175.n5.nabble.com/Resolving-an-Artifact-from-Remote-Repository-instead-of-Local-td5875134.html). A potential workaround is to, when someone sets stubsMode: REMOTE, to provide a fake maven local repo location - https://stackoverflow.com/questions/9123004/maven-is-it-possible-to-override-location-of-local-repository-via-the-use-of-co . I'll create a separate issue for that.


Crate does no longer work with no-std on 0.7

It fails with:
error[E0433]: failed to resolve: could not find `std` in the list of imported crates
  --> src\sanitizers\deduplicator.rs:74:9
   |
74 |         assert_eq!(iter.next(), Some('1'));
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ could not find `std` in the list of imported crates
   |
   = note: this error originates in the macro `assert_eq` (in Nightly builds, run with -Z macro-backtrace for more info)

(feature gating every single test with the std feature would be really annoying, especially when the std feature is disabled by default)
The macro expands in 0.7 to something like this:
{
    match (&(sanitizer.next()), &(None)) {
        (left_val, right_val) => {
            if !(*left_val == *right_val) {
                {
                    ::std::rt::begin_panic_fmt(&match match (
                        &::pretty_assertions::Comparison::new(left_val, right_val),
                    ) {
                        (arg0,) => [::core::fmt::ArgumentV1::new(
                            arg0,
                            ::core::fmt::Display::fmt,
                        )],
                    } {
                        ref args => unsafe {
                            ::core::fmt::Arguments::new_v1(
                                &["assertion failed: `(left == right)`\n\n", "\n"],
                                args,
                            )
                        },
                    })
                }
            }
        }
    }
};

Previously (0.6), it expanded to
{
    match (&(sanitizer.next()), &(None)) {
        (left_val, right_val) => {
            if !(*left_val == *right_val) {
                ::core::panicking::panic_fmt(
                    match match (&::pretty_assertions::Comparison::new(
                        left_val, right_val,
                    ),)
                    {
                        (arg0,) => [::core::fmt::ArgumentV1::new(
                            arg0,
                            ::core::fmt::Display::fmt,
                        )],
                    } {
                        ref args => unsafe {
                            ::core::fmt::Arguments::new_v1(
                                &["assertion failed: `(left == right)`\n\n", "\n"],
                                args,
                            )
                        },
                    },
                )
            }
        }
    }
};

Related issue: KokaKiwi/rust-hex#63

Hey, thanks for the report. I've put together a PR which I think should provide no_std support, as long as you still have an allocator: https://github.com/colin-kiegel/rust-pretty-assertions/pull/83
I don't have masses of experience with no_std support - if you could verify this fix works for your use case, I'll be able to merge and release it faster. You can try out the patch with something like:
pretty_assertions = { git="https://github.com/tommilligan/rust-pretty-assertions", rev="9ba9e84d930493f0218615919ec3868c6d36f47b" }

Thank you, this does work as expected

Closed as released in v1.0.0


Fixed typo

removed an unnecessary [

@YuliKl : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@jborsecnik : Thanks for your contribution! The author(s) have been notified to review your proposed change.

Confirmed correction (and found and fixed another like it). Thanks.


ECONNRESET on read is not a transient network error

In DoRetryWithStatusCodes, we retry for a list of status codes and also retry on temporary network errors. However, I am seeing transient network errors not being retried.
E.g.
compute.DisksClient#List: Failure sending request: StatusCode=0 -- Original Error: Get https://management.azure.com/subscriptions/13979c32-3382-46f7-8f3a-ef86b8270227/providers/Microsoft.Compute/disks?api-version=2018-09-30: read tcp <IP_ADDRESS>:56094-u003e<IP_ADDRESS>:443: read: connection reset by peer
It looks to me that "connection reset by peer" error is not treated as temporary by IsTemporaryNetworkError. Is there anyway I can enable retry on it for all autorest call (other than doing it in my application code itself)?

I dug into this, the behavior is by design; i.e. the standard library doesn't treat ECONNRESET as a transient network error, see test TestNotTemporaryRead in net_test.go.  Do you know why the connection is being reset and does retrying resolve the issue?  I'm hesitant to treat this case as transient if the standard library doesn't (presumably they have a good reason).

@jhendrixMSFT

Do you know why the connection is being reset and does retrying resolve the issue?

FWIW we also see this through our test suite most days across a random selection of tests, presumably because of networking issues between AWS (where our CI runs) and Azure (where the ARM API's are)

Thanks for looking. I am not sure why connection is being reset by Azure as there is no other information in the error logs. Retries fixes this issue, I suspect it might have to do with the load on the server side leading to connection being reset or transient network issue between application and Azure servers.

azure-storage-node API also seem to treat ECONNRESET as retryable error: https://github.com/Azure/azure-storage-node/blob/master/ChangeLog.md

ECONNRESET is handled in the retry policy filter.

Fixed an issue which prevented transient server errors like ECONNRESET, ESOCKETTIMEDOUT and ETIMEDOUT from being retried.

Thanks for the info, I will update the retry logic to include these conditions.

I have a tentative fix here https://github.com/Azure/go-autorest/issues/450.  Would one of you be able to try it out?  I don't have a repro of this issue so had to simulate it via the test in net_test.go cited above.

I just hit this and tried to dump autorest.DetailedError.Original:
*url.Error: Delete [secure]goofys-test-dkwipl51mu9uwhqb/test_dir%2F1%2F7%2F5: read tcp <IP_ADDRESS>:49214-><IP_ADDRESS>:443: read: connection reset by peer

I wonder if it's sufficient to amend the check for net.Error with this since url.Error has https://golang.org/pkg/net/url/#Error.Temporary

We do this already in internal.IsTemporaryNetworkError(), the problem is that the standard lib doesn't treat ECONNRESET as a temporary error (see above discussion).

@jhendrixMSFT I tried to reproduce it in a test environment, but haven't been able to reproduce it. I have only seen it in production, so unable to verify the fix.

@jhendrixMSFT I'm going to try and run our test suite with this patch at some point this week, which should confirm this one way or the other :)

@mgaurav no worries I appreciate the help.
@tombuildsstuff thanks for jumping in :)

@jhendrixMSFT When do you plan to release this change? I would like to incorporate it in our production.

@mgaurav still iterating on the fix, see the PR for the latest comments.

Fixed in https://github.com/Azure/go-autorest/releases/tag/v13.0.2


Write irreversible block delay in mongodb for 24 minutes

We initiated a transaction to query the transaction in hisory_plugin and will soon be able to get the transaction irreversible, but after checking the transaction in mongodb for 24 minutes, why is this?

This happens only in multiple nodes, everything is normal for a single node.

What's your hardware config?

What are your config.ini options?

it is nothing of confiig.because before this error happened,tx is work normally.

It may still be the config, despite it not changing.  Mongodb behavior may change as the amount of stored data becomes very large.  If the problem has been solved, please close this issue.

Improvements have been made to mongodb support when mongodb is configured to run in a cluster.  The modifications are available in v1.6.0-rc1.

It is possible you are seeing this: https://github.com/EOSIO/eos/issues/6409#issuecomment-442866447

No answer from the reporter, closing.


Saving to file a model within TPUStrategy

On Tensorflow 2.0 and 2.1, trying to save to file a TPU trained model or a model that was even created within the scope of a tf.distribute.experimental.TPUStrategy yields error below. Despite throwing an UnimplementedException the code does create a folder on disk with some content.
The reproducible code can be found in a collab notebook here:
https://colab.research.google.com/drive/1DOkwNlzMLsg0wQZe41eq88XhEUqR9Amn
The exception and it's stack
---------------------------------------------------------------------------
UnimplementedError                        Traceback (most recent call last)
<ipython-input-7-b3a563716de1> in <module>()
     48         print("1.4 ",err)
     49
---> 50     model.save("model-new.1")
     51
     52     model.compile(

14 frames
/tensorflow-2.1.0/python3.6/tensorflow_core/python/keras/engine/network.py in save(self, filepath, overwrite, include_optimizer, save_format, signatures, options)
   1006     """
   1007     save.save_model(self, filepath, overwrite, include_optimizer, save_format,
-> 1008                     signatures, options)
   1009
   1010   def save_weights(self, filepath, overwrite=True, save_format=None):

/tensorflow-2.1.0/python3.6/tensorflow_core/python/keras/saving/save.py in save_model(model, filepath, overwrite, include_optimizer, save_format, signatures, options)
    113   else:
    114     saved_model_save.save(model, filepath, overwrite, include_optimizer,
--> 115                           signatures, options)
    116
    117

/tensorflow-2.1.0/python3.6/tensorflow_core/python/keras/saving/saved_model/save.py in save(model, filepath, overwrite, include_optimizer, signatures, options)
     76     # we use the default replica context here.
     77     with distribution_strategy_context._get_default_replica_context():  # pylint: disable=protected-access
---> 78       save_lib.save(model, filepath, signatures, options)
     79
     80   if not include_optimizer:

/tensorflow-2.1.0/python3.6/tensorflow_core/python/saved_model/save.py in save(obj, export_dir, signatures, options)
    914   # SavedModel proto itself.
    915   utils_impl.get_or_create_variables_dir(export_dir)
--> 916   object_saver.save(utils_impl.get_variables_path(export_dir))
    917   builder_impl.copy_assets_to_destination_dir(asset_info.asset_filename_map,
    918                                               export_dir)

/tensorflow-2.1.0/python3.6/tensorflow_core/python/training/tracking/util.py in save(self, file_prefix, checkpoint_number, session)
   1166     file_io.recursive_create_dir(os.path.dirname(file_prefix))
   1167     save_path, new_feed_additions = self._save_cached_when_graph_building(
-> 1168         file_prefix=file_prefix_tensor, object_graph_tensor=object_graph_tensor)
   1169     if new_feed_additions:
   1170       feed_dict.update(new_feed_additions)

/tensorflow-2.1.0/python3.6/tensorflow_core/python/training/tracking/util.py in _save_cached_when_graph_building(self, file_prefix, object_graph_tensor)
   1114         or context.executing_eagerly() or ops.inside_function()):
   1115       saver = functional_saver.MultiDeviceSaver(named_saveable_objects)
-> 1116       save_op = saver.save(file_prefix)
   1117       with ops.device("/cpu:0"):
   1118         with ops.control_dependencies([save_op]):

/tensorflow-2.1.0/python3.6/tensorflow_core/python/training/saving/functional_saver.py in save(self, file_prefix)
    228         # _SingleDeviceSaver will use the CPU device when necessary, but initial
    229         # read operations should be placed on the SaveableObject's device.
--> 230         sharded_saves.append(saver.save(shard_prefix))
    231
    232     with ops.control_dependencies(sharded_saves):

/tensorflow-2.1.0/python3.6/tensorflow_core/python/training/saving/functional_saver.py in save(self, file_prefix)
     67       for spec in saveable.specs:
     68         tensor_names.append(spec.name)
---> 69         tensors.append(spec.tensor)
     70         tensor_slices.append(spec.slice_spec)
     71     with ops.device("cpu:0"):

/tensorflow-2.1.0/python3.6/tensorflow_core/python/training/saving/saveable_object.py in tensor(self)
     50   @property
     51   def tensor(self):
---> 52     return self._tensor() if callable(self._tensor) else self._tensor
     53
     54

/tensorflow-2.1.0/python3.6/tensorflow_core/python/training/saving/saveable_object_util.py in f()
     89         def f():
     90           with ops.device(v.device):
---> 91             x = v.read_value()
     92             # To allow variables placed on non-CPU devices to be checkpointed,
     93             # we copy them to CPU on the same machine first.

/tensorflow-2.1.0/python3.6/tensorflow_core/python/ops/resource_variable_ops.py in read_value(self)
    633     """
    634     with ops.name_scope("Read"):
--> 635       value = self._read_variable_op()
    636     # Return an identity so it can get placed on whatever device the context
    637     # specifies instead of the device where the variable is.

/tensorflow-2.1.0/python3.6/tensorflow_core/python/ops/resource_variable_ops.py in _read_variable_op(self)
    611     variable_accessed(self)
    612     result = gen_resource_variable_ops.read_variable_op(self._handle,
--> 613                                                         self._dtype)
    614     _maybe_set_handle_data(self._dtype, self._handle, result)
    615

/tensorflow-2.1.0/python3.6/tensorflow_core/python/ops/gen_resource_variable_ops.py in read_variable_op(resource, dtype, name)
    477         pass  # Add nodes to the TensorFlow graph.
    478     except _core._NotOkStatusException as e:
--> 479       _ops.raise_from_not_ok_status(e, name)
    480   # Add nodes to the TensorFlow graph.
    481   dtype = _execute.make_type(dtype, "dtype")

/tensorflow-2.1.0/python3.6/tensorflow_core/python/framework/ops.py in raise_from_not_ok_status(e, name)
   6604   message = e.message + (" name: " + name if name is not None else "")
   6605   # pylint: disable=protected-access
-> 6606   six.raise_from(core._status_to_exception(e.code, message), None)
   6607   # pylint: enable=protected-access
   6608

/usr/local/lib/python3.6/dist-packages/six.py in raise_from(value, from_value)

UnimplementedError: File system scheme '[local]' not implemented (file: 'model-new.1/variables/variables_temp_17ffcf98334348fd8ef1e339869f0bfc')
	Encountered when executing an operation using EagerExecutor. This error cancels all future operations and poisons their output tensors. [Op:ReadVariableOp]

System information
The error is reproduced in Collab so I'm skipping tf_env_collect.sh output.

You can/should use  a cloud storage bucket path. See https://cloud.google.com/tpu/docs/troubleshooting#cannot_use_local_filesystem

@vicpara Please let us know if the above comment helps resolve issue.

Yes it does. Thanks.

Hi, I'm using a TPU on Kaggle, and I have no write permission to the GS cloud storage bucket, only the local filesystem. Is there a way around that, so I can save to the local filesystem?
Thanks.

@rafwaf You can save the model with legacy h5 format to local storage (tested on Colab, TF2.7).
Use model.save("/path/to/model.h5") instead of model.save("/path/to/savedModel").
https://www.tensorflow.org/guide/keras/save_and_serialize#whole-model_saving_loading
With TPU backend, you will need a GS bucket to save the model in SavedModel format.

I think we try something like this,
# make model
with strategy.scope():
  model = make_model()

# save model locally from tpu using Tensorflow's "SavedModel" format
save_locally = tf.saved_model.SaveOptions(experimental_io_device='/job:localhost')
model.save('./model', options=save_locally)

# load model in tpu using Tensorflow's "SavedModel" format
with strategy.scope():
    load_locally = tf.saved_model.LoadOptions(experimental_io_device='/job:localhost')
    model = tf.keras.models.load_model('./model', options=load_locally)

i used this   *model_save_kwargs(dict, optional)
save_locally = tf.saved_model.SaveOptions(experimental_io_device='/job:localhost')
push_to_hub_keras(model, "demo",options=save_locally)


doris表新增字段后一直不显示新增的字段.

Search before asking

[X] I had searched in the issues and found no similar issues.

Version
5.1.0
What's Wrong?
doris表新增字段后一直不显示新增的字段.

What You Expected?
可以新增字段。现在新增字段如下：
1064 - errCode = 2, detailMessage = Table[ad_emi_game_data]'s state is not NORMAL. Do not allow doing ALTER ops
How to Reproduce?
No response
Anything Else?
No response
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Alter table is a async operation.


2-4-stable

Any chance of a 2-4-stable branch?

Hi!
I created missing branches, they should work without changes but i didn't test them :)

Thank you very much thats really helpful


feat: configure dashpay for test suite

Issue being fixed or feature implemented

What was done?

How Has This Been Tested?

Breaking Changes

Checklist:

[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have added or updated relevant unit/integration/functional/e2e tests
[ ] I have made corresponding changes to the documentation

For repository code-owners and collaborators only

[ ] I have assigned this pull request to a milestone

This logic already implemented in https://github.com/dashevo/platform


Incorrect first call to renderExternal when using virtualization and initialSlide is non-zero

This is a (multiple allowed):

[x] bug

[ ] enhancement

[ ] feature-discussion (RFC)

Swiper Version: 4.3.5

Platform/Target and Browser Versions: Chrome (69.0.3497.100), macOS High Sierra

What you did
Used Swiper's initialSlide and virtual/renderExternal features together.
Expected Behavior
The data passed to the first call to renderExternal after initialization should reflect initialSlide.
Actual Behavior
The renderExternal function is called once with offset, from, to all set to zero. Then it is called again with the correct offset, from, to etc.
When using virtualization with frameworks like React, there is often a state update in renderExternal. This then causes an extra render with incorrect slides and positions.
P.S. Remember, an issue is not the place to ask questions. You can use Stack Overflow
for that.
Before you open an issue, please check if a similar issue already exists or has been closed before.

I'll also note that it seems like the state internally is being tracked correctly in the sense that the init callback is not called until the correct transition has been set.

Swiper loads and calls renderExternal with the first two slides and the position at 0
Swiper transitions to the index specified in initialSlide. This calls transitionEnd. Realize that this bug/design-flaw also means that transition callbacks are called before initialization. So if you've set runCallbacksOnInit to be false, those callbacks will all be run before init is called. This is bad.
Swiper calls init.

Awesome! Thank you for the fix 🙏🏼


Cotary Create: Update speed buttons

This PR for issue: #406

Ready to merge!
Worth noting that disabling auto-betting against proposals is broken on both dev and this branch so NOT an issue with this pr.


Show error, when package failed to load

Currently if you fail to load a swift package the extension fails quietly. Need more informative messaging.

#272 shows message, but needs to be more informative


Styles with capital "I" in !Important disappear

If I use !Important with capital "I" at the beginning, clean-css just removes all the property.
Fore example
.some-class{
  position: fixed!Important;
}

This selector will just disappear from final file.

Released in 3.4.11, thanks @antonkhrit for the fix!


Release @google-cloud/vision v0.24.0

This pull request was generated using releasetool.

Release is at https://github.com/googleapis/nodejs-vision/releases/tag/v0.24.0

The release build has started, the log can be viewed here. :sunflower:

:egg: You hatched a release! The release build finished successfully! :purple_heart:


tracing: add gpio tracing capabilities and sample

If GPIO tracing is enabled, then the system will track input, output, interrupt attachment/removal and interrupt execution for GPIO pins.

The warnings about braces being on a newline are false positives (it's following the existing convention in the tracing_user.c).

Action shot (heavily customized to use Chrome Trace JSON output) where the trace subsys was used to trace bit-banging JTAG + AXI.

@alexanderlayTT - can you fix the compliance checks and rebase on main?

Yes! We had some other work come up, but we are making progress on this too.

@alexanderlayTT - looks like there is at least one unique build failure.
https://github.com/zephyrproject-rtos/zephyr/actions/runs/11803255472/job/32880806857#step:11:1
What I usually do is copy paste the first build / runtime failure in the logs

and then edit it to run with twister to repro.
twister -i -p native_sim/native -s samples/basic/blinky/sample.basic.blinky

Repeat as necessary 😄
Of course, if you scroll to the end of the buffer, there is also a repro command that you can copy & paste.

@alexanderlayTT - there is still a doc build issue
WARNING: /home/runner/work/zephyr/zephyr/include/zephyr/tracing/tracing.h:2595: unable to resolve reference to 'list' for \ref command
WARNING: /home/runner/work/zephyr/zephyr/include/zephyr/tracing/tracing.h:2595: unable to resolve reference to 'list' for \ref command

You should be able to reproduce with this
https://docs.zephyrproject.org/latest/contribute/documentation/generation.html

Looks good 👍 Thanks for the attention to detail @alexanderlayTT

ping @bjarki-andreasen for a re-review


Overview page

This adds support for the Snapshot (landing) page, and adds a few more fixes/tweaks to the cluster and ring pages. The snapshot should detail unreachable (partitioned) nodes, 'down' nodes, and nodes running low on memory.

Pulled, devrel'd, started, it says my cluster has problems, the cluster and ring page look fine to me (all green.) Wondering what my cluster's problem is.
Also, and this is maybe a different discussion, but given that a large system is always in some degree of failure, do we really want this big red 'X' in the overview, I mean Riak works well whilst nodes a failing, and that is sort of the point of dynamo systems. I agree an overall health view is warranted, but if you can serve reads/writes to some degree then a big red 'X' is just needlessly scary. I hate to sound like one of those "Can it be more blue, please" product manager guys I used to hate, but, how about amber or something unless the cluster is actually down?

Ah, OK, from going to https://<IP_ADDRESS>:8443/admin/overview is seems the problem is all my nodes are low on memory {"unreachable_nodes":[],"down_nodes":[],"low_mem_nodes":["dev1@<IP_ADDRESS>","dev2@<IP_ADDRESS>","dev3<EMAIL_ADDRESS>This is not shown on the overview page.

Pulled the changes, ran it, all looks good now, +1 to merge


Infinite loop sending output to an output channel from an extension makes VS Code stop responding

Previous statement was wrong:
Constantly sending output to an output channel (as in an infinite loop printing data) from an extension makes VS Code stop responding.
This happened both with the stable and alpha release.

@alexandrudima I would believe this is a general issue and not so much about output. Any extension can trigger API in a for loop and that will likely spam the renderer enough to make it unresponsive. Do we do some throttling with regards to data traveling from a spammy extension back to the main side?

If an extension enters an infinite loop, AFAIK we have no graceful way to terminate it besides killing the extension host process.

I will investigate if buffering the output service is something that helps here but it would not help in all cases of abuse.

@digeff I am not able to reproduce with a simple example, is it possible to share the extension with me to reproduce?

@bpasero
Sorry for the delay, I meant something like this (This makes my VS Code unresponsive consistently):
`export function activate(context: vscode.ExtensionContext) {
let disposable = vscode.commands.registerCommand('extension.sayHello', () => {
vscode.window.showInformationMessage('Crashing!');
lotsOfOutput();
});
context.subscriptions.push(disposable);

}
function lotsOfOutput() {
setTimeout(() => {
const outputChannel = vscode.window.createOutputChannel("Test");
outputChannel.show();
outputChannel.appendLine("Stop working please...");
lotsOfOutput();
lotsOfOutput();
}, 100);
}
`

@isidorn can you try this out? asking because you recently pushed a change to handle output updates with a 300ms delay directly in the UI layer and maybe we are good to go already.

So the example that the user provided still can crash vscode because it is a quadratic function which creates new output, shows the output channel and appends the output. However the following can no longer crash / slow down vscode
const outputChannel = vscode.window.createOutputChannel("Test");
setInterval(() => {
    outputChannel.appendLine("Stop working please...");
}, 10);

Sorry, have to reopen, would like to understand why we still crash and if we can prevent that.

@isidorn What if you set the interval time to 0?

Ok, the cause of the crash was the multiple calls to show(), now I protect against that.
Please note that the original sample provided by @digeff still crashes after some time since it converges to infinity.
@digeff regarding interval 0 https://stackoverflow.com/questions/7648557/setinterval-behaviour-with-0-milliseconds-in-javascript

@isidorn What if we use process.nextTick instead of interval 0? (Basically, I want to simulate adding output as fast as possible, while not yet converging to infinity).

@digeff if I setInterval for 1ms it does not crash any more

:heart: