Analyze Histogram Transformation performance for outliers

After testing with NYC Dataset, we've noticed that indexing outliers create a disbalance in the OTree structure. Those outliers can be critical in skewed, "real" datasets, degrading the performance by scanning more files in a filtered search.
Right now, we are only normalizing the data using a linear transformation, which is less flexible and strict when some points outside the minimum and maximum values are inserted into the table. For that reason, we want to understand if using Histograms can help overcome this problem.
The purpose of the issue is to track and document a first trial and determine whether the performance improvement is significant enough to introduce it as a new feature.
TODO:

Load NYC Taxi Dataset.
Create a CDF for each column to index. (In this case, it's going to be Pickup and DropOff location and timestamps)
Save the new DataFrame.
Index the same data twice: one with the original indexed columns, and the second using the values from the CDF.
Measure the performance of range queries.
Analyse the Index Metrics and Min-Max ranges of the files to understand the improvement.

Here's the summary of the initial experiment results.
WITH CDF (histogram scenario):

9 files
avg size 296.8MB.
More cubes, but fractioned in less files.

OTree Index Metrics:
dimensionCount: 4
elementCount: 77966321
depth: 3
cubeCount: 145
desiredCubeSize: 3248596
indexingColumns: DOLocationID_cdf,PULocationID_cdf,tpep_pickup_datetime_cdf,tpep_dropoff_datetime_cdf
avgFanout: 16.0
depthOnBalance: 1.6931305699633146

Stats on cube sizes:
Quartiles:
- min: 3247255
- 1stQ: 3248326
- 2ndQ: 3249473
- 3rdQ: 3249924
- max: 3250627
Stats:
- count: 9
- l1_dev: 3.389840479463196E-4
- l2_dev: 1.2293769288666004E-4
Level-wise stats:
level avgCubeSize stdCubeSize cubeCount avgWeight
    0     3249473           0         1   0.04166
    1     3249074        1135         8    0.3757

WITHOUT CDF
- 15 files
- Avg File Size 180,42MB.
- Less cubes, more defragmented.

OTree Index Metrics:
dimensionCount: 4
elementCount: 77966321
depth: 5
cubeCount: 79
desiredCubeSize: 3248596
indexingColumns: DOLocationID,PULocationID,tpep_pickup_datetime,tpep_dropoff_datetime
avgFanout: 5.2
depthOnBalance: 3.2196329121951317

Stats on cube sizes:
Quartiles:
- min: 3247408
- 1stQ: 3248302
- 2ndQ: 3249966
- 3rdQ: 3260554
- max: 3344342
Stats:
- count: 15
- l1_dev: 0.004943222651672702
- l2_dev: 0.00278163396661163
Level-wise stats:
level avgCubeSize stdCubeSize cubeCount avgWeight
    0     3249966           0         1   0.04166
    1     3249943        1184         4    0.2768
    2     3268407       32146         7    0.6381
    3     3278608       43492         3   0.90334

Min-Max File ranges
I've analyzed the file ranges for the column tpep_pickup_datetime, which is the one that presented more outliers in the dataset. We can see that the min max of the files if more stable when using the CDF columns to index. (See image below)

On the queries side, I don't see much difference. Perhaps because is a small dataset (1.3GB in total). Nevertheless, I will test with fewer resources to notice the computational improvements.

First analysis of the following queries:
// QUERY 1
qbeastTable
  .filter(col("PULocationID") >= 180 && col("PULocationID") <= 220 && col(
    "tpep_pickup_datetime") ><PHONE_NUMBER> && col("tpep_pickup_datetime") <=<PHONE_NUMBER>)
  .count

// QUERY 2
qbeastTable
  .filter(col("PULocationID") >= 180 && col("PULocationID") <= 220 && col(
    "tpep_pickup_datetime") ><PHONE_NUMBER> && col("tpep_pickup_datetime") <=<PHONE_NUMBER>)
  .sort("PULocationID")
  .head(20)

With CDF

QUERY 1: Time taken: 2912 ms
QUERY 2: Time taken: 9181 ms

NO CDF

QUERY 1: Time taken: 3262 ms
QUERY 2: Time taken: 11493 ms

Next steps:

Compare the metrics with the dataset indexed without outliers to check the similarity with the CDF approach.
Evaluate how much time is spent on the computation of the histogram columns. Since the methods to compute the statistics of the table can be expensive, we would need to understand if the tradeoff is worth it to keep developing in that direction.
Test performance with bigger skewed datasets.

IndexMetrics comparison
IndexMetrics with outliers + CDF
OTree Index Metrics:
dimensionCount: 4
elementCount: 77966321
depth: 3
cubeCount: 145
desiredCubeSize: 3248596
indexingColumns: DOLocationID_cdf,PULocationID_cdf,tpep_pickup_datetime_cdf,tpep_dropoff_datetime_cdf
avgFanout: 16.0
depthOnBalance: 1.6931305699633146

Stats on cube sizes:
Quartiles:
- min: 3247255
- 1stQ: 3248326
- 2ndQ: 3249473
- 3rdQ: 3249924
- max: 3250627
Stats:
- count: 9
- l1_dev: 3.389840479463196E-4
- l2_dev: 1.2293769288666004E-4
Level-wise stats:
level avgCubeSize stdCubeSize cubeCount avgWeight
    0     3249473           0         1   0.04166
    1     3249074        1135         8    0.3757

IndexMetrics without outliers
OTree Index Metrics:
dimensionCount: 4
elementCount: 77965705
depth: 3
cubeCount: 74
desiredCubeSize: 3248596
indexingColumns: tpep_pickup_datetime,tpep_dropoff_datetime,PULocationID,DOLocationID
avgFanout: 8.11111111111111
depthOnBalance: 1.9615397304356053

Stats on cube sizes:
Quartiles:
- min: 3246724
- 1stQ: 3248079
- 2ndQ: 3248494
- 3rdQ: 3249954
- max: 3250816
Stats:
- count: 9
- l1_dev: 3.079963022658267E-4
- l2_dev: 1.2662589262706753E-4
Level-wise stats:
level avgCubeSize stdCubeSize cubeCount avgWeight
    0     3250816           0         1   0.04166
    1     3248510        1044         8   0.51244

We can conclude that both indexes look pretty similar, which is a very good outcome: outliers no longer impact the index distribution, making values more evenly spread within childs.
Write Performance
How to write with new CDF columns
To be able to execute the CDF with more than one column, we had to split the dataFrame and process it with a smaller group of columns:
def computeCDFColumn( df: DataFrame, column: String): DataFrame = {
  df.withColumn(s"${column}_cdf", cume_dist().over(Window.orderBy(column)))
}

def computeCDFColumns(df: DataFrame, columns: Seq[String]): DataFrame = {
  columns.foldLeft(df) { (acc, column) =>
    computeCDFColumn(acc, column)
  }
}

val startingTime = System.currentTimeMillis()
// Load
val data = spark.read.format("parquet").load("nyc-yellow-cab-data-norm")
// Compute CDF
val columnsToIndex = Seq("DOLocationID", "PULocationID", "tpep_pickup_datetime", "tpep_dropoff_datetime")
val dataCDF = computeCDFColumns(data.selectExpr(columnsToIndex:_*), columnsToIndex)
// Join with source
val newData = data.join(dataCDF, columnsToIndex)
// Save
newData.write.format("parquet").mode("overwrite").save(s"nyc-yellow-cab-data-norm-with-cdf$startingTime")
println(s"Time to write CDF columns: ${System.currentTimeMillis() - startingTime} ms")

We will compare times within this execution.
What is the overhead of computing the CDF over a column?
170389 ms vs 83130 ms. It is 2x slower to generate the dataset with a single CDF column.
The execution does a sort + window operation in a single partition, which results in heavy computation for even a small dataset.
What is the overhead of computing the CDF over multiple columns?
429183 ms vs 83130 ms. It is 5x slower to generate the dataset with 4 CDF columns.
Can the performance be improved?
This overhead would only be noticed during the first write. Also, in this experiment we are not taking advantage of distributed computing since we load everything into one partition.

Done and opened an issue for the feature in #338


Support Color Fidelity

Can the builder support Color Fidelity as described on Google IO (https://io.google/2023/program/630ffaa3-9b3d-4e30-a89f-91ef3e8052cf/)?

Fidelity is planned and coming soon! (Going to close this ticket for now, but we'll make an announcement!)


Rename master to main

In theory should not have too many major side effects since we don't yet have any actions triggered by certain branches but opdevbot may not like it ..
One set of instructions are here:
https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-branches-in-your-repository/renaming-a-branch

@skattoju is there any impact to commit history? Can you confirm that. I couldn't see that from the docs. It only mentions urls I may have skipped it.

I don't believe there is any impact to commit history tried it locally
 skattoju@skattoju-mac  ~/code/opcap   testing  git log --graph > /tmp/before.txt
 skattoju@skattoju-mac  ~/code/opcap   testing  git branch -m test
 skattoju@skattoju-mac  ~/code/opcap   test  git log --graph > /tmp/after.txt
 skattoju@skattoju-mac  ~/code/opcap   test  diff /tmp/before.txt /tmp/after.txt

Generally, I would be apprehensive of making this change. Since there are a number of forks to the repo. Once the change is made on the upstream repository, there is likely going to be a need for all the forks to be recreated.


Unable to create DB using MongoDB Compass

FerretDB version
version: v1.1.0
commit: 3344cbb98bb744dd044bcf2d51fe9ab65db22f0b
branch: unknown
dirty: true
package: deb
debugBuild: false
Backend
psql (PostgreSQL) 14.7 (Ubuntu 14.7-0ubuntu<IP_ADDRESS>)
Environment

OS: elementary OS 7
Deployment: DEB
MongoDB Compass 1.36.4 (flatpak)

What did you do?

Prepared PostgreSQL and FerretDB for use. Stopped mongod to avoid conflicts.

sudo apt install postgresql
sudo systemctl enable postgresql
sudo -u postgres createuser platon
sudo -u postgres createdb platon

sudo systemctl stop mongod

sudo apt install /path/to/ferretdb.deb
ferretdb

Connected via Compass without any errors.

Then I tried to create DB and table. An error appeared.

What did you expect to see?
Since FerretDB works with Studio 3T, I expect it to work with Compass as well.
What did you see instead?
Full error code printed by ferretdb running in the terminal:
2023-05-12T15:14:31.836+0300	WARN	pg/pg.go:137	DBPool: authentication failed	{"username": "", "error": "pgdb.checkConnection: failed to connect to `host=<IP_ADDRESS> user=platon database=ferretdb`: failed SASL auth (FATAL: password authentication failed for user \"platon\" (SQLSTATE 28P01))"}
2023-05-12T15:14:31.836+0300	WARN	// <IP_ADDRESS>:52620 -> <IP_ADDRESS>:27017 	clientconn/conn.go:569	Response header: length:   347, id:    4, response_to:   23, opcode: OP_MSG
2023-05-12T15:14:31.836+0300	WARN	// <IP_ADDRESS>:52620 -> <IP_ADDRESS>:27017 	clientconn/conn.go:570	Response message:
{
  "Checksum": 0,
  "FlagBits": 0,
  "Sections": [
    {
      "Document": {
        "$k": [
          "ok",
          "errmsg",
          "code",
          "codeName"
        ],
        "ok": {
          "$f": 0
        },
        "errmsg": "[msg_create.go:37 pg.(*Handler).MsgCreate] [pg.go:138 pg.(*Handler).DBPool] pgdb.checkConnection: failed to connect to `host=<IP_ADDRESS> user=platon database=ferretdb`: failed SASL auth (FATAL: password authentication failed for user \"platon\" (SQLSTATE 28P01))",
        "code": 1,
        "codeName": "InternalError"
      },
      "Kind": 0
    }
  ]
}

What credentials did you use in PostgreSQL and MongoDB URIs? See here.

I used the URI from a screenshot I found on the FerretDB blog. The docs section you referred to is difficult to understand.

Well, then, I guess your PostgreSQL installation requires credentials that are not provided. So it is not about MongoDB Compass at all.
See https://ubuntu.com/server/docs/databases-postgresql. The simplest solution for local testing could be enabling trust for all localhost connections; see https://www.postgresql.org/docs/current/auth-trust.html.
Or just use our all-in-one Docker image: https://github.com/FerretDB/FerretDB#quickstart


Consolidate static analysis checks from microbench

When microbench was introduced in #15518, it required new suppressions for the checkstyle plugin and the spotbugs plugin. This PR consolidates the new suppressions in the existing top level files.
This makes suppressions more understandable as they are all found in one single file, and also helps IDE extensions function properly (as there is only one file they must look at).

alluxio-bot, merge this please


B 21326 tget data cleanup int

[!WARNING]
This ticket should not be merged to main until https://github.com/transcom/trdm-lambda/pull/28/ has been released to production

Ticket B-21326
Summary
Fixes existing trailing spaces and empty values
How to test

Insert bad dummy data

INSERT INTO transportation_accounting_codes (
    id, tac, tac_sys_id, loa_sys_id, tac_fy_txt, tac_fnct_poc_nm, created_at, updated_at
)
VALUES
    ('E08FF3DB-F17D-4076-A9E6-AAB6D3F4029C', 'ABCD', 'dummy', '   dummy', 'dummy ', '  ',now(), now());

INSERT INTO lines_of_accounting (
    id, loa_sys_id, loa_dsc_tx, loa_fnct_prs_nm, loa_agnc_acntng_cd, created_at, updated_at
)
VALUES
    ('E08FF3DB-F17D-4076-A9E6-AAB6D3F4029D', 'dummy', '   dummy', 'dummy ', ' ', now(), now());

Preview trailing spaces in dummy data
Run make db_dev_migrate
Verify spaces are gone and if ' ' then null

Couple SIT tests failing

Those should be fixed once he fixes his CONFLICTS CAMERON

Not seeing this work for me locally for the transportation_accounting_codes table specifically. It looks good for the lines_of_accounting table
Running make db_dev_migrate doesn't actually do anything since it has already run when I switch to your branch, so: Steps I'm taking:

Inserting the dummy data into the LOA table
Running your migration directly in dbeaver as a script
Updates the LOA dummy data to not have extra spaces

After:

Not seeing the same TACs

Inserting the dummy data into the TAC table
Running your migration directly in dbeaver as a script
Not updating the TAC dummy data to not have extra spaces

After running migration as script and refreshing dbeaver, no change

@traskowskycaci
This could be due to an issue of when copy and pasting the single quotation ' might be a different ASCII(?) format or dbeaver not visually representing right. I tested again and it worked as expected, could you retry and let me know if it persists?

@traskowskycaci This could be due to an issue of when copy and pasting the single quotation ' might be a different ASCII(?) format or dbeaver not visually representing right. I tested again and it worked as expected, could you retry and let me know if it persists?

Yep, I'll retest!


Glance does not highlight .diff or .patch

Glance does preview both of them.  They're both registered with the system with a UTI of "public.patch-file".  But it does not highlight them.
Chroma knows what a diff is, and will highlight it.  Pasting the same file into the Chroma playground produces a highlighted file.
Environment
Glance 1.2.0 from the MAS
Catalina 10.15.6
Bug Information

What are you trying to do?
Preview .patch and .diff files.

What's the expected behavior?
Previews are displayed with syntax highlighting.

What's the actual behavior?
Previews are displayed without syntax highlighting.

+1


Adding a GitHub Pages site for documentation

Added a site directory to hold the GitHub Pages content
When a new release is created this will be published to the gh-pages branch to make the content publicly accessible
PRs will need to include documentation updates where appropriate
Added development instructions and a Travis CI build for the site
Added CONTRIBUTING.md to link through to the documentation ready for when it is published
Added the site build to the main Travis CI build so that PRs can be validated before publishing

Once this PR is merged the site will need to be published so that there is an initial version and the contribution link works - the site page can then be added to the description box for the repository.

Just one last crease to iron out around trying to use a relative path in the compose file for mounting in the site....

Site published and URL added to the top of the repository page as well. The "gh-pages" branch will normally be updated when a new release is published for this repository, but there might be smaller increments until then as missing sections are added.


Debugging session is ended early when forked process exits

When a ruby process is forked, the debugger initially works as expected. Breakpoints fire correctly in both the parent and child processes.
However, when the child fork exits, the VSCode debugging session seems to end prematurely. That means that further breakpoints in the parent process do not fire correctly.
Here's a minimal reproduction:
$start = Time.now

def log(msg)
  time = (Time.now - $start).round(3)
  puts "[t=#{time}s] [pid=#{Process.pid}] #{msg}"
end

log "Starting script"

fork do
  log "Fork started"
  sleep 1
  # sleep # (uncommenting this allows the main process debugger to work correctly)
  log "Fork done"
end

sleep 3

log "Set breakpoint on this line" # <<<< Breakpoint
log "Done script"

This produces the output
❯ rdbg --command --open --stop-at-load --sock-path=/var/folders/c7/
3jzsz23s16l3_jtzrttkxwfw0000gn/T/ruby-debug-sock-501/ruby-debug-david-51037 -- ruby hello_world.rb
DEBUGGER: Debugger can attach via UNIX domain socket (/var/folders/c7/3jzsz23s16l3_jtzrttkxwfw0000gn/T/ruby-debug-sock-501/ruby-debug-david-51037)
DEBUGGER: Connected.
DEBUGGER: wait for debugger connection...
DEBUGGER:  BP - Line  /Users/david/hello_world.rb:19 (line) is activated.
[t=0.0s] [pid=51105] Starting script
DEBUGGER: Attaching after process 51105 fork to child process 51106
DEBUGGER[hello_world.rb#51106]: Connected.
[t=0.002s] [pid=51106] Fork started
[t=1.004s] [pid=51106] Fork done
DEBUGGER[hello_world.rb#51106]: Disconnected.
DEBUGGER[hello_world.rb#51105]: Disconnected.
[t=3.003s] [pid=51105] Set breakpoint on this line
[t=3.004s] [pid=51105] Done script

The VSCode 'debug' environment appears to terminate at the same time as 'fork done', and the breakpoint is never fired.

Launch configuration:
{
  "name": "Debug Script",
  "command": "ruby",
  "script": "hello_world.rb",
  "request": "launch",
  "type": "rdbg",
},

❯ ruby --version
ruby 3.1.3p185 (2022-11-24 revision 1a6b16756e) [arm64-darwin21]

❯ gem info debug

*** LOCAL GEMS ***

debug (1.6.3)
    Author: Koichi Sasada
    Homepage: https://github.com/ruby/debug
    Licenses: Ruby, BSD-2-Clause
    Installed at: /Users/david/.rvm/rubies/ruby-3.1.3/lib/ruby/gems/3.1.0

    Debugging functionality for Ruby

Extension version v0.0.11

Now vscode-rdbg doesn't support fork well. Thank you for the report.


Create a proper documentation for the package

@Rajdeep1311 the package is already in pypi so it’s not necessary.

@shhossain is this issue still up for grabs? I would like to document the package.

@Tawishi yes.

@shhossain due to some unforeseen situation/work, I will not be able to contribute.

@shhossain just wanted to know what all are you looking forward to seeing in the documentation , I am kinda new to all this but can give it a try if you could give me an overview of what all u are expecting.
Thanks

I'll be happy yo help, I'm currently working in a university lab with a lot of DataScients, or phd, so I think I can be useful. Plus, I'm not afraid to create some documentation nor with anything

@BuerJeanMaxime feel free to submit PR. You can include documentation on docs folder.

Thanks, I fork the repot right now

i think this is still open, I'll be happy to help. add a label hacktoberfest. and also i'll try to add insightface's buffalo model if possible it gives 512 dimensional vector and shows better accuracy on south asian and east asian data. [better than face-recognition and deepface's models].

Is this still being worked on?

@Disco-Danny I got caught up with some other work, so please feel free to work on this..

Can I work on making a documentation for this package?


Remove gem install bundler

We can use the default Bundler that comes with Ruby, and that should automatically install the version locked at BUNDLED WITH inside the lockfile and use that.

Didn't work and I won't dig deeper into why.


What is the status of version 0.3?

I see there's a 0.4 alpha version in releases/c. Is the 0.3 version still considered unstable (per the README.md file that states that 0.2 is the latest stable)?

If you're basing that on these lines in the README.md file:

Status
Version 0.3. The API and ABI aren't stabilized yet.

"Aren't stabilized" refers to the 0 in "Version 0.X". Version 0.N (for N > 3) isn't guaranteed to be API or ABI compatible with 0.3. In contrast, there isn't a version 1.0 yet, but I'd expect 1.0 to be compatible with version 1.1 (but not with version 2.0).
Version 0.3 is just as stable as version 0.2. They are both very unlikely to change from here, other than security updates.
New features, new standard library packages, new API, API/ABI breakage, etc. all happen on the work-in-progress version 0.4 stream.
In summary: use version 0.3.
Sorry for the confusion.

Thanks for the clarification! I should have been more specific about which README.md I was referring to: it was the one in release/c:

Version 0.2 (December 2019) is the latest stable version.

Ah, I forgot about that one. Thanks for the bug report.


Prefix to Postfix in cpp

Have you read the Contributing Guidelines on Pull Requests?
yes
Description
fixes #6387
Checklist

[x] I've read the contribution guidelines.
[x] I've checked the issue list before deciding what to submit.
[x] I've edited the README.md and link to my code.

Related Issues or Pull Requests
issue #6387

@HarshCasper  @ankitaggarwal23  @anushkrishnav Can you please review my pr


172877789 te mc wrapper

This PR includes minor updates to the custom message callback which is needed to send messages from the TE plugin to the MC Library Interactive.  Also includes updates to the sample data to make testing a wrapped MC question easier.

This looks good to me.
I'd remove the TODO since I don't think there is anything needed there.
My only concern is that it seems there is no test that is actually triggering the post("customMessage", message)
It looks like that iframe-runtime.tsx file is covered some by cypress and some by jest.
I don't know the activity player and its test well enough to know if this is something easy to cover or if it would be hard.

I removed the TODO comment - I put that in only to make sure we looked at it before merging.  Not sure on the test.  I would have assumed any needed tests were in place in past PRs.  Not sure exactly how we would test that.  Will require more thought.

This looks good to me.
I'd remove the TODO since I don't think there is anything needed there.
My only concern is that it seems there is no test that is actually triggering the post("customMessage", message)
It looks like that iframe-runtime.tsx file is covered some by cypress and some by jest.
I don't know the activity player and its test well enough to know if this is something easy to cover or if it would be hard.

I removed the TODO comment - I put that in only to make sure we looked at it before merging.  Not sure on the test.  I would have assumed any needed tests were in place in past PRs.  Not sure exactly how we would test that.  Will require more thought.


Fix document.registerElement Chrome warning.

In Chrome you see the following warning:
[Deprecation] document.registerElement is deprecated and will be removed in M73, around March 2019. Please use window.customElements.define instead. See https://www.chromestatus.com/features/4642138092470272 for more details.
I've removed the JS part of coral and update the CSS to use Coral 3.17.0.

It looks good. We have a couple issues though:

If a user already has a libSandbox.html, it's going to still be using CoralUI whereas their viewSandbox.html will be using Spectrum. Not a big deal--I can probably get over that.
With the new libSandbox.html loading spectrum from spectrum-css/2.6.0/standalone/spectrum-light.css, if we ever want to upgrade the sandbox to use a new version of Spectrum or a different style library entirely, we either have to continue bundling and serving spectrum 2.6.0 with the sandbox or accept that any libSandbox.html people have locally that reference spectrum-css/2.6.0/standalone/spectrum-light.css will break. I think our current setup probably isn't scalable and we should talk it over.

I approve.


Guild System

Currently merges into #114 due to master being fucky.
Contains the generic features needed, but no system for perks, housing, leveling and all that.
Code is flawless so probably dont even need to look at it like at all. please
All visual aspects of this is still using "devart" and not touched by artist and designers, so all that will change

Merged into dev so future PRs will branch of that


3D mesh export

Hello. Thank you very much for this great software
Do you have any plans to add 3D mesh export capability in the future?
Thankful

I second this, I would love to be able to export OBJ files.
It would also be neat if we could choose to have the possibility of bilinear filtering, both in the texture of the 3D Object, and as one of the Scale Algorithms (to allow to render 3d files at a higher res and then scale it down, generating antialias.
I understand and use Pixel Composer as a Pixel Art tool, but I think being able to have smoother textures on 3D models can open up many ways to use pixel art in different settings!

Also, exporting as Voxel and vox format is an interesting option

Export... well it's possible. I have a plan to rework 3D system (using BBMOD if you've heard about it), I will put this in todo list along with it.


Weird issue with --coverage flag [jest]

https://github.com/facebook/jest/issues/3158
paste:
--coverage opt broke my tests, why?
I have two scripts at my repo's package. https://github.com/k1r0s/kaop
    "test": "jest",
    "coverage": "jest --coverage",

´coverage´ fails but ´test´ runs as spected. Why?

My problem is about instrument code breaks some calls. This is because some methods/fucntions are mutated by eval. So istanbul try to match expressions which no longer exists in mutated declarations...

There is anyway to skip or bypass some blocks for instrumentation?

https://github.com/gotwarlost/istanbul/blob/master/ignoring-code-for-coverage.md

https://github.com/k1r0s/kaop/commit/a553a6dd41d23868811f2a1754933f09d48bfe9b


Build image from uploaded file: button not enabled

Bug description
On the build image page the "Build the image button" button stays disabled after entering the image name, selecting Upload and uploading a file.

Expected behavior
The button should be enabled so the user can initiate the image build.
Portainer Logs
Provide the logs of your Portainer container or Service.
You can see how here
Steps to reproduce the issue:

Go to Images > Build image
Enter image name
Click on "Upload"
Click on "Select file", select a docker file

Technical details:

Portainer version: 2.15.0
Docker version (managed by Portainer): 20.10.17
Kubernetes version (managed by Portainer): n/a
Platform (windows/linux): Linux
Command used to start Portainer (docker run -p 9443:9443 portainer/portainer):
Browser: Microsoft Edge, Chrome
Use Case (delete as appropriate): Using Portainer in a Commercial setup.
Have you reviewed our technical documentation and knowledge base? Yes

Additional context
This problem only surfaced after upgrading Portainer from v2.13.2 to v2.15.0. No problems building images using the Web editor or URL option.

@duncanw
I believe this issue is related to 7624 and is resolved in 2.15.1.
Thanks!

Thanks! Yes sorry looks like I created a duplicate issue.


Any way to restrict menu opening to a certain point away from the screen edge?

I have a view with a horizontal ScrollView in it and I find myself accidentally opening the menu too often. I've played around with the toleranceX prop a little bit, and this does help - but not completely.
I find it weird that you can drag somewhere very far away from the edge of the screen and have it still trigger the menu. Would it be possible to add a prop like edgeTolerance where I could specify 40 and only touches that originate 40px from the edge can trigger the menu?
This would definitely prevent accidental menu openings when interacting with my horizontal ScrollView

I'm sorry, not sure I got what you mean by this. Why toleranceX doesn't work for you? Very far away from the screen?

@Kureev I'll try to be more clear. toleranceX seems to only affect how FAR you have to drag before it triggers the menu, but fails to take into account the origin of the drag. It seems weird to me to have a left side menu that you can drag open with a touch that originates on the right side of the screen. It would be nice to specify that only touches that start on the left can drag open the menu.
Consider a left side menu. I'm imagining a way to set this hypothetical edgeTolerance to 40 so that only touches that start within the first 40 pixels from the left side of the screen will be able to trigger the menu. If I originate a touch at x=50 and then try to drag to the right, it would have no effect since I specified it must be within 40.

:+1:

@marcshilling I've took a look on it and I can assert that it respects the point when you started your drag. Sorry, that it took more than 1 month for me to check, but if you update the lib to one of the latest versions, you would be able to experience this :+1:


Size parameter in TCPConnection and TCPListener

Work for https://github.com/ponylang/ponyc/issues/653

Hi @Nyarum, this looks good! Two things:

It's an initial size. It is doubled if the TCPConnection reads a "full" buffer, in _pending_reads and _complete_reads (the former for POSIX OSes, the latter for Windows). Can you rename the parameter init_size ?

I think a second parameter of max_size would be good, that defaults to, say, 16384. Then in _pending_reads and _complete_reads, something like:

next = _max_size.min(next * 2)

Thoughts?

@Nyarum once this is squared away with @sylvanc, you'll also need to:
Update the CHANGELOG to reflect your change
Squash the commits down to 1 commit and write a good commit message.
See https://github.com/ponylang/ponyc/blob/master/CONTRIBUTING.md for details and feel free to ask for any assistance you might need. I'm happy to help if you are unfamiliar with squashing a git commit or what a good commit message for this would be.

@sylvanc I think it looks good. I'll make changes for that.
@SeanTAllen okay, thank you.

@sylvanc hey, I have added changes with your comment. Can you check?

@Nyarum this looks pretty good. One question for @sylvanc, should TCPListener take a max_size as well to be passed to in to accept? At the moment, TCPListener can't change that value.

Good point @SeanTAllen , TCPListener should be configured with the initial and max sizes as well. This is a good PR @Nyarum , thanks.

@nyarum up for making that additional change?

@SeanTAllen yeah, I will make changes in 1 day. I will need help with squash commits.

@Nyarum - actually we are investigating a new feature of GitHub that should allow us to squash-on-merge, so you don't have to squash yourself.  Hopefully that makes it easier for more people to contribute.  Thanks for your work so far!

@jemc good news.

awesome @Nyarum. thanks.

@sylvanc @SeanTAllen I made a changes.

Wow, that's cool - using the squash-on-merge feature, the GitHub view for this PR still retains all of the individual commits, so you can see how they relate to the history of the review process.  That's a huge improvement over the case where you squash the branch manually before merging, and lose all that history.  It makes me a little less surly about squashing than I was before :smile_cat:.

Unfortunately we have to write up the commit message at that point. So, not 100% perfect. Almost though.


Felgo intro chapter

Introductory chapter to Felgo, the Felgo SDK.
Includes a quick setup guide, a hello world, a 6 stage guide building a messaging app, and a collection of snippets demonstrating useful features of the SDK.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.1 out of 2 committers have signed the CLA.:white_check_mark: e8johan:x: Swordfish90You have signed the CLA already but the status is still pending? Let us recheck it.


Provide an easy way (or instructions) for stopping a running romanesco-worker

Since we call celery.Celery.worker_main, we can passthrough any arguments we want to the celery worker. We may wish to also expose another entrypoint, or reuse the existing one and accept some other arguments, that could be used to stop a running worker, perhaps via --pidfile.

If we find some way to do this using our already existing entry point, we should simply add the start and stop command to the documentation.

Sounds reasonable to me.


xIisLogging: Moving strings to localization file

Pull Request (PR) description
xIisLogging: Moving strings to localization file
This Pull Request (PR) fixes the following issues
- Fixes #464

Task list

[ ] Added an entry under the Unreleased section of the change log in the CHANGELOG.md.
Entry should say what was changed, and how that affects users (if applicable).
[ ] Resource documentation added/updated in README.md.
[ ] Resource parameter descriptions added/updated in README.md, schema.mof
and comment-based help.
[ ] Comment-based help added/updated.
[x] Localization strings added/updated in all localization files as appropriate.
[ ] Examples appropriately added/updated.
[ ] Unit tests added/updated. See DSC Resource Testing Guidelines.
[ ] Integration tests added/updated (where possible). See DSC Resource Testing Guidelines.
[ ] New/changed code adheres to DSC Resource Style Guidelines and Best Practices.

This change is

Codecov Report

Merging #510 into dev will not change coverage.
The diff coverage is 100%.

@@           Coverage Diff           @@
##              dev     #510   +/-   ##
=======================================
  Coverage   91.39%   91.39%
=======================================
  Files          17       17
  Lines        2661     2661
=======================================
  Hits         2432     2432
  Misses        229      229

Impacted Files
Coverage Δ

...SCResources/MSFT_xIisLogging/MSFT_xIisLogging.psm1
97.84% <100%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 22802e7...514acc4. Read the comment docs.


fix xfs fail, add upload result when timeout

increase the core number to 8 for xfs verify_generic_standard_datadisk
add try logic for exception to upload the test result
reduce the wait timeout in wait by 20s to let wat timeout happen before the whole case timeout.

@LiliDeng LGTM


GKE cluster version error

Issue/Request:
GKE cluster version error when specifying a version patch suffix:
ERROR: (gcloud.container.clusters.create) ResponseError: code=400, message=Master version "1.11.2-gke.9" is unsupported.

I would recommend using just major.minor I.E. 11.1

yes, good point, is there a way to force GKE to deploy always the latest version?

I think it's safe to remove the --cluster-version and let GKE default to the latest version for the region on the master/node


Validation of script references in vue html/template stopped working

What I'm using

VSCode
Quasar (Vue)
Typescript
Eslint/@typescript-eslint
Prettier / prettierhtml / Vetur

Current issue:
Until recently, injected script in the

Please open a new issue in https://github.com/vuejs/vetur, with repro case created from https://github.com/octref/veturpack.


please add np.min()/max() with augment:"axis"

when  dealing with a n-dim array. I can't get the easiest work done like getting the smallest number of each row.

related https://github.com/numba/numba/issues/1269

xref #4414, possible duplicate?

Closing as a duplicate of #1269


Specify Status when registering Service in Consul

When registering a consul health check, the service by default enters in a "critical" state and eventually switches to "passing".  In order to clean up the unhelathy services, I believe its a good idea to pass an initial state to consul registration. Currently the registrator implementation doesn't take the Status flag, it accepts check_http, check_timeout, check_cmd, check_script, check_ttl and check_interval. It will be good if we accept checck_status and pass it to the Status attr of Consul AgentServiceCheck struct. This will enable enter the Service in a warning state for example and become passing later when health check succeeds. Currently when we query Consul for Critical services for clean up, we are not sure whether the service is unhealthy or it was just registered. Will submit a PR. Also planning to check with Consul folks if they can change the initial registration to "warning" rather than passing.

Please review https://github.com/gliderlabs/registrator/pull/343


Revert "Forbid return <expr> when returning void (#1355)"

This reverts commit 1ef624a315b68e3ca51fcb0e5bed2038862f510a.

Motivation

We didn't test this enough, and it's incorrectly making errors when it
shouldn't.
Test plan

Could you illustrate a case where it was wrong?

Could you illustrate a case where it was wrong?

From IRL: it doesn't handle the distinction between blocks and procs.

Also there's just a lot of things we don't fully understand about the Ruby VM here.

@x = nil
def foo(&blk)
  @x = blk.to_proc
  # @x = lambda do
  #   puts 1
  # end
  # @x = lambda do
  #   blk.call
  # end
end

def bar
  foo do
    return 1
  end
end

p bar

def qux
  @x.call
end

p qux

note to future me: we were trying to make sense of this example

More context: This impl depended on the supposed fact that, if you see a return expr inside a method def, then that return is returning out of the method. This is not true, because of procs and lambdas and blocks and things.
Additionally, we think that it is not statically knowable in general whether a return inside a block-looking thing will return from the enclosing method or return from the block itself.

Here's a diabolical example to show that it's not statically knowable:
def sometimes_transform(&b)
  if rand > 0.5
    o = Object.new
    o.define_singleton_method(:block, &b)
    o.block
  else
    b.call
  end
end

def who_knows
  sometimes_transform { return :inner }
  :outer
end

puts who_knows

The return inside the block sometimes returns who_knows, sometimes doesn't.


Automate html table generation from json file

Closes #54

thanks @srinath1412001 , hopefully it will make it easier for people


Add devfs for incremental compiler JavaScript bundle

Description
Adds a WebAssetServer that can serve the JavaScript bundles from the incremental compiler. This isn't used yet.

I've updated the logic based on my local working copy of incremental compiler driven hot reload.
Changes:

Simplified the process of choosing which file to load. The process is now independent of file extension.

Attempt to resolve in in-memory cache.
Attempt to resolve in file system.
Attempt to resolve relative to built asset directory.

Remove /asset/ instead of asset/ so we don't look for absolute paths.
Asset manifest uses file paths instead of file uris.
Adding a manifest returns the list of valid file paths.


slacktee.sh doesn't work with crontab because of $PATH issue...

slacktee.sh by default doesn't work with crontab because /usr/local/bin is not included in crontab path, so manually i have to move into /bin directory, it would be great if you move in /bin or /usr/bin directory by default.
I took quite long to figure out these thing.

I'm sorry for the late reply.
I got your point. You can change the target directly by specifying it as a parameter of install.sh, but doesn't it work for you? Here is the example: bash ./slacktee/install.sh /usr/bin
Please correct me if I'm wrong, but usually /bin is used for storing the commands which need to be available in single user mode, so I don't think it's a good idea to add it to there. (This wikipedia page is very helpful).
/usr/bin is probably fine, but personally, I would prefer to use it for storing the commands which are maintained by package managers. So, that is why I'm using /usr/local/bin as a default directly.
Hopefully, this can convince you ;)

I will close this, but please feel free to let me know if you have any other ideas :)

I have only one idea to add, It will be good to add in readme that it won't work with crontab just because path issue, because I feel most people  gonna use it for scripting automation like me ;) and crontab is one of them, so adding in readme will be helpful.

Thank you so much for the suggestion! I will update README to add the note :)

Updated README!


Moving code for PDF related stuff  in another component is not working

Configuration:

Chrome :89.0.4389.82
Windows 10
"vue-pdf-app": "^1.0.1",

Steps to reproduce the problem:
Download the code from Git hub and try to run typescript app. it works fine now create component folder under src and then create helloworld .vue file there move all pdf related code to this file from app.vue as below
<template>
  <div id="app">
    <pdf-app<EMAIL_ADDRESS>    testing it
  </div>
</template>

<script lang="ts">
import PdfApp from "vue-pdf-app";
import { Component, Vue } from 'vue-property-decorator';

@Component({
  components: {
    PdfApp
  },
})
export default class Hello extends Vue {}
</script>

it gives error Invalid
 Invalid or corrupted PDF file

What is the expected behavior?
it should show file as it was showing earlier while code was in app.vue  file
What went wrong?
it gives error

Attachments (screenshots, links, etc)

It started working when i changed path from
<pdf-app<EMAIL_ADDRESS>
to
 <pdf-app pdf="./sample.pdf"></pdf-app>


Video Renderer

Describe your problem here. If you are modding, please try testing your problem in a clean version of the engine instead, this ensures the problem is actually caused by the engine itself. Also, if you're using an older version of JSE, please try the latest action build in the Actions tab or the latest release build. The issue has probably been fixed since that version. Oh yeah, and also be sure to check the pinned 'JS Engine: Known Issues' issue, to double check if your issue hasn't already been found!
yes ik its me again but hwo do you use Video render in game rendering? I have ffmpeg but every time i try to play a song to record it. It always gets rid of the songs sound and it doesnt actually convert it into .mp4. it doesnt even record the song!
Are you modding a build from source or with Lua?
Lua
What is your build target?
Windows x64
Did you edit anything in this build? If so, mention or summarize your changes.
No response

you have to put your ffmpeg.exe in the same folder that JSEngine.exe is located in.
getting rid of the songs' sound is intentional, when the video is playing obviously it won't have any audio

I did it but whenever i go into a song with video renderer on it shows this error.

how do i fix?

idk man, i don't get that error so i can't help you

dang

idk man, i don't get that error so i can't help you

Simple answer:
Missing the dll file

Ok so basically I renderered a song then I go in cmd and run the command 'ffmpeg -r 60 -i ./RAGE/%07d.jpg  rage.mp4' It does it but then this message comes up

..when formatting to a song path the game doesn't put capital letters

what i dont understand. I made the name (RAGE) lowercased but it still didnt work.

Ok I finally fixed the problem! and i blame myself for this.. I forgot to add ffmpeg to same folder as js engine. The only thing I need to do is add the sound and yeah. Thanks Jordan 👍


Make most public types clonable

Greetings again!
Playing around with this library I found myself trying to to some 'virtual filesystem'-stuff. This involved re-using a few types from russh-sftp. I discovered (via File) that a lot of types there don't implement Clone though they easily could via derive.
I think that for most public types, especially the ones that get give to the user via the Handler trait should implement Clone. The setup of this trait requires a lot of things to be passed as owned values.
An example is the russh_sftp::protocol::Name struct taking an owned Vec<File>. I understand the design decision why this is the case, but then those Files need to come from somewhere. And if I already have them, it would be nice to be able to easily clone them.
One struct I did not add Clone to is client::session::Session. I added a test to keep this as is with a small explanation why.
Happy to hear any feedback on all this and feel free to let me know about other opinions on all of this!

Hi, thanks for your input. Could you provide a more clear example of when this might be useful?

So, I thought a little more about it, and: maybe I got a little carried away  in adding Clone everywhere :smile:
I think my original use case is valid: You have individual clients connecting and serve already cached Files to them. That means storing them somewhere (e.g. the SftpSession) and responding with them on appropriate readdir calls. To do so, You'll have to give an owned Vec<File> as a response, for which it would be great to be able to just clone it out of the stored files.
I got carried away in the sense that cloning something which carries a request id in it's response probably does not make sense. If someone want's to respond, they have to assemble the response with the provided id anew. If they, for some reason, want to do something special with the id, they should store the id separately, no use in making the whole struct cloneable.
So, if you agree with the above use case, I'd change this PR to exclusively implement Clone for File? FileAttributes already has it.

So, if you agree with the above use case, I'd change this PR to exclusively implement Clone for File? FileAttributes already has it.

Yep, I think this is the best solution.

Done. Anywhere you prefer to document this decision or do you think having this PR is sufficient?

Documenting such little details is unnecessary, as they are clearly specified for the type


Regression: Since 0.11.5 – Godot deadlocks, distorted images

I compiled recent github tips from Godot (4.3) and ThorVG today and see a huge increase in deadlocks (I have to switch the editor to single window mode to keep it running) and SVG -> texture, icon distortions:

Godot git-tip + ThorVG v0.11.5 works fine, not sure what the reason is. I can take a closer look next week with the debugger.

@capnm Hello, thorvg is undergoing significant changes for its 1.0 release. With this release, there will be some compatibility breaks, and these changes have already been implemented in the main branch. Meanwhile, we will maintain compatibility during the v0.x releases. Therefore, v0.11 (and upcoming v0.12) must be compatible with Godot. For your information, I guess these two patches in main might occur abi breaks in godot.
commit f2c29063d25cce3efc1b6d4039bd54dc23d8d630
Author: Hermet Park<EMAIL_ADDRESS>Date:   Thu Nov 23 01:10:18 2023 +0900

    api: updated the recent changed api again.

    Reordered by the data packing size.
    Also removed a wrong capi default parameter value.

    Result Picture::load(const char* data, uint32_t size, const std::string& mimeType, bool copy = false, const std::string& rpath = "")
    -> Result load(const char* data, uint32_t size, const std::string& mimeType, const std::string& rpath = "", bool copy = false)

    Tvg_Result tvg_picture_load_data(Tvg_Paint* paint, const char *data, uint32_t size, const char *mimetype, bool copy, const char* resourcePath)
    -> Tvg_Result tvg_picture_load_data(Tvg_Paint* paint, const char *data, uint32_t size, const char *mimetype, const char* resourcePath, bool copy);

commit 548962f5f8868429b9ebf5ce6fb9005dcb9a606a
Author: Sergii Liebodkin<EMAIL_ADDRESS>Date:   Mon Nov 20 10:59:38 2023 +0900

    apis/engines: Revise the clear() buffer behavior.

    ThorVG has offered an option to clear the buffer since version 1.0.
    This is essential when users utilize the canvas target buffer
    with the main render target. They share the buffer
    and need to draw contents onto the existing contents.

    API:
    Result Canvas::clear(bool free = true)
    -> Result Canvas::clear(bool paints = true, bool buffer = true)

    Tvg_Result tvg_canvas_clear(Tvg_Canvas* canvas, bool free);
    -> Tvg_Result tvg_canvas_clear(Tvg_Canvas* canvas, bool paints, bool buffer);

    Issue: https://github.com/thorvg/thorvg/issues/1779

    Co-Authored-By: Hermet Park<EMAIL_ADDRESS>

@capnm I suggest you to see the v0.11.x branch for sync-up, and v0.12.x when it's introduced(planning it end of this month)

@capnm Hello Martin, Happy New Year! ThorVG is set to be released with version 0.12.0. Could you please confirm if there are any regression issues from the Godot? You can see v0.12.x branch for the integration: https://github.com/thorvg/thorvg/tree/v0.12.x

@hermet I'm getting some linker errors right now, but I'll fix it ASAP.
Happy new year!

LGTM!
Current CI run:  https://github.com/capnm/godot4/actions/runs/7370790095


Random server "crash"

On my RP2 is the most thing working fine. But there are a few problems with the netcode (also in tshock) and a random Server crash. After the "crash" the console is refreshing but a command like save-all isn't working. In the log i can't find an error.

If there is no crash in the log are you able to run "mono --trace=program tdsm.exe"? If there's an exception that trace flag should help it be shown in the console.
Unfortunately it wi take some time, you may want to grab a coffee while it loads the world.

I will test it at noon (GMT +2 :D) today. I hope I didn't forget it ^^

It's been a while with no response, so I'm going ahead and closing off this issue. The new connection code is in and some crashes have been fixed. If this issue is not actually resolved, please do create another.


Upgrade Babel to ^6.0.0

Would be great to get this done as soon as possible; keeping an eye on https://github.com/babel/babel-loader/issues/123.

<EMAIL_ADDRESS>has been released with babel 6 api support just have to wait for some other deps to catch up—specifically the react-transform plugin and related repositories
see the discussion on babel-plugin-react-transform issue (dan indicates support is coming with next release, sometime in the next couple weeks)
i'm tracking this closely and will submit pr when deps are ready

Awesome thanks. I was naive in thinking that once babel-loader was caught up that would be most of it.

Is there any update for it?

@starandtina I've had a difficult time figuring out where react-transform/hot-reload and redbox-react are, so hopefully within the next week or so this will finally be @ babel 6. The process has just gotten harder since babel disabled github issues.

@starandtina @davezuko i'm still on top of this guys. there is a PR—used on the babel-6 branch of my fork of react-transform... HMR, catch-errors, etc are all working. waiting for author to merge.
also, to support eslint we'll need to wait for babel ast conversion to esprima, which is also in the works

Thanks @justingreenberg, I just stumbled upon https://github.com/gaearon/babel-plugin-react-transform/pull/50 before seeing your edit so it looks like you're far ahead of me.

@justingreenberg looks like https://github.com/gaearon/babel-plugin-react-transform/pull/50 just got merged and a pre-release is on the way later today.

Awesome—I already have a working branch of this repo with my local fork of react-transform. Just have to review, merge and rebase. I'm meetings until around 4pm EST, will submit PR today whether pre-release get pushed or not :)

Thanks! You're the best haha; I feel like every issue thread I'm in I see you with a PR or something. Anyways, looking forward to it; the test setup will probably have to change a bit now that isparta is on babel 6 (iirc), but either way super excited to finally get this moving!

Haha I totally got sucked in. Babel and Webpack were always kinda just magic to me. Dan's move to london and resulting delay in react-transform ended up being a blessing in disguise—it forced me to sit down read the babel source like a book and totally demystified the ecosystem. Now I'm hooked :)

Not to sidetrack this too much, but is that what helped you the most (reading the source)? Babel is still mostly a black box to me despite some of the talks I've watched... so I'm going to be going through https://github.com/thejameskyle/babel-plugin-handbook today and then maybe dig into the source later.

babel-plugin-handbook is by far the most comprehensive resource—I strongly recommend starting there.
@thejameskyle really does an awesome job introducing and breaking down the internals:
parsing (code → lexical [tokens] → syntax tree [AST])
traversing (visitors, paths, scope, binding)
tranforming nodes in the AST. Babel's API surface is surprisingly small.
Since bascially everything is a plugin in v6, there are a ton of examples in babel-packages for everything from sugaring syntax with babel-plugin-syntax-jsx to transforms like babel-register that traverses an entry module and transpiles all downstream imports/requires. it's super moduler and most of the packages are easily digestible in a few minutes (they are tiny, most are like < 50 LOC) :smile:

Also FYI, hidden away in the documentation branch of his personal babel fork, james has started work on documenting the module APIs—currently there's babel-traverse and babel-types

Thanks a bunch, I'll devote some time to reading through all these when I can.

Closing this for now since it's been merged; additional issues are welcome for any bugs that may arise. Thanks for all your work @justingreenberg!


libsodium: 1.0.17 -> 1.0.18

Motivation for this change
https://github.com/jedisct1/libsodium/releases/tag/1.0.18-RELEASE
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Assured whether relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

Are you sure about the branch, though?

Oh, yikes, rebuilds are higher than I expected/realized.  Definitely staging first, don't believe anything critical here so that seems pretty clearly not appropriate for master :).  Sorry for missing that, thanks!

Sigh.  Wish github had a UI for rebase --onto or someting O:).  Anyway sorry for the above madness.
Looks like this is already on staging, so all is well and nothing more to do here I think!
Thanks for the feedback regardless! <3

@dtzWill I think libsodium aims to be «the default choice when you need crypto and sane defaults but nothing fancy», and they are succeeding in selling this narrative… so — a lot of rev-deps.

«Wish that GitHub would have a consistent or at least working UI for any of the obviously needed features» — oh well


Chart Height is constant even when setting its height with in initOpts

charts are outputted in streamlit app in constant size which makes them sometimes unreadable.
c = Line(init_opts=opts.InitOpts(theme=ThemeType.DARK, height='1600px', width='1600px'))

unlike the stock visualization components of streamlit, the echarts components doesn't have the 'extend' button which the other components have and enables full screen view of the chart.
I know that this probably a streamlit related issue rather than to this package, however - if there is a way to force the chart height or at least add a full screen view option that will be great.
Thanks

Hello @avivazran !
Unfortunately the Pyecharts InitOpts don't seem to be passed as options to echarts, so theme=ThemeType.DARK, height='1600px', width='1600px' need to be passed at the st_pyecharts call instead to be used.

c = (
    Line()
    .add_xaxis(Faker.choose())
    .add_yaxis("商家A", Faker.values())
    .add_yaxis("商家B", Faker.values())
)
st_pyecharts(c, theme=ThemeType.DARK, height="1600px", width="1600px")

I'm also looking into an issue on resizing the chart for responsive design for #27 and #28.
Have a nice day,
Fanilo


Extra EntityManager service kills EntityManager autowiring

Hi buddy!
We were just upgrading to the latest version of the bundle, and suddenly our EntityManager couldn't be autowired into any services anymore! The problem is that this bundle introduces a second service, which uses this class: https://github.com/kbond/ZenstruckRedirectBundle/blob/7504b70a46ab663273a1437bbbdbe37091591a65/src/Resources/config/orm.xml#L8
I see that the purpose is to be able to control which entity manager is used. Could this be done in a simpler way with an alias? For example, don't actually create a service called zenstruck_redirect.entity_manager, simply set an alias in the container so that zenstruck_redirect.entity_manager is an alias to the correct entity manager service (e.g. doctrine.orm.default_entity_manager where default changes to be the model_manager_name config). This would avoid needing to add the additional service to the container, which causes the issue.
And to really try to tug at your heart-strings... it's even technically (though barely) more performant! The current method requires an extra call to $container->get('doctrine')->getManager() to get the service, instead of re-using the existing. Ok, this last reason is CRAP, and would save only 1ms every 1000 requests... but I'm just trying to sweeten the deal ;).
Cheers!

Also, I just asked Beckett, he's totally 👍 for my suggested change.

Hey Ryan!
I can't quite remember out why I did it that way... My guess is I was following the conventions from another bundle.
I'll take a look and get this fixed. Like you say, an alias seems like the best solution.
K


💡 Update Readme.md

I would like to update the readme file for the App-Development, Data-Science and DBMS.
assign me this under GSSOC

What do you mean? Like what update, is there any mistakes, typos, or are you going to add resources. Please specify in your description, thanks

hi @jfmartinz
I would like to work on this issue . so please assign this task to me under the label gssoc24 and level1,2,or 3


Add examples of using the GPU as render target

The current examples focuses on using the CPU based rasterizer only.
As the Skia documentation is a bit thin around using a GPU render target, I would encourage to include examples of using the Skia library to render its surface with the GPU (OpenGL and/or Vulkan). 😃
I might be able to submit an example too, if time allows to dig through the Skia documentation.

It's dangerous with the current delphi code, because we can't mix the firemonkey code (Tcanvas / TBitmap) with the skia code because of the context change that each one must do before using any GLES call.

In the current state, it could be dangerous, if you don't know what you're doing, I agree.
However, there are times where you want to take over the management completely yourself and bypass FMX, then it isn't an issue.
Have you logged tickets in Quality Portal and proposed a change to the FMX code in question to facilitate safe access in FMX?

Have you logged tickets in Quality Portal and proposed a change to the FMX code in question to facilitate safe access in FMX?

Not because there is no easy solution for this, editing in source would only be to raise when a developer mixes firemonkey code inside the anonymous drawing function with skia. Without a doubt, the best solution would be to make a skia canvas that replaces firemonkey's TGPUCanvas, but this is a colossal job and we don't plan on that yet.

In the current state, it could be dangerous, if you don't know what you're doing, I agree.
However, there are times where you want to take over the management completely yourself and bypass FMX, then it isn't an issue.

Anyway, if you want to take this risk on your own, @paulocesarbot could send the example he did a few days ago drawing with the GPU...

Anyway, if you want to take this risk on your own, @paulocesarbot could send the example he did a few days ago drawing with the GPU...

Fix: Not quite ready yet.
We need 1 week, we are planning a big update for this week, after that we will send the GPU sample.

Added demos of using the GPU with Skia in Delphi internally in the canvas for firemonkey.


[#105] new tf-idf which is trained on the 45k precedents

new tf-idf which is trained on the 45k precedents

Please comment out the code i told you in this PR

we shouldn't remove the try and catch because it's a safeguard when running on a machine which doesn't have HDBSCAN prereqs.  it's currently running on VMs, so we don't to worry about it.


0.1 to 0.4 upgrade big slowdown

Hi people,
I have been using node-xml2js v0.1.14's xml2json.Parser().parseString without further options on XML files which kept getting bigger. Now these files up to ~2MB take nearly a second to translate to js.
So I thought, let's update and see if there are speed improvements. However, after I updated node-xml2js to v0.4.4, I noticed that in stead of a speed increase, I get a big slowdown were it now takes 8 to 10 seconds.
Are there some special options necessary in the 0.4 version? Or is this module just not meant for bigger XML? Something else I'm missing?
For now I'm downgrading to the much faster version 0.1, and I'll continue my search for fast converters.

If anyone is interested, easysax seems to be a damn fast parser, written in JS.
I had to fix some bugs in easysax myself and integrate it into xml2js but it sped up things pretty well.
sax x 57,812 ops/sec ±7.41% (78 runs sampled)
node-xml x 76,807 ops/sec ±1.75% (87 runs sampled)
libxmljs x 163,375 ops/sec ±2.58% (88 runs sampled)
node-expat x 201,663 ops/sec ±0.76% (84 runs sampled)
easysax x 828,169 ops/sec ±2.59% (86 runs sampled)

I feel there has to be a catch somewhere but I don't see it yet.

Seems to be streaming. Doesn't convert to object itself.
(Read: Not inline-replacable with node-xml2js)
Am I wrong?

Yeah, this is regarding to the discussion about the sax-js "backend" of xml2js and replacing it with node-expat.
I've seen a drop in CPU usage of around 50% after replacing sax-js with easysax. Much of the time is still spent on building an object of the whole XML so I'm thinking about just straight-up using the SAX parser.
We're saturating our 100Mbps proxy vms with gzipped XML files so for parsing that much of data every little speedup counts.

How does easysax work with browserify?

@csimi Does your version with the easysax backend pass the unit tests? I am in no means married to sax-js, I just want to avoid a dependency that has to be compiled.
On the other hand, if you saturate your network connection, serializing XML into an object is probably going to be inherently expensive, if you have high performance in mind a streaming solution like raw SAX or similar might indeed be preferable.

@Leonidas-from-XIV, if you replace the parser with one that does not work 100% with browserify, then I will be forced to fork it.
Can we introduce an external hook rather than outright replacing it? Side effects could be huge - If one of these replacement libs is actually async, unlike sax-js (because of eventemitter), then a lot of people will come screaming.

@tflanagan I understand your issue and will try to take it into account.
Your complaint is actually why I would be agains supporting multiple backends, because then the semantics of the library might change in unforeseen ways and some people will be surprised trying to have the same behaviour everywhere might end up an uphill battle for little benefit. I'd rather have one solid backend that works for all.

@csimi can you publish your xml2js + easysax repo?

My run on a 46k XML file:
sax x 66.98 ops/sec ±2.62% (50 runs sampled)
node-xml x 70.33 ops/sec ±2.56% (54 runs sampled)
libxmljs x 162 ops/sec ±2.46% (62 runs sampled)
node-expat x 116 ops/sec ±3.50% (59 runs sampled)
ltx x 236 ops/sec ±3.67% (60 runs sampled)
EasySax x 1,167 ops/sec ±4.62% (59 runs sampled)
Fastest is EasySax

@kyrylkov just for comparison, can you add RapidX2J to this comparison? It will illustrate using RapidXML backend.

@Redsandro It doesn't seem to compile with Node.js 5.8.0?  Does it support Node.js 4.x and 5.x?

@kyrylkov oops no idea actually. I'm running legacy (pre io.js-post-fork-merge) node.js for production reasons.

@kyrylkov Compiles on 4.x according to dev:
https://github.com/damirn/rapidx2j/issues/17

it should work with node 4.4.0:

git clone https://github.com/damirn/rapidx2j.git
npm install rapidx2j/
>
> I just tried it on mac os x w/o issues

I'll try to put something usable together during the weekend.
More benchmarks:
xml2js x 2.33 ops/sec ±27.49% (10 runs sampled)
xml2js easysax x 11.31 ops/sec ±19.67% (18 runs sampled)
rapidx2j x 28.87 ops/sec ±16.22% (30 runs sampled)
easysax x 129 ops/sec ±15.94% (30 runs sampled)
The easysax bench doesn't actually build a js object, just runs a SAX pass over my 215KiB XML test file (fastest mode, without parsing attributes, etc).
Rapidx2j is pretty fast but just how much time it takes to actually build a JS object is clearly visible.
I feel like EasySax is too fast (compared to expat for example) to be standards compliant. There has to be a catch somewhere.

how much time it takes to actually build a JS object is clearly visible.

How? I take any parser and it just takes time.
Is there a (simple) way to use EasySax for getting the JS object out of xml? I'd like to see if it can work for me. But in the docs it seems to just throw events on every element encountered (that's what I assumed was streaming, as SAX works like that too).
So far, rapidx2j seems the fastest but I cannot use it without forking, as the current module implements a custom lossy standard, changing caps and such.


README clarity sought

A fan of the project says it's not very clear from the README what problems it solves--ie the awesomeness of Tailor isn't fully conveyed. I can edit/add if someone sends bullet points to address his issue. Info that expresses "what this solves and what you can do with it," "this is easy to use because ...," and "this is different from what's already out there because ..." will be great.

@LappleApple we are working on the blog post about Tailor, maybe we can reuse the material to improve the readme. Will send you the draft once its ready.

Hey @w0rm and @vigneshshanmugam, I have time to help with the README this week. I have a template that makes README-writing super-simple and would like to follow that format. Can we do a 15/30-minute chat? Or you can just type out the answers. The template's in your Google Drive.

@LappleApple #31 Pleas take a look.


AutoAWQ Integration Script

Description
This PR introduces:

A "quantize" handler in the djl_serving.huggingface handler that uses AutoAWQ to quantize a model.
A code path in the serving partitioning scripts that allows DIY users to run quantization.

This is enabled when a user passes the --quantization awq option when running the partition script or option.quantize=awq in serving.properties

A Neo handler script that allows Neo to run quantization through Neo's expected interface.

Note on serving tensor_parallel_degree
For Llama-2-7b, the tp_degree is limited to 1,2 based on vLLM AWQ implementation. The reason is the tp_degree must satisfy
(intermediate_size / tp_degree) % group_size == 0

Where:

llama-2-7b intermediate_size = 11008
group_size = 128 which is defined in quant_config in djl_serving.huggingface.quantize()

Validation
Llama-2-7b (Working)
This feature has been tested with Llama-2-7b:
Quantization command:
docker run -it --rm \
        -v ./llama-2-7b:/opt/ml/input/data/training \
        -v ./logs:/opt/djl/logs \
        -v ./output:/opt/djl/output \
        --runtime=nvidia \
        --shm-size=12gb \
        deepjavalibrary/djl-serving:lmi-nightly partition --save-mp-checkpoint-path /opt/djl/output --skip-copy

Serving.properties:
engine=MPI
option.tensor_parallel_degree=8
option.quantize=awq

The outputted model was loaded and served with the LMI container.
Llama-2-70b (Not passing)
Quantization currently failing with
AssertionError
    model_service.invoke_handler("quantize", inputs)
  File "/tmp/djlserving/cache/djl_python/service_loader.py", line 29, in invoke_handler
    return getattr(self.module, function_name)(inputs)
  File "/tmp/djlserving/cache/djl_python/huggingface.py", line 607, in quantize
    _service.quantize(inputs.get_properties())
  File "/tmp/djlserving/cache/djl_python/huggingface.py", line 555, in quantize
    awq_model.quantize(self.tokenizer, quant_config=quant_config)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/awq/models/base.py", line 186, in quantize
    self.quantizer.quantize()
  File "/usr/local/lib/python3.10/dist-packages/awq/quantize/quantizer.py", line 156, in quantize
    scales_list = [
  File "/usr/local/lib/python3.10/dist-packages/awq/quantize/quantizer.py", line 157, in <listcomp>
    self._search_best_scale(self.modules[i], **layer)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/awq/quantize/quantizer.py", line 277, in _search_best_scale
    best_scales = self._compute_best_scale(
  File "/usr/local/lib/python3.10/dist-packages/awq/quantize/quantizer.py", line 334, in _compute_best_scale
    self.pseudo_quantize_tensor(fc.weight.data)[0] / scales_view
  File "/usr/local/lib/python3.10/dist-packages/awq/quantize/quantizer.py", line 69, in pseudo_quantize_tensor
    assert torch.isnan(w).sum() == 0

Serving.properties:
engine=MPI
option.tensor_parallel_degree=8
option.quantize=awq

Update: these last few commits include:

Refactoring quantization out of HuggingFace.py and into partition.py.
envvar configuration support for partitioning, using existing functionality from trt_llm_partition.py
bugfix for loading .bin files


add vnet cargo feature

fix https://github.com/webrtc-rs/util/issues/6
vnet mod is used only by https://github.com/webrtc-rs/turn package, i will make PR

turn PR is here https://github.com/webrtc-rs/turn/pull/5

I thought #6 mentioned about mDNS crate, instead of TURN crate, won't use vnet feature.  So, we don't need to change TURN right now.


RuntimeError: Expected object of backend CPU but got backend CUDA for argument #2 'mat2'

I try to run the demo.py, but get the following errors:
load model start
load model finished
start opencv
opencv finished
Traceback (most recent call last):
  File "demo.py", line 54, in <module>
    hand_verts, j3d_recon = mano(pose0, shape.float())
  File "/home/zhy/anaconda3/envs/minimal-hand-torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/zhy/anaconda3/envs/minimal-hand-torch/lib/python3.7/site-packages/manopth/manolayer.py", line 188, in forward
    self.th_posedirs, th_pose_map.transpose(0, 1)).permute(2, 0, 1)
RuntimeError: Expected object of backend CPU but got backend CUDA for argument #2 'mat2'

How to fix this bug?

Please refer to https://github.com/MengHao666/Minimal-Hand-pytorch/issues/11 for more information.


Operator DNSPacketDPDKSource documentation

As part of the product the network toolkit will be available in the toolkit directory of the product install directory. Therefore it is not necessary to mention in the documentation of the DNSPacketDPDKSource operator that the network toolkit should be retrieved from github. The following sentence should be deleted:
"If you don't already have the Streams network toolkit, get it by typing these commands at a prompt:
bash> cd $HOME/git
bash> git clone git@github.com:ejpring/streamsx.network.git"
Further references to github should also be removed.
Additionally some sections seem to be incomplete. They should be filled:
"# bind 'Mellanox' ethernet device to DPDK
To be done ..."    and
"# unbind 'Mellanox' ethernet device from DPDK
To be done ..."

DNSPacketDPDKSource.xml looks fine in 3.0.3


tiny inconsistemcy with hyphen / en-dash

@michaelnmmeyer
hyphens in page runs get converted to en-dash but not hyphens in year ranges. can the conversion be implemented also for the latter?

Done in https://github.com/michaelnmmeyer/dharma/commit/febafe2017b058da18f3d735dd1ee60044ab941d.


Compose Storybooks

https://storybook.js.org/docs/react/workflows/storybook-composition

We are linking to storybooks which suffice for now.


Git action events (e.g. clone and push) appear in the audit logs

Summary
We are adding Git CLI activities to the audit log on Github Enterprise Cloud. This data will sit alongside events captured through the web interface to provide enterprise customers with a vastly expanded data set for security and compliance audits.
Intended Outcome
Customers will be able to more fully observe UI and CLI activity on their account through the audit log. This will help customers to better meet administration, compliance, and security response needs.
How will it work?
Two new events, git.clone and git.push, will be incorporated alongside existing audit log events and available for search, export, and access via the API. The git.clone event will be recorded on clone, fetch, and pull.

https://github.blog/changelog/2020-12-10-audit-log-git-events-and-rest-api-now-available-in-limited-public-beta/


Disallowing configure AM with the v1 api

AM V1 API is deprecated and removed.

https://github.com/prometheus/alertmanager/pull/2970

This PR is just not allowing to configure AM with the V1 api anymore.

Thanks for the PR!
I'm not sure about the compatibility policy regarding AM but I don't think we should just stop supporting v1 because newer AM no longer ship it.
Maybe we can start by marking v1 as deprecated and remove it in future versions (maybe Prometheus 3.0?)

I think it's safer to wait until v3, we can continue living with it.
if you want to create an issue I can add it to the milestone.
In the meantime, maybe we could add some deprecation warning logs (I don't know if we're required to do so).

The API stability page says: Some features, which are cosmetic, still under development, or depend on 3rd party services, are not covered by this.
I therefore think we can get rid of the V1 alertmanager alerts format.

Thanks, I forgot about that page, but it's so vague that I bet we can identify a "3rd party service" for every user for every Prometheus feature :)

Yeah, would be cool to get this into v3 to avoid any discussion if this needs a major version bump or not.
I'll add this to the project and mark it as a "breaking" change, but @jan--f please dismiss that if it seems inappropriate with your v3 hat on.

[this is a copy-paste message on all unfinished breaking change items]
What is the status of this item?  We are beginning to reach the point where breaking changes may have to miss 3.0 unless they are wrapped up soon.

@alanprot any update?

since this is mostly good to go other than the last comments, if @alanprot isn't able to get to this I can just adopt it and apply the notes as a second commit.

Oh.. i missed that one.. i will try to address the comments later today, but if u wanna wrap up faster and create a new PR with the change im fine as well! :D

🤞 haha :D


4 arguments register may be a perf. issue for 64 bits type

How about to merge temporary registers to arguments registers?

I believe there needs to be at least two temporaries that are not part of the argument registers for some linkage conventions to work - deferring to others on this issue.

Get it. I realize that the point here is x0-x4 take away 5 GPR away, so 4 arguments registers sounds reasonable. Thanks.


azure.: multiple ResourceGroup instances are not getting in Ready state within reasonable timeframe

What happened?
Encountered during work on upbound/official-providers#501 and upbound/official-providers#508
When multiple examples are executed in parallel and ResourceGroups are involved, they are not getting ready and uptest fails
   logger.go:42: 13:45:22 | case/0-apply | error: timed out waiting for the condition on resourcegroups/example-containerservice-op-th45fgdy
    logger.go:42: 13:45:22 | case/0-apply | command failure, skipping 71 additional commands
    logger.go:42: 13:45:23 | case/0-apply | test step failed 0-apply
    case.go:361: failed in step 0-apply

How can we reproduce it?
Run uptest with multiple azure examples like
/test-examples="provider-azure/examples/authorization,provider-azure/examples/cache,provider-azure/examples/containerservice,provider-azure/examples/cosmosdb,provider-azure/examples/dbformariadb,provider-azure/examples/dbforpostgresql,provider-azure/examples/devices,provider-azure/examples/eventhub,provider-azure/examples/insights,provider-azure/examples/keyvault,provider-azure/examples/network/loadbalancer.yaml,provider-azure/examples/network/networkinterface.yaml,provider-azure/examples/operationalinsights,provider-azure/examples/sql,provider-azure/examples/storage/blob.yaml"

What environment did it happen in?

Crossplane Version: universal-crossplane-1.9.0-up.3
Provider Name: azure
Provider Version: 4542bf42d9911fb4377d33d8859de226d2046f26

While working on this issue, I ran into an error with the deletion of the ResourceGroup resource.
After creating dependent resources to ResourceGroup I am facing this error while deleting ResourceGroup even though I deleted the dependent resources.
NAME                                                           READY   SYNCED   EXTERNAL-NAME                   AGE
resourcegroup.azure.upbound.io/uptest-resourcegroup-example    False   False    uptest-resourcegroup-example    12h

The error message I encountered in describe

    Message:               delete failed: cannot destroy: destroy failed:
    Reason:                ReconcileError


harbor 1.2.0rc1 - reboot, services do not all restart

If you are reporting a problem, please make sure the following information are provided:
1)Version of docker engine and docker-compose.

rhel 7.2
Docker version 17.06.1-ce, build 874a737
docker-compose version 1.16.0rc1, build 9bea70a
harbor 1.2.0rc1

2)Config files of harbor, you can get them by packaging "harbor.cfg" and files in the same directory, including subdirectory.
located at: https://gist.github.com/8f6be32e626e6e59c4a430eb98f492f3
3)Log files, you can get them by package the /var/log/harbor/ .
located at: https://gist.github.com/828d362973e1b791c0c8f8512b08fbb9
On rebooting my VM,   all harbor related services do not restart.
Before reboot

6df9b7e31d90        vmware/harbor-jobservice:v1.2.0-rc1    "/harbor/harbor_jo..."      35 minutes ago      Up 35 minutes       <IP_ADDRESS>:80->80/tcp, <IP_ADDRESS>:443->443/tcp, <IP_ADDRESS>:4443->4443/tcp   nginx
487653f039c5        vmware/harbor-ui:v1.2.0-rc1            "/harbor/harbor_ui"      35 minutes ago      Up 35 minutes                                                                          harbor-ui
3a4ea7f6c424        vmware/registry:2.6.2-photon           "/entrypoint.sh se..."   35 minutes ago      Up 35 minutes       5000/tcp                                                           registry
4093efd01eba        vmware/harbor-db:v1.2.0-rc1            "docker-entrypoint..."   35 minutes ago      Up 35 minutes       3306/tcp                                                           harbor-db
ce404c06984d        vmware/harbor-adminserver:v1.2.0-rc1   "/harbor/harbor_ad..."   35 minutes ago      Up 35 minutes                                                                          harbor-adminserver
9e7368d54f4d        vmware/harbor-log:v1.2.0-rc1           "/bin/sh -c 'crond..."   35 minutes ago      Up 35 minutes       <IP_ADDRESS>:1514->514/tcp                                            harbor-log

After Reboot

6df9b7e31d90        vmware/harbor-jobservice:v1.2.0-rc1    "/harbor/harbor_jo..."   37 minutes ago      Up 36 seconds                                                                                harbor-jobservice
3ae85b1b455f        vmware/nginx-photon:1.11.13            "nginx -g 'daemon ..."   37 minutes ago      Up 37 seconds                   <IP_ADDRESS>:80->80/tcp, <IP_ADDRESS>:443->443/tcp, <IP_ADDRESS>:4443->4443/tcp   nginx
487653f039c5        vmware/harbor-ui:v1.2.0-rc1            "/harbor/harbor_ui"      37 minutes ago      Up 36 seconds                                                                                harbor-ui
3a4ea7f6c424        vmware/registry:2.6.2-photon           "/entrypoint.sh se..."   37 minutes ago      Up 37 seconds                   5000/tcp                                                           registry
4093efd01eba        vmware/harbor-db:v1.2.0-rc1            "docker-entrypoint..."   37 minutes ago      Up 37 seconds                   3306/tcp                                                           harbor-db
ce404c06984d        vmware/harbor-adminserver:v1.2.0-rc1   "/harbor/harbor_ad..."   37 minutes ago      Exited (2) About a minute ago                                                                      harbor-adminserver
9e7368d54f4d        vmware/harbor-log:v1.2.0-rc1           "/bin/sh -c 'crond..."   37 minutes ago      Up 37 seconds                   <IP_ADDRESS>:1514->514/tcp                                            harbor-log

NOTE: This is true with the clair and/or notary installs as well

Could you provide the log files under /var/log/harbor? if the service fails to start there should be some error messages in the latest log files.

should be in the gist log above.  Either way  I will recapture tonight

Log files
adminserver.log.txt
jobservice.log.txt
mysql.log.txt
proxy.log.txt
registry.log.txt
ui.log.txt

upgraded docker-compose to 1.16.0rc1, no help.  updated initial bug report

it's because that docker will boot all the no-stopped container but suppose not according to the sequence defined in the docker-compose yml files at the system start up phase.
If you want to ensure Harbor works, you probably need to add some logic in the systemd to control the lifecycle.

Not to be blunt, but shouldn't this already be baked into harbor? from https://github.com/vmware/harbor "An enterprise-class container registry server...".   From that perspective, shouldn't harbor be able to recover from and outage, manual reboot, other?
second it fails in such a bad way, not even a docker-compose down; docker-compose up -d can
get it back  on it's feet.  In fact the only recovery I have right now is docker-compose down.  then
rerun sudo ./install [ my options ]

OK i've been looking at harbor 1.2.0rc2
Basic install is much better at recovering from a reboot.  All services
still do not restart.  but at least it seems I can do a docker-compose start
and they successfully restart without errors.  Although it would be extremely
nice if they just all restarted.
Clair and Notary services differ at restarting.  Starting to research clair first

So I'm definitely happier with harbor 1.2.0rc2,   the docker-compose will start everything without error
But I find it ironic that the docker-compose files say always restart, yet they won't on a reboot or powerhit

There is no update on installation and deployment in v1.2.0-rc2.
Harbor doesn't provide any solution for system reboot/outage so far. But for this, you can definitely use docker-compose down/up to resolve it(If not, please open an issue). No need to perform the install.sh here as all the docker images has already loaded into local system.

The behavior over OS restart is beyond the scope of this project, closing.


how to display the date  from db in gigo grid

Check https://gijgo.com/grid/configuration/column.format out.


Problem compiling/linking on an ESP32-C3

I've looked long and hard for any other related issues to no avail.  Hopefully someone may spot the blindingly obvious problem I've missed for the past few hours . . .
System:
Home Assistant 2023.2.5 Supervisor 2023.01.1 Operating System 9.5 Frontend 20230202.0 - latest (been running for around a year on an HP T530 thin client with 4Gb RAM and a 32Gb SSD).
Latest version of SmartIR (as of 16th February 2023)
Trying to install on an ESP32-C3, using a direct USB connection to the host that is running HAOS.
YAML file:
`esphome:
name: esp32c3-sensor1
esp32:
board: esp32-c3-devkitm-1
framework:
type: arduino
Enable logging
logger:
Enable Home Assistant API
api:
services:
- service: send_raw_command
variables:
command: int[]
then:
- remote_transmitter.transmit_raw:
code: !lambda 'return command;'
encryption:
key: ""redacted""
ota:
password: ""redacted""
wifi:
ssid: !secret wifi_ssid
password: !secret wifi_password
Enable fallback hotspot (captive portal) in case wifi connection fails
ap:
ssid: "Esp32C3-Sensor1 Fallback Hotspot"
password: ""redacted""
Setup IR transmitter
remote_transmitter:
pin: GPIO4
carrier_duty_percent: 50%
captive_portal:
`
ESPHome fails to compile.  Spent several hours going through everything I can think of.  I've cleaned out old build files, got rid of anything else in ESPHome.  Issue seems to be specific to the API.  If I comment out the API services (except for the encryption key) then it compiles, links and uploads OK.  Debug log below:
`INFO Generating C++ source...
INFO Compiling app...
Processing esp32c3-sensor1 (board: esp32-c3-devkitm-1; framework: arduino; platform: platformio/espressif32 @ 5.2.0)
HARDWARE: ESP32C3 160MHz, 320KB RAM, 4MB Flash

toolchain-riscv32-esp @ 8.4.0+2021r2-patch3
LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
Dependency Graph
|-- AsyncTCP-esphome @ 1.2.2
|-- WiFi @ 2.0.0
|-- FS @ 2.0.0
|-- Update @ 2.0.0
|-- ESPAsyncWebServer-esphome @ 2.1.0
|   |-- AsyncTCP-esphome @ 1.2.2
|-- DNSServer @ 2.0.0
|-- ESPmDNS @ 2.0.0
|-- noise-c @ 0.1.4
|   |-- libsodium @ 1.10018.1
Linking /data/esp32c3-sensor1/.pioenvs/esp32c3-sensor1/firmware.elf
/data/cache/platformio/packages/toolchain-riscv32-esp/bin/../lib/gcc/riscv32-esp-elf/8.4.0/../../../../riscv32-esp-elf/bin/ld: /data/esp32c3-sensor1/.pioenvs/esp32c3-sensor1/src/main.cpp.o: in function esphome::api::ListEntitiesServicesArgument::ListEntitiesServicesArgument()': /config/esphome/.esphome/build/esp32c3-sensor1/src/esphome/components/api/api_pb2.h:800: undefined reference to esphome::api::enums::ServiceArgType esphome::api::to_service_arg_type<std::vector<long, std::allocator > >()'
collect2: error: ld returned 1 exit status
*** [/data/esp32c3-sensor1/.pioenvs/esp32c3-sensor1/firmware.elf] Error 1
========================= [FAILED] Took 17.19 seconds =========================`

Not sure what's going on, TBH.  I can't see any obvious reason for the problem.  The YAML checks out OK, just consistently throws an error when compiling.

I've worked around this.  Hard coding the Toshiba AC IR codes I need to use into the ESPHome yaml works OK (I just cut and pasted these from the 7260.json codes file).   I'm not using SmartIR now, just sending the IR commands directly and using a thermostat integration, plus a manual override button integration.
Not at all sure why there was a compile issue with SmartIR, but one positive comment I can give that may help others is that the codes in the 7260.json file work just fine with the Toshiba RAS-B10BKVG-E unit.


Multiple processors in single program

Is it possible somehow to analyze different parts of a program using different processors?
For example, x86 BIOS code may contain 16-bit real mode, 32-bit protected mode and 64-bit long mode instructions in one binary image. Can I markup different kinds of "segments"?
The same for CLI mixed mode assemblies, which contains x86 code and CLI code

Mixed x86 modes are possible.  The artificial context registers addrsize and opsize can be used to set an area in one mode or another.
addrsize=(0,1)        # =0  16-bit addressing  =1  32-bit addressing   =2 64-bit addressing
opsize=(2,3)          # =0  16-bit operands    =1  32-bit operands     =2 64-bit operands

Using a 32-bit processor spec you would only be able to get 16/32 mode.  As long as there is really only one segment in 16-bit mode, then you will most likely get the behavior you need.
It had been discussed to add actions for setting each of these modes, similar to the ARM/Thumb disassembly.  However there can be ramifications to mixing what should be segmented memory into 32/64-bit memory, so it wasn't added so as not to encourage this mixing leading to potentially unknown odd behavior.
CLI and x86 mixed is an issue, since they are implemented with two different processor definitions.
This has also been discussed, but there are assumptions in many places that there is only one language, compiler spec, and other specifications.
One possibility, once the CLI has matured, is to include the CLI spec into the x86 processor module.
One way to handle mixed mode, for the mentioned cases, is to create multiple programs with external references to another portion of the program using a different processor when necessary.

A similar enhancement request has been created for this (#510), so I'll close out this question.

Thanks for your reply!

@emteere sorry for the necrobump, but you mention it can be done. Do you think you could elaborate on exactly how this is supposed to be done, or point to some resources on the matter?
My use case is the 'BIOS Binary' entry in amd/ryzen's PSP firmware table; its the first bit of x86 code ran on such a system, and is comprised of a zlib-compressed UEFI filesystem trailed by some 'real' (whether or not real mode or what is still a mystery to me, trying to piece it together when all the docs are behind NDA) x86 code, with the last 0x10 bytes containing a jmp instruction backwards (encoded as 0xe9 0x63 0xe8)

To flip modes you need to set the following context registers across the area of memory that you want to disassemble in a different mode, say 16-bit in a 32-bit program.  There can be issues as mentioned above, due to the underlying memory model.
addrsize=(0,1)        # =0  16-bit addressing  =1  32-bit addressing   =2 64-bit addressing
opsize=(2,3)          # =0  16-bit operands    =1  32-bit operands     =2 64-bit operands
16-bit set - addrsize=0, opsize=0
32-bit set - addrsize=1, opsize=1
64-bit, set - addrsize=2, opsize=2
To set them across an area of memory, select the area of memory, and use SetRegisterValues... from the listing popup menu.
As long as there isn't real 16-bit segment addressing going on with multiple segments, you can probably get away with a 32-bit program and set the 16-bit addrsize/opsize across the area that is 16-bit.

hrm. Won't let me set addr/op size to 0


Build failed with M1 Mac

Build fails on an M1 MacBook. Looks like ld in the binutils package doesn't like aarch64. I guess a linux arm64 would have the same issue (raspberry pi 4 for example).
bash-3.2$ ./build-all.sh
Installing latest environment status
HEAD is now at 4459dab Merge pull request #102 from uyjulian/buildsystem_improve
Already on 'master'
Your branch is up to date with 'origin/master'.
Installing latest environment status
HEAD is now at 484a8dd Merge pull request #12 from uyjulian/buildsystem_improve
Already on 'main'
Your branch is up to date with 'origin/main'.
  Stop.
../scripts/001-binutils.sh: Failed.
../scripts/001-dvp.sh: Failed.
../scripts/001-toolchain.sh: Failed.

also with noting that gnu sed is being used:
ian@Ians-MBP ps2dev % sed --version
sed (GNU sed) 4.9

TL;DR: ARM64 target for Darwin (macOS) is not really supported by binutils and GCC used in ps2sdk toolchains.

Looks like the main issue preventing binutils-dvp from being built is that configure scripts used in binutils have no idea what arm64 is and detect host as -apple-darwin23.1.0, breaking everything.
You can "fix" this by manually overriding host detection, adding this after line 37 of ps2toolchain-dvp/scripts/001-binutils.sh:
if [ $OSVER == Darwin ] && [ $(uname -p) == arm ]; then
   TARG_XTRA_OPTS="--host=arm-apple-darwin"
fi

ld will still fail to build, but AFAIK ld from GNU binutils has never supported macOS in the first place. That doesn't prevent the binaries from working, anyway.

To build GCC for IOP toolchain, I had to add ./contrib/download_prerequisites to ps2toolchain-iop/scripts/002-gcc-stage1.sh before line 58, otherwise it always complains about not having GMP, MPFR and MPC (despite being already installed via Homebrew):
configure: error: Building GCC requires GMP 4.2+, MPFR 2.3.1+ and MPC 0.8.0+.
You'll also have to disable precompiled headers by commenting out a few lines, see this:
https://github.com/riscv-software-src/homebrew-riscv/issues/47#issuecomment-772306695
Don't forget to remove git fetch/reset/checkout from the script (lines 27-30) to keep your changes from being reset every time you launch it.

To build binutils for EE toolchain, I had to add --with-libgmp-prefix=/opt/homebrew/Cellar/gmp/6.3.0 to configure at line 60 of ps2toolchain-ee/scripts/001-binutils.sh, otherwise it couldn't find GMP to configure and build gdb. It still complains about other libraries, but at least they are not required.
Workarounds for building IOP GCC are also required for building stage1 and stage2 of EE GCC.

After all of that I was able to build PS2SDK and make a working build of OPL from source.
To honest, I'm not sure why GCC and binutils can't find any of the libraries (which causes half of the problems here). pkg-config is working properly on my system and resolves all names except mpc for some reason.

@pcm720 You are truly a wonderful human. I don't know you, but you made my life a little better. Thank you. I hope you understand the extent to which I appreciate what you've done here.

Could be nice to update scripts for allowing M1 machines…

This was already solved, please re-open if it is not working for you


added new connectors

This adds connectors to the karaf deployment to read queues for both fits and hypercube.

Not related to the webservice, just another way of making it all
available.  I've figured a way to use the same port as the other
microservices, so the PR has been updated.
On Tue, Jul 2, 2019 at 6:39 AM Jared Whiklo<EMAIL_ADDRESS>wrote:

@whikloj commented on this pull request.
In defaults/main.yml
https://github.com/Islandora-Devops/ansible-role-alpaca/pull/5#discussion_r299487944
:

@@ -62,3 +67,8 @@ alpaca_blueprint_settings:
derivative_service_url: http://localhost:8000/homarus/convert
error_max_redeliveries: 5
camel_context_id: IslandoraConnectorHomarus

pid: ca.islandora.alpaca.connector.fits

in_stream: activemq:queue:islandora-connector-fits
derivative_service_url: http://localhost:8050/fits/process

Is this due to the FITS webservice as all the other microservices are
running on port 8000 without clashing. Do we need to add a new virtualhost
to the Apache config perhaps?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Islandora-Devops/ansible-role-alpaca/pull/5?email_source=notifications&email_token=AADY2JYSTN4FKTUXSGBMD43P5NLAXA5CNFSM4H37FEY2YY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOB5H7RZQ#discussion_r299487944,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADY2J7675GRDOJPVJUN333P5NLAXANCNFSM4H37FEYQ
.

--
Alan Stanley
Developer and Training Specialist
Agile Humanities


Google virtual keyboard element image broken for Arabic

Can you put the image directly instead of using screenshot.googleplex.com? Because screenshot.googleplex.com is only visible to Googlers, but non-Googlers also look at issues.
I often use screenshot.googleplex.com during code reviews with Googlers because it's convenient and usually only the reviewer and the author reads code reviews. But we should not use it for issue description.

i see! Uploaded image.

Are we sure this is a Person Finder issue rather than a bug with the keyboard itself? Does this image come from Person Finder code, or is it otherwise affect by PF code?

I don't know. It looks likely an issue of the keyboard itself. It might be good to confirm that it is not caused by our usage of the keyboard, but I'm personally also OK to simply close the bug.

I think I've confirmed the issue is with the keyboard itself: it is pointing to a reasonable-looking URL (http://www.google.com/webelements/images/gwe_transparent_130x20.png), there just isn't a file there anymore. Since the Virtual Keyboard was deprecated in 2014, it's not surprising that files may have moved or been deleted without anyone updating the keyboard code.


Update cases in samples and tests

API change check
API changes are not detected in this pull request.

Question: In the first sentence of the Examples section, a link to a repo containing the hotel data set is given (which should maybe be updated to the hotels folder directly). I can only assume that this is the referred to dataset. It's still using TitleCase names instead of camelCase.
Not sure if that would be an issue or not. Will the examples still work with that dataset with the switch to camelCase?

Good catch. Unfortunately we cannot modify the data because it is shared across all languages.
Well, it should still work.


Writing unfinalized blocks to static files

What to do with the unfinalized blocks (from finalized to head) that can be reorged?
Currently, we write unfinalized blocks to the database, and on reorg we remove the data from the database. Eventually, we need to store headers, transactions and receipts in static files.
There are a couple of options.
Write to database, retire to static files when finalized.
Pros:

Data written to static files can't be changed, so there's no need for reverts from the static files.
No issues with inconsistent view of the read transactions that access both database and static files.

Cons:

Write amplification.

We need to write to the database first, then remove from it and write to static files.

Write to static files directly.
Pros:

No database to static files retirement logic (simpler).
Lower write amplification due to writing to the database first, then removing from it and writing to static files.

Cons:

On reorgs, the deletion from static files needs to be aware of current readers of the database.

If there's an active read-only reader of data (e.g., RPC query), it needs to have a consistent view of both database and static files.
It means that we can actually revert the data from static files only once there are no readers, which is hard.

Persist in memory, write to static files when finalized.
Pros:

On reorgs, we don't revert any data from database or static files.

So, once data is written to database or static files, we're sure it won't be reverted.

Cons:

On restart, you need to re-sync blocks from finalized to head.

It will significantly slow down the restart which harms debugging, validators, and overall UX.

Stateroot calculation becomes hard, because we currently hold intermediate nodes in db. We could hold the diffs in memory, but not implemented yet

We will write to static files directly.