Disable zoom on scroll

Hey,
I would like to have the mouse scrolling trigger panning rather than zooming. Is there a way to disable the zooming on scroll?

Hey @emielwit
good idea! The new option zoomOnScroll is released in v4.1.0. You can find an example here https://react-flow.netlify.app/interaction


Compress intermediate results

We should reduce the size of intermediate data (raw) without sacrificing too much performance. Good candidate might be using parquet and snappy.

Resolved by #336 which indeed uses parquet and snappy.


Implement user event import from Bigquery

Rewrite the existing Python Google Cloud function as a Ruby service class. This receives an event type and a date, and makes the necessary request to Discovery Engine to import BigQuery user event data.

Add DiscoveryEngine::UserEvents::Import service to implement importing of user events from BigQuery along the lines of the original Python code
Add GOOGLE_CLOUD_PROJECT_ID app configuration
Add a rake task to call the service
Ensure tests require Google API namespace for stubbing

see the vertex_events_push function on the infrastructure repo: https://github.com/alphagov/search-v2-infrastructure/blob/2991588b5dae11a20fc80432393fd134c7acb53c/terraform/environment/files/vertex_events_push/main.py

Is the idea that each day we'll run the rake task for the intraday events multiple times a day (using rake user_events:import_intraday_events) then once per day we'll import the previous day's events (using rake user_events:import_yesterdays_events)?

Spot on – I've already drafted scheduled tasks for this in integration: https://github.com/alphagov/govuk-helm-charts/pull/2803/files
And then there is a third Rake task for a specific date which is something that has occasionally come up as a manual task that needs doing (for example, if the import fails for some reason). This way someone can manually execute the Rake task in an environment.


Question about the visualization for VIT?

Hi, thank you very much to opensource such a wonderful work!
I have a question for the visualization of VIT.
In the notebook for VIT.
You use the rule 6 to calculate the relevence. But as you mentioned in the paper, "Since Eq. 5 uses gradients to average across heads, the values of R_qq are typically small due to the multiplication with the gradients. We wish to account equally both for the fact that each token inﬂuences itself and for the contextualization by the self-attention mechanism. Therefore, we normalize each row in R qqso that it sums to 1."
So, why you do not normalize each row for the visualization of VIT?

Hi @zhaoxin94 thanks for your interest in our work!
What you are referring to is a normalization we use for the multi modal update rule.
For pure self attention architectures, this issue of no normalization will indeed cause the values in the diagonal to be bigger than those outside it, but for self attention, we take the row corresponding to the CLS token, and disregard the CLS score itself.
So basically, the values in the diagonal do not influence the visualization in the self attention case, and regularization has little to no impact.
This question was raised by the reviewers as well, so we ran experiments of normalizing Eq. 6  and found results to be very similar with a marginal difference.
I hope this answers your question, please let me know if you require any further clarifications.
Best,
Hila.

@zhaoxin94 closing this issue due to inactivity. Please reopen if necessary.


KML string validation with extendedData

Describe the bug
When creating a KML String with KML.writeFeatures(), with extendedData, the format isn't validated correctly.
To Reproduce
Steps to reproduce the behavior:

Create an array of features, e.g. Line with style and custom property 'lineDash'.
Write a KML string with:
featString = new KML({ extractStyles: true, defaultStyle: [kmlStyle], }).writeFeatures(exportFeatures);

Output:
< kml
xmlns="http://www.opengis.net/kml/2.2" xmlns:gx="http://www.google.com/kml/ext/2.2"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.opengis.net/kml/2.2
https://developers.google.com/kml/schema/kml22gx.xsd">
< Document>
< name>Editor< /name>
< Placemark>
< ExtendedData>
< Data name="lineDash">< value>1,10< /Data>
< /ExtendedData>
< Style>
< LineStyle>< color>ff0000ff< /color>< width>4< /width>< /LineStyle>

< LineString>
< coordinates>
7.969207763671874,46.941824493511206 8.666839599609375,47.246610279847005
8.735504150390627,47.039246258977784
< /coordinates>
< /LineString>
< /Placemark>
< /Document>
< /kml>

Validate the KML string against the KML schema kml22gx.xsd fails:
https://developers.google.com/kml/schema/kml22gx.xsd

Expected behavior
The tag < ExtendedData > should be located after the

Pull request welcome.

@ahocevar here is the Pull request: https://github.com/openlayers/openlayers/pull/9500


build: rename core package to loki

Coverage remained the same at 94.709% when pulling a01c4bc7aee12f9951366c1182d3f23225bf4b82 on update_build into 6e481a08b9c0e0361c22de978c3aebbc94356f5a on master.

Coverage remained the same at 94.709% when pulling a01c4bc7aee12f9951366c1182d3f23225bf4b82 on update_build into 6e481a08b9c0e0361c22de978c3aebbc94356f5a on master.


Price Range Facet No Longer Works After Update to 3.2.5

Issue Summary
After updating Django Oscar to version 3.2.5, the price range facet in search results stopped working correctly and shows "no matches" for all queries. This issue seems to stem from a discrepancy between the facet field name expected by Oscar and the field name generated by Haystack/Solr.
Here's the relevant configuration from OSCAR_SEARCH_FACETS:
OSCAR_SEARCH_FACETS = {
    "fields": OrderedDict(
        [
            ("product_class", {"name": _("Type"), "field": "product_class"}),
            ("rating", {"name": _("Rating"), "field": "rating"}),
        ]
    ),
    "queries": OrderedDict(
        [
            (
                "price_range",
                {
                    "name": _("Price range"),
                    "field": "price",
                    "queries": [
                        (_("0 to 250"), "[0 TO 250]"),
                        (_("250 to 500"), "[250 TO 500]"),
                        (_("500 to 1000"), "[500 TO 1000]"),
                        (_("1000 to 2000"), "[1000 TO 2000]"),
                        (_("2000+"), "[2000 TO *]"),
                    ],
                },
            ),
        ]
    ),
}

Upon investigating, I found that Haystack always generates a price_exact field for faceting, while Oscar expects the field to be price. When I switch to price_exact in OSCAR_SEARCH_FACETS, the reverse issue occurs — now the queries return results for price, but the context data preparation breaks in the final stage.
Example facet response:
{'start_offset': 0, 'result_class': <class 'haystack.models.SearchResult'>, 'end_offset': 1, 'facets': {'product_class': {}, 'rating_exact': {}}, 'narrow_queries': {'price:([0 TO 250])'}, 'query_facets': [('price_exact', '[0 TO 250]'), ('price_exact', '[250 TO 500]'), ('price_exact', '[500 TO 1000]'), ('price_exact', '[1000 TO 2000]'), ('price_exact', '[2000 TO *]')]}

Example Solr request:
/solr/pshop-en/select/?q=(is_public:(true)+AND+structure:("standalone"+OR+"parent"))&fl=*+score&df=text&start=0&rows=1&spellcheck=true&spellcheck.collate=true&spellcheck.count=1&facet=on&facet.field=product_class&facet.field=rating_exact&facet.query=price_exact:[0+TO+250]&facet.query=price_exact:[250+TO+500]&facet.query=price_exact:[500+TO+1000]&facet.query=price_exact:[1000+TO+2000]&facet.query=price_exact:[2000+TO+*]&fq=django_ct:(catalogue.product)&fq=price:([0+TO+250])&wt=json

Debugging
I believe the issue is related to how the FacetMunger class processes facet queries. Specifically, the mismatch between the price field in Oscar's search facets and the price_exact field generated by Haystack/Solr causes the final stage of context data preparation to fail.
Snippet from FacetMunger:
def munge_query_facet(self, key, facet, clean_data):
    clean_data[key] = {"name": facet["name"], "results": []}
    # Loop over the queries in OSCAR_SEARCH_FACETS rather than the returned
    # facet information from the search backend.
    for field_value, query in facet["queries"]:
        field_name = facet["field"]
        is_faceted_already = field_name in self.selected_facets

        match = "%s:%s" % (field_name, query)
        if match not in self.facet_counts["queries"]: # Always not match because price_exact instead of price!!!
            # This query was not returned
            datum = {
                "name": field_value,
                "count": 0,
                "show_count": True,
                "disabled": True,
            }
        else:
            count = self.facet_counts["queries"][match]
            datum = {
                "name": field_value,
                "count": count,
                "show_count": not is_faceted_already,
                "disabled": count == 0 and not is_faceted_already,
                "selected": False,
            }
            if query in self.selected_facets.get(field_name, []):
                # Selected
                datum["selected"] = True
                datum["show_count"] = True
                url = self.base_url.remove_query_param("selected_facets", match)
                datum["deselect_url"] = self.strip_pagination(url)
            else:
                url = self.base_url.append_query_param("selected_facets", match)
                datum["select_url"] = self.strip_pagination(url)
        clean_data[key]["results"].append(datum)

Related Issue:
Even if the price range is disabled, Oscar doesn't filter out old selected_facets parameters from URLs. This can result in broken queries that trigger a Solr error:
HTTP 400: Invalid Number: [250 to 500] for field price_exact

This can happen if old URLs (e.g., from search indexing bots) still contain parameters related to previously valid facets, leading to unwanted Solr errors.
Technical details

Python version: 3.10.12.
Django version: 4.2.16.
Oscar version: 3.2.5.
Haystack version: 3.3.0.
Solr version: 9.6.

Hi!
see https://github.com/django-oscar/django-oscar/pull/4187/files
I restored field_name = "%s_exact" % facet["field"] in munge_query_facet and the price_range reappered in my shop

I sent a PR #4376 but the tests did not pass:
INTERNALERROR>     THUMBNAIL_STORAGE = settings.STORAGES['default']['BACKEND']
INTERNALERROR> AttributeError: 'Settings' object has no attribute 'STORAGES'
don't know how to fix that

fyi #4376 was merged into master.

fixed in; https://github.com/django-oscar/django-oscar/pull/4376


Lazyload and printing

lazyload works great, when scrolling down the images are beeing loaded.
but when i print the page i only will see the "placeholder" no image is beeing loaded. can this somehow be achieved?
i have tried to add an onbeforeprint-listener. i also could call trigger('appear')
window.onbeforeprint = function () {
    $("img.lazy").each(function() { $(this).trigger("appear"); console.log(1)});
}

i can see all the images loaded, but when i then print in firefox the images  are hidden again - i think firefox "reloads" the page from server. i also tried the noscript but this does not work. can i do it other then
<img class="lazy" src="/_standards/leer.gif" data-original="<%firstPhotoThumb%>" width="120" height="120">
<img class="print-only" src="<%firstPhotoThumb%>" width="120" height="120">

Hi. Just wondered if anyone had found a solution on this as I'm having the same issue in that the printed doc displays loaded images ok but placeholders for the unloaded lazyload ones. Just wondered if there was an extra plug-in perhaps (I'm using Lazy Load 1.9.3).
Thanks in advance

Hello! I was wondering if any proper solution was found when exploring around lazyload and printing.
Thanks!

Hello!
I'm the author of the vanilla-lazyload script and I've recently developed a feature that makes print of all images possible!
Tested cross browser using this repo code which is live here.
Take a look and let me know what you think!
I'm open to pull requests on GitHub of course.

I use the same onbeforeprint solution, it's buggy in Firefox. I think the event fires before the print dialog but after it's already too late.  Either way, all I get are the alt's in the print preview.
Relying on a print button is a lousy workaround.  It's a shame.


Your examples no longer work

Summary:
Your examples on codepen no longer work.

@Keltin42 I've just testes the examples and they are all working.
I'll close for now, but please, comment if you can investigate a little further on why it's not working.


[BUG]: LLMServer Generate Error, Please CheckErrorInfo.: 'tuple' object has no attribute 'split'

Describe the bug
I am trying to run the DB-GPT with the WizardLM, but it is throwing the error as "LLMServer Generate Error, Please CheckErrorInfo.: 'tuple' object has no attribute 'split'". This error occurs when I am asking any question.
OS: Amazon Linux
Resource: Single GPU, 16GB, A100

Hi, @Pradeep-Vanapalli, can you provide some log information? In addition, I recommend that you try other models, such as vicuna v1.5, to see if you have similar problems.

Please try the latest version  v0.4.3


Please consider fixing issues detected by repository checker

Notification from ioBroker Check and Service Bot
Dear adapter developer,
I'm the ioBroker Check and Service Bot. I'm an automated tool processing routine tasks for the ioBroker infrastructure. I have recently checked the repository for your adapter rct for common errors and appropiate suggestions to keep this adapter up to date.
This check is based the current head revisions (master / main  branch) of the adapter repository
Please see the result of the check below.
ioBroker.rct
   -
:thumbsup: No errors found
WARNINGS:

[ ] :eyes: [W853] .npmignore found - consider using package.json object "files" instead.

:thumbsup: No suggestions found
Please review issues reported and consider fixing them as soon as appropiate.
Warnings reported by repository checker should be reviewed. While some warnings can be ignored due to good reasons or a dedicated decision of the developer, most warnings should be fixed as soon as appropiate.
You may start a new check at any time by adding the following comment to this issue:
@iobroker-bot recheck
Please note that I (and the server at GitHub) have always plenty of work to do. So it may last up to 30 minutes until you see a reaction. I will drop a comment here as soon as I start processing.
Feel free to contact me (@iobroker-bot) if you have any questions or feel that an issue is incorrectly flagged.
And THANKS A LOT for maintaining this adapter from me and all users.
Let's work together for the best user experience.
your
ioBroker Check and Service Bot
@mcm1957 for evidence
Last update at Fri, 06 Sep 2024 10:03:50 GMT

Issue outdated due to RECREATE request. Follow up issue #166 has been created.
This issue will be closed.
your
ioBroker Check and Service Bot


[Youtube_Videos], no solution means we're robbing creators?

I want to play youtube videos on my app, however, every solution I found so far, gets rid of the ads, cards, speed and quality controls, cc, etc...
How can flutter and youtube both be owned by google and no solution for this exist? I mean, I'll still play the videos but at the expense of the livelihood of the YouTuber, and youtube itself since it takes 45% of the ad revenue...
we NEED a native youtube player on flutter. that we can customize to show what we want (for example show channel cards but not the youtube final card grid...)
I Have no idea how to create this or I would myself... but this is a must for yesterday

Duplicate of #13756


Unable to enable WSL 2: 'WSL 2 requires an update to its kernel component' after kernel update MSI installed

Environment
Win 10 - 10.0.19624.0
Windows build number: [run `[Environment]::OSVersion` for powershell, or `ver` for cmd]
Your Distribution version: [On Debian or Ubuntu run `lsb_release -r` in WSL]
Whether the issue is on WSL 2 and/or WSL 1: [run `cat /proc/version` in WSL]

Steps to reproduce
I've followed the instructions to enable WSL, VM components, reboot, run MSI. MSI installs without error, but installs quickly and without elevation prompts as mentioned in the article. Go to set WSL 2 as default and get 'WSL 2 requires an update to its kernel component. For information please visit https://aka.ms/wsl2kernel.'
Folowing: https://docs.microsoft.com/en-us/windows/wsl/install-win10#step-4---download-the-linux-kernel-update-package

WSL logs:
Expected behavior

Actual behavior

PS C:\> wsl --set-default-version 2 WSL 2 requires an update to its kernel component. For information please visit https://aka.ms/wsl2kernel For information on key differences with WSL 2 please visit https://aka.ms/wsl2

most likely, the elevation prompts is blocked by UAC.

Download the kernel to a fold like c:\temp which doesn't include space
start cmd.exe with admin permission, and goto above folder.
msiexec /i "wsl_update_x64.msi

Thanks, @licanhua but I ended up wiping the OS. All is good now.


Avoid timezone issues by using UTC dates

Refs peopledoc/support#2984

[ ] Tests
[ ] Review

Close in favor of https://github.com/novafloss/formidable-core/pull/67
Thanks you @mgu


More detailed logging during a provision

When you run a provision, we output the number of retries left and the return of oc logs -f. Once the patch  for the Kubernetes client merges, we can do a lot more.  We can parse the pod status, see if the image is pulling, make a note what node the pod is on, ect...

[2017-05-22T17:07:33.705-04:00] [INFO] Container not up yet, retrying 1 of 150 on pod aa-ee5b096d-d46b-46ff-bcef-f2b89dab8606
[2017-05-22T17:07:33.929-04:00] [DEBUG] oc log output:
Error from server: container "aa-ee5b096d-d46b-46ff-bcef-f2b89dab8606" in pod "aa-ee5b096d-d46b-46ff-bcef-f2b89dab8606" is waiting to start: ContainerCreating
[2017-05-22T17:07:33.929-04:00] [DEBUG] status: still waiting to start
[2017-05-22T17:07:39.929-04:00] [INFO] Container not up yet, retrying 2 of 150 on pod aa-ee5b096d-d46b-46ff-bcef-f2b89dab8606

This would change with https://github.com/openshift/ansible-service-broker/pull/124


PoSAccounting simulation test

Add tests that perform random operations and check storage consistency on every iteration.
Closes #568

Not sure that I understand your concern, isn't it what happens in production? Users doing random actions. Btw everything is random but undo is always for the latest action.

Random, but they're in order because of utxos.


Language editor

This PR introduces a language editor for things like keywords and subject where you can pick a language.

Looking good. Merging this.


changed app.aragonpm.test to app.aragonpm.eth`

aragonpm.eth is the correct registry deployed on the devchain. This might have just been a typo?

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Add nix support

Self-explanatory, nix support was missing.

Thanks for the PR! Could you also add nix to the supported languages list here: https://github.com/JoosepAlviste/nvim-ts-context-commentstring/blob/main/doc/nvim-ts-context-commentstring.txt#L13

I've updated the docs :+1:

Thanks for the contribution!


RDS MySQL - Column user_username and column username in FK constraints incompatible

I'm attempting to put together a Terraform script for automatic deployment of a JATOS node on EC2. I'm using an RDS database running MySQL 8.0.28. However, JATOS cannot seem to be able to get past creating the Foreign Key to connect an API Token's user_username field to the User table's username field. I get the following error in the play_evolutions table:
CREATE TABLE `ApiToken` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`tokenHash` varchar(255) NOT NULL,
`name` longtext NOT NULL,
`user_username` varchar(255) NOT NULL,
`creationDate` datetime NOT NULL,
`expires` int(11) DEFAULT NULL,
`active` tinyint(1) NOT NULL,
PRIMARY KEY (`id`)
);

ALTER TABLE `ApiToken` ADD KEY `FK_lghqbQuIHvMEqpdJHkjdQHYbe` (`user_username`);
ALTER TABLE `ApiToken` ADD CONSTRAINT `FK_lghqbQuIHvMEqpdJHkjdQHYbe` FOREIGN KEY (`user_username`) REFERENCES `User` (`username`);

Referencing column 'user_username' and referenced column 'username' in foreign key constraint 'FK_lghqbQuIHvMEqpdJHkjdQHYbe' are incompatible. [ERROR:3780, SQLSTATE:HY000]

For reference, here's what both ApiToken and User look like using SHOW CREATE TABLE ______;:
CREATE TABLE `User` (
  `username` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
  `name` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `passwordHash` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `worker_id` bigint DEFAULT NULL,
  `authMethod` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT 'DB',
  `lastLogin` datetime DEFAULT NULL,
  `active` tinyint(1) NOT NULL DEFAULT '1',
  `email` varchar(1023) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  PRIMARY KEY (`username`),
  KEY `FK_pk77d8680811astbnoae923x1` (`worker_id`),
  CONSTRAINT `FK_pk77d8680811astbnoae923x1` FOREIGN KEY (`worker_id`) REFERENCES `Worker` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci

CREATE TABLE `ApiToken` (
  `id` bigint NOT NULL AUTO_INCREMENT,
  `tokenHash` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
  `name` longtext COLLATE utf8mb4_unicode_ci NOT NULL,
  `user_username` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
  `creationDate` datetime NOT NULL,
  `expires` int DEFAULT NULL,
  `active` tinyint(1) NOT NULL,
  PRIMARY KEY (`id`),
  KEY `FK_lghqbQuIHvMEqpdJHkjdQHYbe` (`user_username`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci

As far as I can see, there doesn't appear to be any conflicts between the types of the two columns. The weird thing is, I can perform the operation myself afterwards and it goes off without a hitch:
ALTER TABLE `ApiToken` ADD CONSTRAINT `FK_lghqbQuIHvMEqpdJHkjdQHYbe` FOREIGN KEY (`user_username`) REFERENCES `User` (`username`);
Query OK, 0 rows affected (0.18 sec)
Records: 0  Duplicates: 0  Warnings: 0

ApiToken | CREATE TABLE `ApiToken` (
  `id` bigint NOT NULL AUTO_INCREMENT,
  `tokenHash` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
  `name` longtext COLLATE utf8mb4_unicode_ci NOT NULL,
  `user_username` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
  `creationDate` datetime NOT NULL,
  `expires` int DEFAULT NULL,
  `active` tinyint(1) NOT NULL,
  PRIMARY KEY (`id`),
  KEY `FK_lghqbQuIHvMEqpdJHkjdQHYbe` (`user_username`),
  CONSTRAINT `FK_lghqbQuIHvMEqpdJHkjdQHYbe` FOREIGN KEY (`user_username`) REFERENCES `User` (`username`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci

So I'm not sure why the Play evolutions aren't working, and I have no idea how to trigger them to "continue" after I have manually fixed the issue. I'm also pretty rusty with my SQL, so if there's something I'm missing in setting up the table it would be an immense help if anyone could point me in the right direction. A local database works, but I want to be able to persist the data effectively––and while I could just host the local database in the EC2 container in an EBS volume that is persisted between calls, that doesn't seem like it's the greatest solution.
For reference, here's the terraform definition of the RDS table:

Show Terraform code

# Stores data from the JATOS server
module "db" {
  source = "terraform-aws-modules/rds/aws"

  identifier = "${local.prefix}-db-mysql"

  engine               = "mysql"
  engine_version       = "8.0"
  family               = "mysql8.0"
  major_engine_version = "8.0"

  # Instance information
  instance_class    = "db.t3.micro"
  allocated_storage = 30
  storage_encrypted = true

  # Connectivity information
  db_name  = "jatos"
  username = var.db_username
  port     = 3306

  iam_database_authentication_enabled = true
  availability_zone                   = var.aws_availability_zone

  vpc_security_group_ids = [module.db_sg.security_group_id]

  maintenance_window      = "Mon:00:00-Mon:03:00"
  backup_window           = "03:00-06:00"
  backup_retention_period = 0

  subnet_ids          = data.aws_subnets.all.ids
  deletion_protection = false

  # Very important––set the character set and collation to UTF-8
  parameters = [
    {
      name  = "character_set_client"
      value = "utf8mb4"
    },
    {
      name  = "character_set_server"
      value = "utf8mb4"
    },
    {
      name  = "collation_connection"
      value = "utf8mb4_unicode_ci"
    },
    {
      name  = "collation_server"
      value = "utf8mb4_unicode_ci"
    }
  ]
}

It's probably the difference in collation and character set between table User and table ApiToken.
A bit of background: JATOS originally used the utf8 character set and collation which was a (common mistake)[https://adamhooper.medium.com/in-mysql-never-use-utf8-use-utf8mb4-11761243e434]. To make JATOS fully UTF-8 compatible all collation and character set had to be changed to utf8mb4. But to be backwards compatible that is done by JATOS itself in some kind of clandestine clean-up operation at every start of JATOS.
What likely happened in your case is when you updated to 3.8.1 and the database evolution script added user_username varchar(255) NOT NULLwithout theutf8mb4_unicode_ci, the following foreign key adding failed because the collation was different. Then JATOS continued starting up and added the utf8mb4_unicode_ci`. Now the foreign key adding works as you have shown by doing it by hand.
I'm going to fix this in 3.8.1. I just have to figure out which MySQL sequence of command to use that is also compatible with H2 and the evolution scripting. Until this is fixed I put the 3.8.1 release on hold. Argh!, this utf8 chaos in MySQL already cost me a lot of nerves :(.

Well your answer actually solved my issue––instead of trying to pre-emptively set utf8mb4 character sets and collations, I forced the MySQL instance down to utf8 and let JATOS take care of the upgrading. Now everything is working as expected, albeit not in the most beautiful solution...

New Terraform setup:
# Stores data from the JATOS server
module "db" {
  source = "terraform-aws-modules/rds/aws"

  identifier = "${local.prefix}-db-mysql"

  engine               = "mysql"
  engine_version       = "8.0"
  family               = "mysql8.0"
  major_engine_version = "8.0"

  # Instance information
  instance_class    = "db.t3.micro"
  allocated_storage = 15
  storage_encrypted = true

  # Connectivity information
  db_name  = "jatos"
  username = var.db_username
  port     = 3306

  iam_database_authentication_enabled = true
  availability_zone                   = var.aws_availability_zone

  vpc_security_group_ids = [module.db_sg.security_group_id]

  maintenance_window      = "Mon:00:00-Mon:03:00"
  backup_window           = "03:00-06:00"
  backup_retention_period = 0

  subnet_ids          = data.aws_subnets.all.ids
  deletion_protection = false

  # Very important––set the character set and collation to UTF-8
  parameters = [
    {
      name  = "character_set_client"
      value = "utf8"
    },
    {
      name  = "character_set_connection"
      value = "utf8"
    },
    {
      name  = "character_set_database"
      value = "utf8"
    },
    {
      name  = "character_set_filesystem"
      value = "utf8"
    },
    {
      name  = "character_set_results"
      value = "utf8"
    },
    {
      name  = "character_set_server"
      value = "utf8"
    }
  ]
}

Nice you got it working :). I still in the process of finding a general way that is compatible with MySQL, H2 and the framework JATOS is using.

I had to remove one line in the database evolution script for 3.8.1 to make it working. This line is now removed:
ALTER TABLE `ApiToken` ADD CONSTRAINT `FK_lghqbQuIHvMEqpdJHkjdQHYbe` FOREIGN KEY (`user_username`) REFERENCES `User` (`username`);

If you have already any running JATOS instances with 3.8.1 you might want to update them. Otherwise future updates might run into trouble.


[AutoDiff upstream] SILOptimizer tests

autodiff_indirect_diagnostics.swift from tensorflow branch pasted into differentiation_diagnostics.swift, without any modifications necessary
activity_analysis.swift from tensorflow branch copied into differentiation_activity_analysis.swift with only 2 modifications necessary: (1) import _Differentiation, (2) rename some of the symbols in the CHECKs to account for the filename change.

@swift-ci please test


Keras Mask RCNN: Fatal error: CropAndResize is not a registered function/op

I followed the Mask RCNN convert instruction from https://github.com/onnx/keras-onnx/tree/master/applications/mask_rcnn, and copied convert_tf_crop_and_resize function from test_utils, converting seems fine with warnings:
The layer <keras.layers.core.Lambda object at 0x000002DAD8574278> doesn't have a specific converter, fall back.
The layer <keras.layers.core.Lambda object at 0x000002DAD8574278> doesn't have a specific converter, fall back.
The layer <keras.layers.core.Lambda object at 0x000002DAD8574278> doesn't have a specific converter, fall back.
The layer <keras.layers.core.Lambda object at 0x000002DAD8574278> doesn't have a specific converter, fall back.
The layer <keras.layers.core.Lambda object at 0x000002DAD8574278> doesn't have a specific converter, fall back.
The node number after optimization: 1739 -> 879

But error occurs when onnxruntime inference:
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Fatal error: CropAndResize is not a registered function/op
FYI my onnxruntime version is 1.2.0 and keras2onnx is 1.7.0

(1) Make sure you pull latest keras2onnx and onnxconverter-common from master.
(2) For onnxruntime 1.2.0, you need make one line change in convert_tf_crop_and_resize, that is to add argument op_version=1 in oopb.add_node('CropAndResize',

It's not working with onnxruntime 1.2.0
(1) My keras2onnx and onnxconverter-common is installed via these commands:
pip install git+https://github.com/microsoft/onnxconverter-common
pip install git+https://github.com/onnx/keras-onnx

(2) I copied test_utils.py and added the argument :
def convert_tf_crop_and_resize(scope, operator, container):
    if operator.target_opset < 11:
        raise ValueError("CropAndResize op is not supported for opset < 11")
    oopb = OnnxOperatorBuilder(container, scope)
    node = operator.raw_operator
    mode_value = node.get_attr('method')
    transpose_node = oopb.apply_transpose(operator.inputs[0].full_name,
                                          name=operator.full_name + '_transpose_1',
                                          perm=[0, 3, 1, 2])
    cropandresize = oopb.add_node('CropAndResize',
                                  transpose_node + operator.input_full_names[1:],
                                  operator.full_name + '_crop_and_resize',
                                  op_domain='com.microsoft',
                                  op_version=1,
                                  mode=mode_value)
    oopb.apply_op_with_output("apply_transpose",
                              cropandresize,
                              operator.output_full_names,
                              name=operator.full_name + '_transpose_final',
                              perm=[0, 2, 3, 1])

However, both mask_rcnn.py and test_mask_rcnn.py works with onnxruntime 1.1.1 after I added convert_DetectionLayer,convert_BatchNorm in the later one , that's very confusing. Is there something changed with 'com.microsoft' domain in onnxruntime version 1.2.0?

Yes onnxruntime 1.2.0 has some changes on op_version for 'com.microsoft' but it is not finally settled. Maybe you can try the patch here for onnxruntime 1.2.0. This patch is an work-around so currently it is not going to check-in.

Thanks @jiafatom , the patch works! 👍
But not all ops are supported by CUDA backend, I will pay attention to update.

The patch is merged into master branch as PR. So the master branch can convert the model on onnxruntime 1.2.0.


Unexpected lint failure

We are getting a consistent lint failure/crash
2024-02-08 14:43.46: New job: Lint
2024-02-08 14:43.46: Checking out commit 2e8fbc5b. To reproduce:
                       git clone --recursive "https://github.com/ocaml/opam-repository.git" && cd "opam-repository" && git fetch origin "refs/pull/25068/head" && git reset --hard 2e8fbc5b
2024-02-08 14:43.46: Exec: "cp" "-a" "--" "/var/lib/ocurrent/var/git/opam-repository.git-a0bc4b41ddb868605dc1500002c27cfa79faf4389fe53b6754817ea16b48d458/.git"
                           "/tmp/git-checkout1f11de52"
2024-02-08 14:43.49: Exec: "git" "-C" "/tmp/git-checkout1f11de52" "submodule"
                           "deinit" "--force" "--all"
2024-02-08 14:43.49: Exec: "git" "-C" "/tmp/git-checkout1f11de52" "reset"
                           "--hard" "-q" "2e8fbc5bbaed0409e84a25b51e8da7f617c53b44"
2024-02-08 14:43.53: Exec: "git" "-c" "protocol.file.allow=always" "-C"
                           "/tmp/git-checkout1f11de52" "submodule" "update"
                           "--recursive" "--init" "--no-fetch"
2024-02-08 14:43.53: Exec: "git" "merge" "-q" "--" "d87a1eb8c3d0e6228e3f957b25aa87f49e397d03"
2024-02-08 14:44.10: Job failed: Failure("Failed to get packages with 'opam list --all'") -- Raised by primitive operation at Lwt_unix.read_bigarray.(fun) in file "src/unix/lwt_unix.cppo.ml", line 688, characters 8-59
Called from Lwt_unix.wrap_syscall.(fun) in file "src/unix/lwt_unix.cppo.ml", line 571, characters 17-28

on https://opam.ci.ocaml.org/github/ocaml/opam-repository/commit/2e8fbc5bbaed0409e84a25b51e8da7f617c53b44 (from https://github.com/ocaml/opam-repository/pull/25068)
I tried reproducing it locally with ocaml 2.1.5 and ocaml 2.2.0~beta but it works fine, and the lint passes.

It started happening on other PRs as well: https://opam.ci.ocaml.org/github/ocaml/opam-repository/commit/88c5a5527c000a148ab0f39cde4af8b5b2d4e71d/variant/(lint)

Thanks for the report, I've temporarily disabled the feature that causes the issue and I'll work on a fix. Please let us know if it persists.

Fixed with https://github.com/ocurrent/opam-repo-ci/pull/270.


Add DB seed and tests for coverart in API

Following up on PR#245, add a DataBase Seeder that uses our scrubbed
database dump
(https://github.com/LibriVox/librivox-ansible/blob/master/roles/db_import/files/librivox_catalog_scrubbed.sql.bz2), and start using it to test coverart in the API.

@garethsime @redrun45 thoughts?

I'll take a look tomorrow evening! 🙂


Project Proposal (README.md)

Rough outline:
Research question: “Will a car crash result in fatalities?”. Sub-questions:
Methodological:
“What features are relevant? Should simplify.”
“What model would perform best?”
“Which hyperparameters are best?”
“How will we deal with missing values?”
Research:
What are the characteristics that lead to a severe car crash?
Weather
Road type
Time of year
Age of driver
Vehicle type
3. Try different classification models, tune hyperparameters, pick the best performing one.

We will use .info() and .describe() for EDA tables and overlapping histograms for numeric features separated by class value.
Table of scoring metrics.

Hey Daniel, I reviewed your proposal and added a few comments (minor ones). Otherwise, all look good to me. Thank you!


[Bug]: The email box of newsletter section is not styled properly in dark mode.

Is there an existing issue for this?

[X] I have searched the existing issues

Describe the bug

The email box of newsletter section is not styled properly in dark mode.

Expected behavior

The email box of newsletter section should look dark in dark mode.

Add ScreenShots

On which device are you experiencing this bug?
No response
Record

[X] I have read the Contributing Guidelines
[X] I'm a GSSOC'24 contributor
[ ] I'm a VSOC'24 contributor
[X] I have starred the repository

@itsekta please assign this issue to me.


[shell] use nix-profile

Summary
Experimenting.
Motivation:
We can define a custom nix-profile, and install packages scoped to this profile.
In devbox shell:

previously, we would manually add the path to each package to PATH.
now, we can add the path to the nix-profile to PATH and skip manually adding each package.

The hope is that this makes it easier to add packages when within devbox shell. Currently,
users need to exit the shell and restart it for the nix package to be installed - clearly
not ideal. With a nix-profile, if we install the package using that profile, then the
package's binary should be downloaded and visible within the shell.
How was it tested?

[ ] in testdata/rust/rust-stable:

open devbox shell
verify openssl not installed via which openssl and getting /usr/bin/openssl
do devbox add openssl
do which openssl and see it point to a nix-store location

Current dependencies on/for this PR:

main

PR #188   👈

This comment was auto-generated by Graphite.

@loreto @LucilleH PTAL. Made a number of changes:

Pushed logic from CLI into devbox package.
Fixed shell.Exec() functionality
added nix.WithProfile api to handle the profileBinDir that is used in the shellrc file.


fn: perform call.End() after request is processed

call.End() performs several tasks in sequence; insert call,
insert log, (todo) remove mq entry, fireAfterCall callback, etc.
These currently add up to the request latency as return
from agent.Submit() is blocked on these. We also haven't been
able to apply any timeouts on these operations since they are
handled during request processing and it is hard to come up
with a strategy for it. Also the error cases
(couldn't insert call or log) are not propagated to the caller.
With this change, call.End() handling becomes asynchronous where
we perform these tasks after the request is done. This improves
latency and we no longer have to block the call on these operations.
The changes will also free up the agent slot token more quickly
and now we are no longer tied to hiccups in call.End().
Now, a timeout policy is also added to this which can
be adjusted with an env variable. (default 10 minutes)
This accentuates the fact that call/log/fireAfterCall are not
completed when request is done. So, there's a window there where
call is done, but call/log/fireAfterCall are not yet propagated.
This was already the case especially for error cases.
There's slight risk of accumulating call.End() operations in
case of hiccups in these log/call/callback systems.

Resp latency improved, but it brings in these race conditions to the surface I think. We'll probably have to adjust TestRouteRunnerIOPipes, etc.


LinearNDInterpolator crashes/finishes with exit code 0xC0000005

scipy.interpolate.LinearNDInterpolator crashes with exit code -1073741819 (0xC0000005)
Inside Data.zip are the used numpy arrays (see example)
Reproducing code example:
import numpy as np
import scipy.interpolate
points = np.load('points.npy')
values = np.load('values.npy')
scipy.interpolate.LinearNDInterpolator(points, values, fill_value=0)
Error message:
Process finished with exit code -1073741819 (0xC0000005)
Scipy/Numpy/Python version information:
1.3.1 1.17.2 sys.version_info(major=3, minor=7, micro=4, releaselevel='final', serial=0)
Im a using a 64 bit Python and 64 bit numpy:
Python 3.7.4 (tags/v3.7.4:e09359112e, Jul  8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)] on win32

Same, fixed on my side
>>> import sys, scipy, numpy; print(scipy.__version__, numpy.__version__, sys.version_info)
1.7.0.dev0+b9b8d9b 1.19.4 sys.version_info(major=3, minor=8, micro=5, releaselevel='final', serial=0)

Feel free to re-open the issue if you still encounter the bug after updating.

I am still encountering this error. My version of scipy was:
import sys, scipy, numpy; print(scipy.__version__, numpy.__version__, sys.version_info)
1.7.3 1.21.2 sys.version_info(major=3, minor=8, micro=8, releaselevel='final', serial=0)

I updated scipy in case that fixed the issue, but it did not
1.9.0 1.21.2 sys.version_info(major=3, minor=8, micro=8, releaselevel='final', serial=0)
It seems to only happen with some subsets of the points I am trying to query. It does not seem to be a data size issue.

@ddileonardo do you have a reproducer?

That dataset is large. I'm looking for a small subset that fails, and one that works that I can provide.

Thank you for your quick response. I've got to get permission to share the data, which may take a couple of days.
Can you tell me anymore information about what caused this error before? Maybe that will shed some light for us. The error code does not provide very much information.

We've found that the length of the output in these cases does not match the length of the input. A thorough investigation of the data we are using does not reveal anything that would cause the issue. (Though I will still try to provide some example data)

Update: I have attempted to re-build my conda environment a few times now, and have not had success in running this function through all of our data. Unfortunately I can't share the data. I've done an extensive dive into the points where the issue occurs and there is nothing different about them compared to points that have no problems.
One of my coworkers is able to run this function through without the error, which lead me to think the issue was related to my conda environment. However, he is running Linux - Ubuntu OS and I am running Windows, so after rebuilding the environment multiple times, I now suspect the different operating systems might have something to do with the problem . I have included a yaml file that I used to build my conda environment, in case it is useful to anyone.
I don't have a solution for this issue. Instead I'll be transitioning to a Windows-Linus subsystem environment and changing somethings in the code to avoid the issue.
If there is interest in looking deeper into this issue, I can try to recreate the problem with public data.
requirements.txt

If there is interest in looking deeper into this issue, I can try to recreate the problem with public data.

Yes please. Without a reproducer, we cannot even tell if this a scipy issue or a packaging problem.


Release Google.Cloud.Dataproc.V1 version 3.4.0

Changes in this release:
New features

Spark runtime versioning for Spark batches (commit b6189ae)
Custom image containers for Spark batches (commit b6189ae)
Auto-diagnostic of failed Spark batches (commit b6189ae)
Local SSD NVME interface support for GCE clusters (commit b6189ae)

Created release for Google.Cloud.Dataproc.V1-3.4.0


Models Incompatible with Stardist package - need .h5 files

Hello,
I would like to use the models found here https://github.com/stardist/stardist-imagej/tree/master/src/main/resources/models/2D with the main stardist package. However, when I register the models using
 from csbdeep.models import register_model, register_aliases, clear_models_and_aliases
register_model(StarDist2D,   'stardist-imagej', 'https://github.com/stardist/stardist-imagej/blob/master/src/main/resources/models/2D/dsb2018_heavy_augment.zip', None)

I get the warning: base_model.py (163): Couldn't find any network weights (*.h5, *.hdf5) to load.. Could the zip files be updated with the .h5 files so the models can be easily used with the stardist api?

The models given here https://github.com/stardist/stardist-imagej/tree/master/src/main/resources/models/2D give very similar results to the models in the stardist package, such as 2d_versatile_fluo. If anyone is having trouble using the results of the model out of the box, I recommend making sure the images are normalized between 0 and 1. Thresholding to remove any noise will also help improve model results.

I would like to use the models found here https://github.com/stardist/stardist-imagej/tree/master/src/main/resources/models/2D with the main stardist package

Sorry, the Java models are in a different format and not usable from Python.

The models given here https://github.com/stardist/stardist-imagej/tree/master/src/main/resources/models/2D give very similar results to the models in the stardist package, such as 2d_versatile_fluo.

Yes, they are very similar but not identical. It's a bit confusing, sorry.


doc: describe the sidebar components (Aesop, Init, Lake, Lean, Mathlib, etc) and where their code is located

doc-gen4 (unlike the original) does not intend to be a tool that is only used within the mathlib eco system. People that build their own mathlib independent (or mathlib extending) eco systems should be able to do that just like that.
Because of this a text like this is not included currently and I also wouldn't like to include one that statically  refers to some part of the eco system.
The correct solution if we wish to extend the index is (IMO) to provide users with an ability to supply a markdown file (or in the future: A file in the format that David is working on) which gets rendered into the index.

Understood, but it seems that a bullet/blurb is needed for each source, which can be combined into a list. Or more simply a toggle to include my language only for mathlib. I can work on something if there is an agreed approach.

Understood, but it seems that a bullet/blurb is needed for each source, which can be combined into a list. Or more simply a toggle to include my language only for mathlib. I can work on something if there is an agreed approach.

I think there is a misunderstanding here? I do not intend for mathlib specific code to end up in doc-gen, also not as a switch, toggle etc. There shouln't be any features that are available to mathlib but not to other libraries that exist in the Lean universe.
To extend on my last comment, if people wish to have a custom index the way to go is IMO as follows:

Decide on some location for a markdown file, lets say docs/doc-gen-index.md or dcindex.md, doesn't really matter in the end.
Add code in the index renderer that checks whether this file is available and if it is uses the CMark dependency (already used for rendering doc-strings) to turn that file into HTML. Note that it should be possible to have a repo without this file still so people can just use doc-gen out of the box without any additional work.
Put that HTML into the generated index.html page
Add such a file to mathlib (or any other library) to get a custom index.

OK, so for Lean 4 itself, the file doc-gen-index.md would consist of: "Init is the initial code loaded by Lean 4 and is maintained in  the github repository  https://github.com/leanprover/lean4. Std is the standard library distributed with Lean 4 and is maintained in the github repository https://github.com/leanprover/std4." Or perhaps in two files in the two repositories.
Am I right that doc-gen4 has not been used to generate output posted somewhere for Lean 4 alone and not with mathlib4? And that the Lean 4 documentation, web page, and the book Functional Programming in Lean 4 do not refer anywhere to doc-gen4 output?

Am I right that doc-gen4 has not been used to generate output posted somewhere for Lean 4 alone and not with mathlib4?

yes

And that the Lean 4 documentation, web page, and the book Functional Programming in Lean 4 do not refer anywhere to doc-gen4 output?

I do not know.

OK, so for Lean 4 itself, the file doc-gen-index.md would consist of: "Init is the initial code loaded by Lean 4 and is maintained in the github repository https://github.com/leanprover/lean4. Std is the standard library distributed with Lean 4 and is maintained in the github repository https://github.com/leanprover/std4." Or perhaps in two files in the two repositories.

I wouldn't overcomplicate it so much for now. Once such a feature is implemented I'd suggest to just put a file with your text in the mathlib4 repo. doc-gen just renders that file (and none from other repos) when it is called for that repo. If the other libraries wish to have other, possibly long introductory texts they can do that in their own renders as well.
Long time the vision is to have doc-gen be used in a more https://docs.rs fashion where the libraries are hosted centrally and you dont generate one large doc blob for the entire transitive closure of your lib but instead just your library alone and the rest gets automatically linked to other libraries on that hypothetical future website.

Maybe there should be a feature which let each module provide a short description of it, and then the doc-gen4 gather all of them to the front page. Not hard-coded one.

I think this is related to #43.


Mobile Gestures

Is there a way to enable mobile gestures? I can think of swipe to load the next / previous image and of pan/zoom to show the index ... The documentation states there are mobile friendly gestures but I am not able to find more documentation about this specific topic.

Yes, it is explained in the VueFlux component.

In touchable screens you can slide right and left to show previous or next image. Will also display index if defined sliding up.

And in options section you will find enableGestures
Maybe I should create another section in wiki for it so will be easier to find the info.

XD totally missed that one. Thanks!

No problem, the component became so big that the documentation resulted in very long texts.
Regards!


Websocket scalability

From the docs I read this:

The websocket server scales naturally on Node.js cluster without any 3rd party dependencies like Redis. Also there is no need of sticky sessions.

It's pretty cool (I can suggest it handles by master process) but, it's only scalability on the one machine.
How it can be scale for several servers ?
If one connection will serve by one node, and the second connection in another node. So we can't send message from first node socket to another node sockets, isn't it?
Can we resolve this with adonis ?
I mean schema something like this
With socket.io it's just makes through redis adapter.

This issue was moved to adonisjs/adonis-websocket#67


Raise key length to 16bits and value length to 32bits

@fmorency We can make shards for values that are greater than 16 bits in length in the kvstore itself, so we technically don't need this PR. But if we want to have it, we should merge it before alpha.

@hansl let's discuss this PR today if you have some time.


Fixing server side export of scene

Initial draft for fixing https://github.com/excalidraw/excalidraw/issues/1468

@vjeux thanks a lot for the review!. I'll make the changes today and will notify you.

@vjeux before i commit these changes, I had a question. why do we want to rely on feature checking for these individual functions, rather than just checking if window exists, given that the all browsers support these?

I've been working on a lot of different environments (different browsers, jest, jsdom, node, v8/jsc/hermes...) over the years and I've bitten so many times by variations that now I don't take chances anymore.
In this case, I'm pretty sure that jsdom would have the window object but I wouldn't willing to bet that it has devicePixelRatio, so now you'd feed in undefined to a bunch of math computation and get NaN everywhere and have a hard time debugging it.


Cannot select transcript from media browser

From the forum:
Hi, I am trying to use the Transcript Reader template and upload a time text file.
I can upload the prepared XML file, but I am unable to see the uploaded file in the media browser. But the file is there on OS level.
Same happens with text files, they are not shown.

This is caused by the exclusion of certain mimetypes in elfinder


add a wizard for OSIO

lets add a simple Wizard for use by OSIO letting OSIO pick from the latest booster catalog code
fixes #44

thanks for the detailed review @gastaldi!
As I tried to reuse the ProjectInfoStep cleanly; the easiest thing to do was just rework our wizard entirely to:

reuse the ChooseMissionStep and ChooseRuntimeStep steps
override the ProjectInfoStep to add the github/gogs/gitlab organisation input and override the validation logic

It was quite a lot of refactoring to work in this way but it hopefully helps aligns the UX between OSIO and LSIO.
One remaining issue is that the README of the generated apps doesn't have much relevance to using OSIO but other than that things are looking pretty good.
I'm hoping once the forge E2E tests are sorted on PRs we can get the launchpad-addon's CI to generate PRs against fabric8-generator to ensure we keep in sync going forwards


Sensor Number Not Showing

Please, this could completely be user error, so I apologize in advance if that is the case.
I'm configuring HA sensor values into StreamDeck buttons.   They show for all that I am providing, except for one.
I have Powerwall Solar Now, Load Now, PW Now and Grid Export showing properly.  However, when I attempt to show the Powerwall charge, I just get the HA icon with no values.  Powerwall charge provides a percentage number, where the others provide kW.
Any thoughts?

The % sensor is top right.

It is the % that messes it up. If the {{state}} has the number portion of the title and {{unit_of_measurement}} holds the "%" sign then you need to use the formatting below to get the "%" to show up.
{{state}} {{ unit_of_measurement | urlencode }}
If you don't have {{ unit_of_measurement }} the use,
{{state}} {{ "%" | urlencode }}
Hope that helps.

Thanks @DarkPaladin3, I can understand the issue.  I set the states in the custom label and it worked as you suggested.  Best presentation in my instance was '{{state}}{{ "%"}}'


ARROW-9424: [C++][Parquet] Disable writing files with LZ4 codec

Due to ongoing LZ4 problems with Parquet files, this patch disables writing files with LZ4 codec by throwing a ParquetException.
In progress: adding exceptions for pyarrow when using LZ4 to write files and updating relevant pytests
Mailing list discussion: https://mail-archives.apache.org/mod_mbox/arrow-dev/202007.mbox/<CAJPUwMCM4ZaJB720%2BuoM1aSA2oD9jSEnzuwWjJiw6vwXxHk7nw%40mail.gmail.com>
Jira ticket: https://issues.apache.org/jira/browse/ARROW-9424

Ah I see that you're adding Python changes.  I fixed the lint problems here so be sure to rebase your changes

@patrickpai do you anticipate to complete this today? We are hoping to cut a release candidate tomorrow during the workday Central Europe Time so I can help finish this if needed

@wesm I think it's best if I get help on this. I'm totally new to the python codebase and wasn't expecting to finish today. I can let you take over.

No problem, I can take it from here.

@pitrou @xhochy It seems that despite adding the LZ4_FRAME format we've been continuing to use LZ4_RAW for Parquet files. Unfortunate that this hasn't seen more compatibility testing.

Wasn't it deliberate? IIRC we didn't want to break compatibility with existing files.

In any case, the format used by Hadoop is neither of both, it's LZ4_RAW with a custom header...

I don't recall but that may have been the case. Either way it's a giant mess since many people use pyarrow to write Parquet files to be consumed by JVM-based systems. I think we can infer that LZ4 is not often used from the fact that we haven't had more bug reports about it.

OK, writing is disabled but old files can still be read
n [2]: pq.write_table(table, 'not_allowed.parquet.lz4', compression='lz4')
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-2-597ef4749b0a> in <module>
----> 1 pq.write_table(table, 'not_allowed.parquet.lz4', compression='lz4')

~/code/arrow/python/pyarrow/parquet.py in write_table(table, where, row_group_size, version, use_dictionary, compression, write_statistics, use_deprecated_int96_timestamps, coerce_timestamps, allow_truncated_timestamps, data_page_size, flavor, filesystem, compression_level, use_byte_stream_split, data_page_version, **kwargs)
   1632                 data_page_version=data_page_version,
   1633                 **kwargs) as writer:
-> 1634             writer.write_table(table, row_group_size=row_group_size)
   1635     except Exception:
   1636         if _is_path_like(where):

~/code/arrow/python/pyarrow/parquet.py in write_table(self, table, row_group_size)
    586             raise ValueError(msg)
    587
--> 588         self.writer.write_table(table, row_group_size=row_group_size)
    589
    590     def close(self):

~/code/arrow/python/pyarrow/_parquet.pyx in pyarrow._parquet.ParquetWriter.write_table()
   1406
   1407         with nogil:
-> 1408             check_status(self.writer.get()
   1409                          .WriteTable(deref(ctable), c_row_group_size))
   1410

~/code/arrow/python/pyarrow/error.pxi in pyarrow.lib.check_status()
     97                 raise IOError(errno, message)
     98             else:
---> 99                 raise IOError(message)
    100         elif status.IsOutOfMemory():
    101             raise ArrowMemoryError(message)

OSError: Per ARROW-9424, writing files with LZ4 compression has been disabled until implementation issues have been resolved. It is recommended to read any existing files and rewrite them using a different compression.
In ../src/parquet/arrow/writer.cc, line 684, code: WriteColumnChunk(table.column(i), offset, size)

In [3]: pq.read_table('example.parquet.lz4').to_pandas()
Out[3]:
   f0
0   1
1   2
2   3
3   4
4   5


[Bug] Out of bounds error

var cmd = new CommandBuffer();
var e = cmd.Create(new ComponentType[..]);
var er = e.Reference(); // Cause out of bounds error

Well this kinda makes sense. It creates a buffer entity which does not exist yet (in the world, since its buffered)... and e.Reference() creates a reference to it, which is non existing and thus throws an exception.

I also wanted to ask how to do it right in that case. I create entity with command buffer and I need to save reference for it. It there some right approach to do it? Like a pattern probably.


relayed IPC from node to parent

Please run pnpm lint --fix to fix your stylistic changes.

Thanks for the PR but looks like it may have been abandoned.
This effort will be succeeded by https://github.com/privatenumber/tsx/pull/413


fix: Fix namespace-scoped install

Motivation
There are some problems when installing in namespace-scoped mode:

Though the docs mentioned that the RBAC should be changed, the install script currently applies the controller ClusterRole and ClusterRoleBinding (including namespace resource permission) regardless of whether the --namespace-scope-mode option is provided
The combination of the NAMESPACE_SCOPE=true env var and namespace resource permissions results in a broken/conflicted state where the controller will "mostly" operate in cluster scoped mode and watch / delete resources in other namespaces

Modifications

Split RBAC manifests into cluster-scope and namespace-scope variants, the latter including Role/RoleBinding instead of Cluster level equivalents
Have install script apply the appropriate RBAC based on the namespace-scope-mode cli setting (in addition to setting the controller env var as is already done)
Change the controller init logic to always use namespace mode if the env var is set
Update documentation accordingly

Result
Namespace scoped install works properly, won't touch any other namespaces.
It would probably also be good to add some safeguard checks to detect/prohibit cluster scope installs in more than one namespace, but that's outside of the scope of this PR.

/lgtm


req.rawBody is no longer available (regression #34)

req.rawBody is no longer available in 1.5.1, I think it was the new "formidable" integration in connect/bodyParser that removed the feature. Is there another way to get the raw content after bodyParser parses the body?

This is probably a result of a change made to Connect. senchalabs/connect@c3170eee8cd60c92bcccca6054c1ebbb93a1a821

@ryanrolds is there another way to get rawBody?
https://github.com/senchalabs/connect/commit/c3170eee8cd60c92bcccca6054c1ebbb93a1a821

I've had a few requests to add this back, however with Connect now supporting multipart this is really unlikely, it's a huge waste of memory. What do you need it for?

Sorry, pulled my comment because I was only confirming what you already knew. I don't have an answer, but your use case will affect what you can do about; which is likely why TJ is asking what you need it for.

That makes sense, but my agent is sending "application/json" and the server is using a custom JSON schema validator. I have two options, either send the request in another not parseable format like text/plain (the client has to change it's implementation and that's just not the right) or Stringify and Parse again. Point is I don't want bodyParser to work on some routes because I'd like to read the body myself, is there a way to create a custom express pipeline for certain routes? That would do the trick.
I think if we are gonna get rid of a feature like this we really need to offer an alternative, not to mention including it in the migration guide in http://expressjs.com/.

yeah things get a bit tricky when you want to exclude the middleware for some only, there are a few ways to approach the issues, for example if you wanted to ignore all /api/* pathnames you could do something like:
var parse = express.bodyParser();
app.use(function(req, res, next){
  if (0 == req.url.indexOf('/api') return next();
  parse(req, res, next);
});

I'm gonna try something like that, thanks TJ and Ryan.

One way I found to get around the "exclude one" middleware issue is to have another app (with routes) and then put the app.use(other_app.router()); before any middleware I did not want run. My use case was to avoid hitting the session for certain routes but I could see this being used for this case as well.
@visionmedia On a side note, how much memory is used by rawBody? Is it really that much that it is a problem to have it? Won't it be cleaned up anyway once the request is done.

@shtylman well it depends on the request, if you upload several large images, a video etc you'll be out of mem in no time

In that case, don't you need to use the raw body? Or would it just be delivered in chunks (which would mean rawBody would contain the whole request when that was not needed)?

the data is just streamed through and into temp files

@visionmedia Thanks for sharing the middleware tip.
Actually I need the progress event of formidable, and the only way I found to do that in express 2.5.1 is using the tricks here. Hope there could be some opt-out for not using the formidable parser in connect.

i'll probably add some options to make it simpler but you can also disable the multipart support with delete express.bodyParser.parse['multipart/form-data']

Far more better! Thank you TJ!

Was there any resolution as to how to get the raw body back? I use it for some authentication (the user needs to sign the bytes they send in the body) and so on the server I need to be able to look at the exact bytes they sent and do the same signature to validate them.

im still iffy about having rawBody, waste of memory if you're posting large json etc, it should really be a separate middleware buffer('json'), buffer('xml', 'json') or something

@visionmedia I like the idea of it being separate middleware that can also buffer the data how it wants. How would I ensure proper middleware flow given that scenario tho? I see that the bodyParser waits for the 'end' event before proceeding to the next middleware. I would image that my middleware would also need to wait for said event to make sure I have the full buffer, however if it happens before the bodyParser middleware then the body parser will not be able to gather the data itself. Maybe I need my own version of the body parser middleware in that case.

yeah it would be awkward you would have to re-emit on next tick or something :s

Actually, the following middeware before bodyParser seems to work:
app.use (function(req, res, next) {
    req.rawBody = '';
    req.setEncoding('utf8');
    req.on('data', function(chunk) { req.rawBody += chunk });
});

This seems reasonable as all of the 'data' events must happen before any of the 'end' events and so the 'rawBody' will be populated as expected. Thanks for the tip.

Question on this: Is there any way to get rawBody back? I tried this middleware chunk up above and it doesn't work. I'm using rawBody to get an HMAC signature from a post request.

@visionmedia thanks. Any good documentation as to how? not trying to be dense. I used to be able to just do hmac.update(body.rawData)

you can call update() several times


Regenerate Manifest.toml with Julia 1.9.2

Update of #379, let's see how many tutorials are (not) broken.
Broken/assertions fail:

[x] Logistic Regression: Fixed by https://github.com/TuringLang/TuringTutorials/pull/394/commits/c4242604dec1245dbcf12fdc1e28755f3cd8b68d
[x] Linear regression: Fixed by https://github.com/TuringLang/TuringTutorials/commit/2d64593d459f2fcaf40f0b19199dccc33f8d025a
[x] Variational inference: Fixed by https://github.com/TuringLang/TuringTutorials/commit/ef59c15c698e216bb8d07a86d09c20b55656a24e
[x] Gaussian processes: Disabled in #375, so does not matter :stuck_out_tongue:
[x] Seasonal time series: Fixed by https://github.com/TuringLang/TuringTutorials/pull/394/commits/33892a8659af4d4fdf53768816c2f07e7e5ce9e5

@torfjelde The PR is almost ready but I won't be able to finish it today. It seems the VI tutorial has to be updated due to the recent changes in Bijectors - if you want, you can have a look what's the issue and push a fix. Otherwise I'll check it tomorrow.
The time series tutorial I have to run locally to see whether it's an actual problem or we just have to adjust the assertion.
Generally, in case https://github.com/JuliaArrays/FillArrays.jl/pull/273 is merged and released soon, it would be nice to wait for it to avoid having to set FillArrays to 1.0.0 in the tutorials explicitly (otherwise Turing will be downgraded if you try to update the Manifest files). This would also allow us to benefit from some fixes and performance improvements in more recent FillArrays versions.

Awesome! I'll have a look at the Bijectors stuff :+1:

@torfjelde I fixed the VI tutorial, seems to run fine now.

Once you are fine with the changes, @yebai and @torfjelde, we have to update the buildkite hashes before merging the PR (currently I have to unblock manually every buildkite run). Actually I can't do that but I'll ping Elliot.

@staticfloat can you please regenerate the keys? I switched from Julia 1.6 to 1 in this PR since the latest Turing releases don't support Julia < 1.7 🙂

Thank you @staticfloat :slightly_smiling_face:


Remove Docker Cache action

When the Docker images were loaded from cache, tests started to fail. This pull request removes temporarily the Docker Cache action from the workflows until a solution is provided for this.

coverage: 99.236% (+0.1%) from 99.108%
when pulling 47b4d23859d2d44be1ca0b96d02654d448378266 on remove_docker_cache_action
into bee7007aca83220c93478ed23d4a9797a47d4f78 on master.


session timeout not working as expected

with 0.16.0 I did configure the session timeout in the application.properties with

dolphinPlatform.sessionTimeout=20

In the server log I can see:

DEBUG com.canoo.impl.server.config.DefaultPlatformConfiguration - Dolphin Platform starts with value for sessionTimeout = 20

I start my webapplication, log in and then I watch the console and the network traffic. After several minutes I still see all the polling calls return with code 200. And if I then decide to click on that causes the invocation of an dolphin action to check if we are still connected, the application works without problems. This means for me the session did not time out.
Expected behavior:

timeout after 20s
background polling to return once with no HTTP ok, but an unauthorized. After that no more polls
application is able to react to the timeout event
causing any dolphin action will not execute and produce error messages in the web console

turns out the timeout is in minutes, not in seconds. I still see the polling too often in case of a timeout, but this is acceptable


Reorganize backend and fix weird repository bug

Lightning fast issue resolution

Resolved with commit https://github.com/cpe308-309/Group-name/commit/3c9c233d8937a335f7b4cb4aefaa80da5eb439e7